Problem

ACR Solutions ?

1. getting data

2. apply transforms

3. look up hashes

4. find shops

Steps

System Architecture

Audio Evolution

    // assuming a video is available in the DOM as: <video src="..." id="video"/>

    var uncompressedData = []
    
    var atx = new AudioContext()
    var source = atx.createMediaElementSource(document.getElementById('video'))
    

    var processor = atx.createScriptProcessor()
    processor.onaudioprocess = function (e) {
        var inputBuffer = e.inputBuffer
        var maxChannels = inputBuffer.numberOfChannels
        for (var channel = 0; channel < maxChannels; channel++) {
        	var input = inputBuffer.getChannelData(channel)
        	uncompressedData.push(input)
        }
    }
    

    source.connect(processor)
    processor.connect(atx.destination)

MediaElementAudioSource

CORS vs CDN

MediaElementAudioSource outputs zeroes due to CORS access restrictions for https://sassets.vevo.com/adsales/v3/SchwaebischHall/DE_Sponsorship3_300915/…0-a628-1b5eb184bb7b_mid_800k_640x360_h264_800_aac_128.mp4?c=WEB&cver=html5

Uncontrolled Environment

Uncaught InvalidStateError: Failed to execute 'createMediaElementSource' on 'AudioContext': HTMLMediaElement already connected previously to a different MediaElementSourceNode

Bypass CORS

Chrome Extension *

(* WebExtensions API in Firefox and MS Edge)

BackgroundScript to set 'Access-Control-Allow-Origin'

outgoing requests:

=> chrome.webRequest.onBeforeSendHeaders 

incoming response:

=> chrome.webRequest.onHeadersReceived

Record Streams

    // SourceBuffer.prototype is not accessible - wrap factory

    const BUFFERS = new WeakMap()

    const addSourceBuffer = MediaSource.prototype.addSourceBuffer
    MediaSource.prototype.addSourceBuffer = function (type) {
     
      const SourceBuffer = addSourceBuffer.call(this, type)
      const [ mime, codecs ] = type.split(';')
    
      // meta information
      BUFFERS.set(SourceBuffer, {
        'mime': mime,
        'codec': codecs.match(/codecs="(\S+)"/)[1],
        'segments': []
      })
    
      const appendBuffer = SourceBuffer.appendBuffer
      SourceBuffer.appendBuffer = function (data) {
        BUFFERS.get(this).segments.push(data)
        return appendBuffer.call(this, data)
      }
      return SourceBuffer
    }

Format/Codec Support

Converter

Custom (BinaryParser)

C/C++  =>  LLVM  =>  Emscripten  =>  JavaScript

optimized emscripted FFmpeg

CORS WebWorker Script

iFrame postMessages

Audio != Web Audio

Opus Codec can't be used in Chrome

WAV conversion 

Example

Open Source Implementations

Echoprint - official Service stopped

Chromaprint - requires complete audio file

Acoustic Fingerprint ?

Result