Barcelona Future JS 2014

Everybody dance now
with Web Audio API

Jaume Sanchez @thespite

http://www.clicktorelease.com

Audio on the browser

Flash and other plugins

HTML5 Audio Element

Hybrid approach

http://www.schillmania.com/projects/soundmanager2/

Web Audio API

(and a bit of Mozilla Audio Data API)

The Web Audio API allows developers to manipulate

and play audio assets on a Web page or application.

"Web Audio API is to HTML5 <audio>

what canvas is to <img>"

Mozilla Audio Data API tried with a

JavaScript-centric approach.

How to use

var context = new AudioContext();

Usually you'll want one AudioContext (AC) per page.

Once there's an AC, you can start

adding nodes and connecting them.

Nodes

(nodes everywhere)

Modular routing.

Basic audio operations are performed with audio nodes.

Audio nodes are linked together to form audio routing graphs.

Audio nodes are linked via their inputs and outputs.

This modular design allows the flexibility to create complex

audio functions with dynamic effects.

Routing graphs

Sources

We need some element to input data into the system.

(something has to make noise)

createBufferSource

createMediaElementSource

createMediaStreamSource

createOscillator

     var context = new AudioContext();
     var source = context.createBufferSource();
     source.buffer = buffer;
     source.start();

Destinations

Any other node, or the mixer.

The graph usually ends with context.destination

     var context = new AudioContext();
     var source = context.createBufferSource(); 
     source.buffer = buffer;
     source.connect( context.destination );
     source.start();

but the combinations are endless

     oscillator.connect( gain );
     gain.connect( convolver );
     convolver.connect( notchFilter );
     notchFilter.connect( context.destination );

Buffer Source

Creates an AudioBufferSourceNode interface.

context.createBufferSource();

buffer attribute: IEEE 32 bit float ranging -1 to 1

playbackRate attribute: AudioParam

loop/loopStart/loopEnd attributes

start() and stop() methods

onended event

Buffer Source demo

var context = new AudioContext();
var request = new XMLHttpRequest();
request.open( 'GET', 'sound.mp3', true );
request.responseType = 'arraybuffer';
request.onload = function() {

	context.decodeAudioData( request.response, function( buffer ) {
		
		var source = context.createBufferSource(); 
		source.buffer = buffer;
		source.connect( context.destination );
		source.start();
	
	}, function() {
		console.log( 'error' );
	} );
	
};
request.send();

Load Sound Demo

Load Sound reusing AudioBuffer Demo

Analyser

Creates an AnalyserNode to provide real-time

frequency and time-domain analysis information

context.createAnalyser()

fftSize attribute: size of FFT - power of two

getFloatFrequencyData: copies frequency-domain data

getByteFrequencyData: copies frequency-domain data

getByteTimeDomainData: copies time-domain data

Load Sound and show FFT Demo

getUserMedia Demo

AudioParam

float value (and defaultValue)

Allows scheduling of value changes

setValueAtTime

linearRampToValueAtTime

exponentialRampToValueAtTime

setTargetAtTime

setValueCurveAtTime

cancelScheduledValues

Gain

Create a GainNode to control the volume.

context.createGain();

gain attribute: AudioParam

     var context = new AudioContext();

     var source = context.createBufferSource(); 
     source.buffer = buffer;

     var volume = context.createGain();
     volume.gain.value = .1;

     source.connect( volume );
     volume.connect( context.destination );

     source.start();

Delay

Creates a DelayNode to add delay.

context.createDelay( maxDelayTime );

delayTime attribute AudioParam

     var context = new AudioContext();

     var source = context.createBufferSource(); 
     source.buffer = buffer;

     var delay = context.createDelay( 100 );
     delay.delayTime.value = 3;
     
     source.connect( delay );
     delay.connect( context.destination );

     source.start();

Delay + Gain = Reverb

Creating a loop between a gain and a delay node

creates a reverb effect on the stream.

     var context = new AudioContext();
     var a = document.createElement( 'audio' );
     a.src = 'track.wav';
     a.loop = false;
     var audioSource = context.createMediaElementSource( a );
     var delayNode = context.createDelay( 1 );
     var gainNode = context.createGain();
     gainNode.gain.value = .5;
     delayNode.delayTime.value = .1;
     audioSource.connect( delayNode );
     delayNode.connect( gainNode );
     gainNode.connect( delayNode );
     delayNode.connect( context.destination );
     a.play();

Gain + Delay Demo

Oscillator Source

Creates an OscillatorNode.

context.createOscillator();

type attribute:

SINE, SQUARE, SAWTOOTH,

TRIANGLE, CUSTOM

setWaveTable() method for CUSTOM type

frequency attribute AudioParam

     var context = new AudioContext();
     var oscillator = context.createOscillator();
     oscillator.frequency.value = 440;
     oscillator.type = 4;
     oscillator.connect(context.destination);
     oscillator.start();

Filters!

Creates a BiquadFilterNode that provides several

default stock second order filters.

context.createBiquadFilter();

type attribute

LOWPASS, HIGHPASS, BANDPASS,

LOWSHELF, HIGHSHELF, PEAKING,

NOTCH, ALLPASS

frequency attribute AudioParam

Q attribute AudioParam

gain attribute AudioParam

Oscillator and Biquad Filter Demo

Convolution (euh!)

Creates a ConvolverNode for linear convolution effects.

context.createConvolver();

buffer attribute: AudioBuffer with impulse response.

Convolution Demo

Panner

Creates a PannerNode for spatialisation.

context.createPanner();

Defines:

- position, velocity, direction of sample

- distance model

- directional sound cone

Tied to AudioListener (context.listener) for

spatial sound and doppler shift effect.

Other

createChannelMerger

createChannelSplitter

createDynamicsCompressor

createPeriodicWave

createWaveShaper

createJavaScriptNode

All together

There's a main ambient sound loop with gain.

The camera has a sound loop with gain.

Each cube has:

- a basic sound loop with gain

- a proximity sound loop with gain

- an activation sound with gain

- a deactivation sound with gain

- a sound analyser

- a panner

All sounds are merged in a convolver, a filter, a gain

and a dynamics compressor

Awesome.

Now what?

WebRTC getUserMedia

Web MIDI API

Procedural sound generation

Real-time effects

References

The spec

http://www.w3.org/TR/webaudio/

Getting Started with Web Audio API | HTML5 Rocks

http://www.html5rocks.com/en/tutorials/webaudio/intro/

Mixing Positional Audio and WebGL | HTML5 Rocks

http://www.html5rocks.com/en/tutorials/webaudio/positional_audio/

Thanks!

Questions?

Let's talk!