Week 6: Audio
Looks only get you so far...
why audio?
Audio in games provides:
- Immersion
- Excitement
- Nonvisual cues
- Ambience
- etc...
audio on the web
For most of the web's history...
<embed>
Then...
<object>!
And now...
<audio>?
webaudio api
Now we can control audio in JS!
A page has an AudioContext with graphs
setup
Get the song in a buffer then play.
var context = new webkitAudioContext();
var req = new XMLHttpRequest();
req.open('GET', 'killerqueen.mp3', true);
req.responseType = 'arraybuffer';
req.onload = function() {
createSource(req.response);
}
function createSource(data) {
var source = context.createBufferSource();
source.buffer = context.createBuffer(data);
source.connect(context.destination);
source.noteOn(0);
} modifying the graph
Insert nodes into the chain.
var volumeNode = context.createGainNode();
volumeNode.gain.value = 0.1;
source.connect(volumeNode);
volumeNode.connect(context.destination); More nodes
As before, we just insert them into the chain.
var filterNode = context.createBiquadFilter();
filterNode.type = 0; // low-pass filter
filterNode.frequency.value = 220; // filter > 220Hz
volumeNode.connect(filterNode);
filterNode.connect(context.destination); spatializing
We need 3D positioning for 3D applications!
context.listener.setPosition(20, 5, 0)
var panner = context.createPanner();
panner.setPosition(10, 5, 0);
source.connect(panner);
panner.connect(context.destination); impulses & convolution
We can apply special effects to the sound with
impulse responses
, or
"acoustic photographs."
Convolution takes two wave forms and combines
them with fancy math.
var convolver = context.createConvolver();
loadSound("church.wav", convolver);
source.connect(convolver);
convolver.connect(context.destination); waveform analysis
Brief aside on sounds. Sounds are waves,
or amplitude over time. (time domain)
We can analyze them in the frequency
domain, too.
waveform analysis
Analysis in the frequency domain gives us
pitch and loundess.
waveform analysis
From this info, we can find:
- Music notes
- Beats
- Intensities
- etc.
analyzers
var processor = context.createJavaScriptNode(2048);
processor.onaudioprocess = process;
var analyzer = context.createAnalyzer();
analyzer.fftSize = 2048;
source.connect(processor);
processor.connect(analyzer);
analyzer.connect(destination);
function process() {
var inputArrayL = event.inputBuffer.getChannelData(0);
var inputArrayR = event.inputBuffer.getChannelData(1);
var outputArrayL = event.outputBuffer.getChannelData(0);
var outputArrayR = event.outputBuffer.getChannelData(1);
var n = inputArrayL.length;
for (var i = 0; i < n; ++i) {
outputArrayL[i] = inputArrayL[i];
outputArrayR[i] = inputArrayR[i];
}
var freqByteData = new Uint8Array(analyzer.frequencyBinCount);
analyzer.getByteFrequencyData(freqByteData);
// do something with freqByteData
} that's it!
Almost, anyway...
Week 6
By Will Crichton
