Web Audio API

History of Audio in the Web

In the behinning there was Flash
- Plugin
- not secure
- ...Flash
HTML5: <audio></audio>
- Cross Browser support by 2011
- No precise timing controls
- No ability to apply real-time effects
- No way to analyze sounds
Web Audio API
- Awesome!
- The Best!

Web Audio API Support

What is Web Audio?

This specification describes a high-level JavaScript API for processing and synthesizing audio in web applications. The primary paradigm is of an audio routing graph, where a number of AudioNode objects are connected together to define the overall audio rendering. The actual processing will primarily take place in the underlying implementation (typically optimized Assembly / C / C++ code), but direct JavaScript processing and synthesis is also supported.

- W3C Abstract

Audio Routing Graph

Audio Context

Destination

Visualization

Gain

Analyser

BufferSource

User Control

Atwood's Law

Any application that can be written in JavaScript, will eventually be written in JavaScript.

- Jeff Atwood

Open DVS Demo

Indexed Db

No storage limit
Store objects (not just JSON)
- Supports circular references
Transactional
Async
Key-value Store (not relational)

[.25, .5, .7, .8, .82, .7, .2, 0, -.2, -.5, -.7]

Sample Rate	Bit Depth	4 minute song
44,100	16bit	21MB

The Fourier Series

s_N(x) = \frac{A_0}{2} + \sum_{n=1}^N A_n\cdot \sin(\tfrac{2\pi nx}{P}+\phi_n), \quad \scriptstyle \text{for integer}\ N\ \ge\ 1.

s_N(x) = \frac{A_0}{2} + \sum_{n=1}^N A_n\cdot \sin(\tfrac{2\pi nx}{P}+\phi_n), \quad \scriptstyle \text{for integer}\ N\ \ge\ 1.

Takes a signal and represents it as a series of sin waves.
Coverts between the "time domain" and "frequency domain"

Discrete Fourier Transform (DFT)

{\displaystyle X_{k}=\sum _{n=0}^{N-1}x_{n}\cdot (\cos(-2\pi k{\frac {n}{N}})+i\sin(-2\pi k{\frac {n}{N}})),\quad k\in \mathbb {Z} \,}

X_{k}=\sum _{n=0}^{N-1}x_{n}\cdot (\cos(-2\pi k{\frac {n}{N}})+i\sin(-2\pi k{\frac {n}{N}})),\quad k\in \mathbb {Z} \,

Like the Fourier series but works with Discrete signals
- e.g. a digital sampled audio signal

Fast Fourier Transform (FFT)

Any algorithm that can solve a DFT in

O(n\log n)

O(n\log n)

Joseph Fourier

1768 - 1830

Used by .jpg to compress images
Ised by .mp3 to compress audio

Open-DVS waveform / crossfade demo

E(|x+y|)

E(|x+y|)

x, y \in [-1, 1]

x, y \in [-1, 1]

E(|x|) = 0.5

E(|x|) = 0.5

E(|y|) = 0.5

E(|y|) = 0.5

X

Y

1981
Companies such as Roland, Yamaha, Korg and Kawai start discussing coming up with a standard protocol

MIDI Specification finalized
1983

MIDI Timeline

2015
Chrome implements the Web MIDI API

Web-MIDI API for Web browsers is the most significant advancement of MIDI since... MIDI itself

- The MIDI Association

1TTTCCCC 0AAAAAAA 0BBBBBBB

⏟

⏞

⏟

⏞

Message Type

000 - Note Off
001 - Note On
010 - Aftertouch
011 - Control Change
100 - Prog Change
101 - Chan Pressure
110 - Pitch Bend
111 - System Exclusive

Channel

1 - 16

Data Byte 1: 0 - 127

If type is "Note On" this represents the note number.
60 (0111100) === Middle C

Data Byte 2: 0 - 127

If Type is "Note On" this represents note velocity.

10110000 00000111 01000000

⏟

Control Change

⏞

Channel 1

⏟

111₂ === 7

Volume

1000000₂ === 64

⏞

Set channel 0 volume to 64/127

Open DVS MIDI Demo

Web RTC

RTC === Real Time Communication
Released by Google in 2011
Standerdized by W3C and IETF
Components
- getUserMedia: allows a web browser to access the camera and microphone and to capture media
- RTCPeerConnection: sets up audio/video calls
- RTCDataChannel: allows browsers to share data via peer-to-peer

∫

Open DVS Script Processor Node

Web Audio

By Rob McDiarmid