Web Audio API

History of Audio in the Web

  • In the behinning there was Flash
    • Plugin
    • not secure
    • ...Flash 
  • HTML5: <audio></audio>
    • Cross Browser support by 2011
    • No precise timing controls
    • No ability to apply real-time effects
    • No way to analyze sounds
  • Web Audio API
    • Awesome!
    • The Best!

Web Audio API Support

What is Web Audio?

This specification describes a high-level JavaScript API for processing and synthesizing audio in web applications. The primary paradigm is of an audio routing graph, where a number of AudioNode objects are connected together to define the overall audio rendering. The actual processing will primarily take place in the underlying implementation (typically optimized Assembly / C / C++ code), but direct JavaScript processing and synthesis is also supported.

- W3C Abstract

Audio Routing Graph

Audio Context

Destination

Visualization

Gain

Analyser

BufferSource

User Control

Atwood's Law

Any application that can be written in JavaScript, will eventually be written in JavaScript.

- Jeff Atwood

Open DVS Demo

Indexed Db

  • No storage limit
  • Store objects (not just JSON)
    • Supports circular references
  • Transactional
  • Async
  • Key-value Store (not relational)
[.25, .5, .7, .8, .82, .7, .2, 0, -.2, -.5, -.7]
Sample Rate Bit Depth 4 minute song
44,100 16bit 21MB

The Fourier Series

s_N(x) = \frac{A_0}{2} + \sum_{n=1}^N A_n\cdot \sin(\tfrac{2\pi nx}{P}+\phi_n), \quad \scriptstyle \text{for integer}\ N\ \ge\ 1.
sN(x)=A02+∑n=1NAn⋅sin(2πnxP+ϕn),for integer N ≥ 1.s_N(x) = \frac{A_0}{2} + \sum_{n=1}^N A_n\cdot \sin(\tfrac{2\pi nx}{P}+\phi_n), \quad \scriptstyle \text{for integer}\ N\ \ge\ 1.
  • Takes a signal and represents it as a series of sin waves.
  • Coverts between the "time domain" and "frequency domain"

Discrete Fourier Transform (DFT)

{\displaystyle X_{k}=\sum _{n=0}^{N-1}x_{n}\cdot (\cos(-2\pi k{\frac {n}{N}})+i\sin(-2\pi k{\frac {n}{N}})),\quad k\in \mathbb {Z} \,}
Xk=∑n=0N−1xn⋅(cos(−2πknN)+isin(−2πknN)),k∈Z{\displaystyle X_{k}=\sum _{n=0}^{N-1}x_{n}\cdot (\cos(-2\pi k{\frac {n}{N}})+i\sin(-2\pi k{\frac {n}{N}})),\quad k\in \mathbb {Z} \,}
  • Like the Fourier series but works with Discrete signals
    • e.g. a digital sampled audio signal

Fast Fourier Transform (FFT)

  • Any algorithm that can solve a DFT in 
O(n\log n)
O(nlogn)O(n\log n)

Joseph Fourier

1768 - 1830

  • Used by .jpg to compress images
  • Ised by .mp3 to compress audio

Open-DVS waveform / crossfade demo

E(|x+y|)
E(∣x+y∣)E(|x+y|)
x, y \in [-1, 1]
x,y∈[−1,1]x, y \in [-1, 1]
E(|x|) = 0.5
E(∣x∣)=0.5E(|x|) = 0.5
E(|y|) = 0.5
E(∣y∣)=0.5E(|y|) = 0.5
X
XX
Y
YY

1981
Companies such as Roland, Yamaha, Korg and Kawai start discussing coming up with a standard protocol

MIDI Specification finalized
1983

MIDI Timeline

                                       2015
Chrome implements the Web MIDI API

Web-MIDI API for Web browsers is the most significant advancement of MIDI since... MIDI itself

- The MIDI Association

1TTTCCCC          0AAAAAAA          0BBBBBBB

⏟

⏞

⏟

⏞

Message Type

  • 000 - Note Off
  • 001 - Note On
  • 010 - Aftertouch
  • 011 - Control Change
  • 100 - Prog Change
  • 101 - Chan Pressure
  • 110 - Pitch Bend
  • 111 - System Exclusive

Channel

1 - 16

Data Byte 1: 0 - 127

If type is "Note On" this represents the note number.
60 (0111100) === Middle C

Data Byte 2: 0 - 127

If Type is "Note On" this represents note velocity.

10110000          00000111          01000000

⏟

Control Change

⏞

Channel 1

⏟

111₂ === 7

Volume

1000000₂ === 64

⏞

Set channel 0 volume to 64/127

Open DVS MIDI Demo

Web RTC

  • RTC === Real Time Communication
  • Released by Google in 2011
  • Standerdized by W3C and IETF
  • Components
    • getUserMedia: allows a web browser to access the camera and microphone and to capture media
    • RTCPeerConnection: sets up audio/video calls
    • RTCDataChannel: allows browsers to share data via peer-to-peer

Σ

    ∫

Open DVS Script Processor Node

Made with Slides.com