Web Audio API

Web Audio API, History of the Web, IndexedDB, Digital Signal Processing(DSP), Combining Uncorrelated Signals with Constant Power, Canvas, Analog Sound, Digital Audio, Fourier Transforms, Audio Visualization, Angular 2,  Web MIDI API, Binary Arithmetic, MIDI, WebRTC, Auto Correlation, Cross Correlation, The Art of Turntablism

History of Audio in the Web

  • In the behinning there was Flash
    • Plugin
    • not secure
    • ...Flash 
  • HTML5: <audio></audio>
    • Cross Browser support by 2011
    • No precise timing controls
    • No ability to apply real-time effects
    • No way to analyze sounds
  • Web Audio API
    • Awesome!
    • The Best!

Web Audio API Support

What is Web Audio?

This specification describes a high-level JavaScript API for processing and synthesizing audio in web applications. The primary paradigm is of an audio routing graph, where a number of AudioNode objects are connected together to define the overall audio rendering. The actual processing will primarily take place in the underlying implementation (typically optimized Assembly / C / C++ code), but direct JavaScript processing and synthesis is also supported.

- W3C Abstract

Audio Routing Graph

Audio Context

Destination

Visualization

Gain

Analyser

BufferSource

User Control

Atwood's Law

Any application that can be written in JavaScript, will eventually be written in JavaScript.

- Jeff Atwood

Open DVS Demo

Indexed Db

  • No storage limit
  • Store objects (not just JSON)
    • Supports circular references
  • Transactional
  • Async
  • Key-value Store (not relational)
[.25, .5, .7, .8, .82, .7, .2, 0, -.2, -.5, -.7]
Sample Rate Bit Depth 4 minute song
44,100 16bit 21MB

The Fourier Series

s_N(x) = \frac{A_0}{2} + \sum_{n=1}^N A_n\cdot \sin(\tfrac{2\pi nx}{P}+\phi_n), \quad \scriptstyle \text{for integer}\ N\ \ge\ 1.
sN(x)=A02+∑n=1NAn⋅sin(2πnxP+ϕn),for integer N ≥ 1.s_N(x) = \frac{A_0}{2} + \sum_{n=1}^N A_n\cdot \sin(\tfrac{2\pi nx}{P}+\phi_n), \quad \scriptstyle \text{for integer}\ N\ \ge\ 1.
  • Takes a signal and represents it as a series of sin waves.
  • Coverts between the "time domain" and "frequency domain"

Discrete Fourier Transform (DFT)

{\displaystyle X_{k}=\sum _{n=0}^{N-1}x_{n}\cdot (\cos(-2\pi k{\frac {n}{N}})+i\sin(-2\pi k{\frac {n}{N}})),\quad k\in \mathbb {Z} \,}
Xk=∑n=0N−1xn⋅(cos(−2πknN)+isin(−2πknN)),k∈Z{\displaystyle X_{k}=\sum _{n=0}^{N-1}x_{n}\cdot (\cos(-2\pi k{\frac {n}{N}})+i\sin(-2\pi k{\frac {n}{N}})),\quad k\in \mathbb {Z} \,}
  • Like the Fourier series but works with Discrete signals
    • e.g. a digital sampled audio signal

Fast Fourier Transform (FFT)

  • Any algorithm that can solve a DFT in 
O(n\log n)
O(nlogn)O(n\log n)

Joseph Fourier

1768 - 1830

  • Used by .jpg to compress images
  • Ised by .mp3 to compress audio

Open-DVS waveform / crossfade demo

E(|x+y|)
E(∣x+y∣)E(|x+y|)
x, y \in [-1, 1]
x,y∈[−1,1]x, y \in [-1, 1]
E(|x|) = 0.5
E(∣x∣)=0.5E(|x|) = 0.5
E(|y|) = 0.5
E(∣y∣)=0.5E(|y|) = 0.5
X
XX
Y
YY

1981
Companies such as Roland, Yamaha, Korg and Kawai start discussing coming up with a standard protocol

MIDI Specification finalized
1983

MIDI Timeline

                                       2015
Chrome implements the Web MIDI API

Web-MIDI API for Web browsers is the most significant advancement of MIDI since... MIDI itself

- The MIDI Association

1TTTCCCC          0AAAAAAA          0BBBBBBB

⏟

⏞

⏟

⏞

Message Type

  • 000 - Note Off
  • 001 - Note On
  • 010 - Aftertouch
  • 011 - Control Change
  • 100 - Prog Change
  • 101 - Chan Pressure
  • 110 - Pitch Bend
  • 111 - System Exclusive

Channel

1 - 16

Data Byte 1: 0 - 127

If type is "Note On" this represents the note number.
60 (0111100) === Middle C

Data Byte 2: 0 - 127

If Type is "Note On" this represents note velocity.

10110000          00000111          01000000

⏟

Control Change

⏞

Channel 1

⏟

111₂ === 7

Volume

1000000₂ === 64

⏞

Set channel 0 volume to 64/127

Open DVS MIDI Demo

Web RTC

  • RTC === Real Time Communication
  • Released by Google in 2011
  • Standerdized by W3C and IETF
  • Components
    • getUserMedia: allows a web browser to access the camera and microphone and to capture media
    • RTCPeerConnection: sets up audio/video calls
    • RTCDataChannel: allows browsers to share data via peer-to-peer

Σ

    ∫

..., .25, .5, .7, .8, .82, .7, .2, 0, -.2, -.5, -.7, ...

Script

[.25, .5, .7, .8, .82, .7, .2, 0, -.2, -.5, -.7]

Open DVS Script Processor Node

Made with Slides.com