The Web Audio API

Using the Web Audio API for for demos

High-five at the Mozilla Festival!

Outline of the talk

  • Overview
  • Nodes presentation
  • Beat detection
  • Scheduling
  • Audio synthesis

What is Web Audio, in one sentence

What does the spec tell us?

[...] A high-level JavaScript API for processing and synthesizing audio in web applications. The primary paradigm is of an audio routing graph, where a number of AudioNode objects are connected together to define the overall audio rendering.
http://webaudio.github.io/web-audio-api/index.html#abstact

What is Web Audio, in one picture (and some code)

Web Audio Graph
var ac = new AudioContext(); 
ac.decodeAudioData(ogg_arraybuffer, function(audiobuffer) { 
  var source = ac.createBufferSource();
  source.buffer = audiobuffer; 
  var d = ac.createDelay()
  var osc = ac.createOscillator();
  osc.type = "square";
  osc.frequency.value = 100; // Hz
  var g = ac.createGain();
  source.connect(d);
  source.connect(ac.destination);
  d.connect(ac.destination);
  g.connect(d);
  d.connect(g);
  osc.connect(g);
}, function error() { alert("This is broken"); });
      

Use cases

  • Games
  • Audio visualization
  • Web Application audio feedback
  • Musical applications
  • DEMOS
  • Fancy streaming music player (until we have MediaSource Extension widely available Firefox bug, spec)
  • (Ideally) Anything on the web that makes non-trivial noises
  • (including offline rendering)

Feature Overview

  • Asynchronous compressed audio decoding to ArrayBuffers (formats: same as <audio>)
  • Precise timing for sample playback
  • Arbitrary audio routing/mixing
  • Effect transition, scheduling (automation)
  • Ability to analyse audio (e.g. FFT)
  • Low-latency playback

Nodes

  • Input nodes: produce audio -- audio sources
  • Output nodes: receive audio -- audio sinks
  • Processing nodes: process incoming audio, output processed signal

Source Nodes

  • AudioBufferSourceNode, sample player, plays AudioBuffer. One shot, but cheap to create
  • OscillatorNode, sine, square, saw, triangle, custom from FFT. One shot (use a GainNode)
  • ScriptProcessorNode (with no input) to generate arbitrary waveforms using JavaScript (but beware, it's deprecated !)
  • MediaElementAudioSourceNode to pipe the audio from <audio> or <video> in an AudioContext
  • MediaStreamAudioSourceNode, from PeerConnection, getUserMedia.

Processing Nodes

  • GainNode: Change the volume
  • DelayNode: Delays the input, in time. Needed for cycles.
  • ScriptProcessorNode (with both input and output connected): arbitrary processing (but beware, it's deprecated)
  • PannerNode: Position a source and a listener in 3D and get the sound panned accordingly
  • Channel{Splitter,Merger}Node: Merge or Split multi-channel audio from/to mono
  • ConvolverNode: Perform one-dimensional convolution between an AudioBuffer and the input (e.g. reverb)

Processing Nodes (moar)

  • WaveShaperNode: Non-linear wave shaping (e.g. guitar distortion)
  • BiquadFilter: low-pass, high-pass, band-pass, all-pass, etc.
  • DynamicsCompressorNode: adjust audio dynamics

Output Nodes

  • MediaStreamAudioDestinationNode: Outputs to a MediaStream(send to e.g. a WebRTC PeerConnection or a MediaRecorder to encode)
  • AudioDestinationNode: Outputs to speakers/headphones
  • ScriptProcessorNode: arbitrary processing on input audio
  • AnalyserNode: Get time-domain or frequency-domain audio data, in ArrayBuffers

Example 0: litsynth

  • I wrote a literate javascript program that explains it all
  • litsynth

Example 1: cheap beat detection

  • Audio -> AnalyserNode
  • Only keep low frequency (tune for the tune's kickdrum)
  • Threshold
  • Live example

<audio>

  • It still exists ! Perfect for playing streaming music !
  • For when you don't care about perf, scheduling effects
  • For when you don't want to have the data loaded in memory
  • Think about using MediaElementAudioSourceNode, best of both worlds!

Cheat sheet

Web Audio API cheat sheet : Here !

Firefox Web Audio Editor

Open the devtools, cog icon, enable it, reload the page that has an AudioContext.

In heavy development (Made by Jordan Santell (@jsantell)), available in Nightly builds. (The canvas editor can also be useful for demos)

Possible demo page (but any page should work, or that's a bug).

Questions ?

Thanks