Web Audio

Web Audio

High-five at the Mozilla Festival!

A year ago, same room, same guy...

Gecko's Web Audio Journey

$ hg log -r : -l 1 content/media/webaudio
changeset:   104061:ee9b17600025
user:        Ehsan Akhgari <ehsan@mozilla.com>
date:        Fri Aug 31 16:59:37 2012 -0400
summary:     Bug 775302 - Stub out the implementation of mozAudioContext; r=bzbarsky,khuey

816 commits so far, feature complete implementation, landed in Firefox 25 (2013-10-29).

Commit time repartition:

commits time repartition

Use cases

  • Games
  • Music visualization
  • Application feedback
  • Musical applications
  • Fancy streaming music player (until we have MediaSource Extension widely available Firefox bug, spec)
  • (Ideally) Anything that should make noises

Feature Overview

  • Audio decoding to ArrayBuffers
  • Precise timing of sound clips
  • Arbitrary audio routing/mixing
  • Effect transition, scheduling
  • Ability to analyse audio
  • Integration with the web platform
  • Low-latency playback

What is Web Audio, in one sentence

What does the spec tell us?

[...] A high-level JavaScript API for processing and synthesizing audio in web applications. The primary paradigm is of an audio routing graph, where a number of AudioNode objects are connected together to define the overall audio rendering.

What is Web Audio, in one picture (and some code)

Web Audio Graph
var ac = new AudioContext(); 
ac.decodeAudioData(ogg_arraybuffer, function(audiobuffer) { 
  var source = ac.createBufferSource();
  source.buffer = audiobuffer; 
  var d = ac.createDelay()
  var osc = ac.createOscillator();
  osc.type = "square";
  osc.frequency.value = 100; // Hz
  var g = ac.createGain();
}, function error() { alert("This is broken"); });

Source Nodes

Those nodes output sound, to be processed by other nodes.
  • MediaStreamAudioSourceNode, from WebRTC, getUserMedia, etc.
  • AudioBufferSourceNode to play a raw PCM audio buffer
  • OscillatorNode to play predetermined waveforms
  • ScriptProcessorNode (with no input) to generate arbitrary waveforms using JavaScript
  • MediaElementAudioSourceNode to pipe the audio from <audio> or <video> in Web Audio

Processing Nodes

Take an input, tweaks it somehow, and outputs it.
  • GainNode: Change the volume
  • DelayNode: Delays the input, in time
  • ScriptProcessorNode (with both input and output connected): arbitrary processing
  • PannerNode: Position a source and a listener in 3D and get the sound panned accordingly

Processing Nodes (moar)

  • Channel{Splitter,Merger}Node: Merge or Split multi-channel audio
  • ConvolverNode: Perform one-dimensional convolution (e.g. church-like reverb)
  • WaveShaperNode: Non-linear wave shaping (e.g. guitar distortion)
  • BiquadFilter: low-pass, high-pass, etc. (e.g. graphical equalizer)
  • DynamicsCompressorNode: harmonizes audio levels

Output Nodes

  • MediaStreamAudioDestinationNode: Outputs to a MediaStream(send to a PeerConnection, a MediaRecorder)
  • AudioDestinationNode: Outputs to speakers
  • ScriptProcessorNode: arbitrary processing on input audio
  • AnalyserNode: Get time-domain or frequency-domain audio data, in ArrayBuffers


But there is still more work to be done !

  • We need to get people to support unprefixed AudioContext and the new function names (We have a wiki page to explain the process).
  • A bunch of bugs to fix in Gecko, some bugs to fix in Webkit/Blink
  • A massive amount of stuff to fix in the spec

Possible direction of the API

  • AudioContext in workers, so ScriptProcessorNode can actually work properly.
  • Promises for all the asynchronous operations.
  • Who knows ?

Questions ?


W3C AudioWG Logo Firefox Logo