Web Audio

$ hg log -r : -l 1 content/media/webaudio
changeset:   104061:ee9b17600025
user:        Ehsan Akhgari <ehsan@mozilla.com>
date:        Fri Aug 31 16:59:37 2012 -0400
summary:     Bug 775302 - Stub out the implementation of mozAudioContext; r=bzbarsky,khuey
        

816 commits so far, feature complete implementation, landed in Firefox 25 (2013-10-29).

Commit time repartition:

• Games
• Music visualization
• Application feedback
• Musical applications
• Fancy streaming music player (until we have MediaSource Extension widely available Firefox bug, spec)
• (Ideally) Anything that should make noises

• Audio decoding to ArrayBuffers
• Precise timing of sound clips
• Arbitrary audio routing/mixing
• Effect transition, scheduling
• Ability to analyse audio
• Integration with the web platform
• Low-latency playback

What does the spec tell us?

[...] A high-level JavaScript API for processing and synthesizing audio in web applications. The primary paradigm is of an audio routing graph, where a number of AudioNode objects are connected together to define the overall audio rendering.

var ac = new AudioContext(); 
ac.decodeAudioData(ogg_arraybuffer, function(audiobuffer) { 
  var source = ac.createBufferSource();
  source.buffer = audiobuffer; 
  var d = ac.createDelay()
  var osc = ac.createOscillator();
  osc.type = "square";
  osc.frequency.value = 100; // Hz
  var g = ac.createGain();
  source.connect(d);
  source.connect(ac.destination);
  d.connect(ac.destination);
  g.connect(d);
  d.connect(g);
  osc.connect(g);
}, function error() { alert("This is broken"); });
      

• MediaStreamAudioSourceNode, from WebRTC, getUserMedia, etc.
• AudioBufferSourceNode to play a raw PCM audio buffer
• OscillatorNode to play predetermined waveforms
• ScriptProcessorNode (with no input) to generate arbitrary waveforms using JavaScript
• MediaElementAudioSourceNode to pipe the audio from <audio> or <video> in Web Audio

• GainNode: Change the volume
• DelayNode: Delays the input, in time
• ScriptProcessorNode (with both input and output connected): arbitrary processing
• PannerNode: Position a source and a listener in 3D and get the sound panned accordingly

• Channel{Splitter,Merger}Node: Merge or Split multi-channel audio
• ConvolverNode: Perform one-dimensional convolution (e.g. church-like reverb)
• WaveShaperNode: Non-linear wave shaping (e.g. guitar distortion)
• BiquadFilter: low-pass, high-pass, etc. (e.g. graphical equalizer)
• DynamicsCompressorNode: harmonizes audio levels

• MediaStreamAudioDestinationNode: Outputs to a MediaStream(send to a PeerConnection, a MediaRecorder)
• AudioDestinationNode: Outputs to speakers
• ScriptProcessorNode: arbitrary processing on input audio
• AnalyserNode: Get time-domain or frequency-domain audio data, in ArrayBuffers

• Web Audio Playground (on GitHub)
• Warwar (unreleased)

• We need to get people to support unprefixed AudioContext and the new function names (We have a wiki page to explain the process).
• A bunch of bugs to fix in Gecko, some bugs to fix in Webkit/Blink
• A massive amount of stuff to fix in the spec

• AudioContext in workers, so ScriptProcessorNode can actually work properly.
• Promises for all the asynchronous operations.
• Who knows ?

• Paul Adenot
• Web Audio API specification editor, Gecko developer
• padenot@mozilla.com
• @padenot
• #media on irc.mozilla.org to complain about Gecko
• https://github.com/WebAudio/web-audio-api/issues to complain about the spec.
• Slides: http://padenot.github.io/FOSDEM-14