The Web Audio API

• Overview
• Processing model
• Nodes presentation
• AudioParam
• Known shortcomings
• Firefox Devtool
• Gecko/Blink/Webkit/(Spartan?) Compatibility

What does the spec tell us?

[...] A high-level JavaScript API for processing and synthesizing audio in web applications. The primary paradigm is of an audio routing graph, where a number of AudioNode objects are connected together to define the overall audio rendering.

var ac = new AudioContext(); 
ac.decodeAudioData(ogg_arraybuffer, function(audiobuffer) { 
  var source = ac.createBufferSource();
  source.buffer = audiobuffer; 
  var d = ac.createDelay()
  var osc = ac.createOscillator();
  osc.type = "square";
  osc.frequency.value = 100; // Hz
  var g = ac.createGain();
  source.connect(d);
  source.connect(ac.destination);
  d.connect(ac.destination);
  g.connect(d);
  d.connect(g);
  osc.connect(g);
}, function error() { alert("This is broken"); });
      

• Games
• Audio visualization
• Web Application audio feedback
• Musical applications
• Fancy streaming music player
• (Ideally) Anything on the web that makes non-trivial noises
• (including offline rendering)

• Asynchronous compressed audio decoding to ArrayBuffers (formats: same as <audio>)
• Precise timing for sample playback
• Arbitrary audio routing/mixing
• Effect transition, scheduling (automation)
• Ability to analyse audio (e.g. FFT)
• Integration with the web platform (MediaRecorder, WebRTC, <audio>)
• Low-latency playback

• Message passing for everything to avoid blocking the audio thread (that would result in hearable glitches)
• You schedule things to happen on the audio thread, they do not happen immediatly, but at the next audio callback
• You can't retrieve values from the audio thread (they would be meaningless, because late)

• Input nodes: produce audio -- audio sources
• Output nodes: receive audio -- audio sinks
• Processing nodes: process incoming audio, output processed signal

• AudioBufferSourceNode, sample player, plays AudioBuffer. One shot, but cheap to create
• OscillatorNode, sine, square, saw, triangle, custom from FFT.
• ScriptProcessorNode (with no input) to generate arbitrary waveforms using JavaScript (but beware, it's broken!)
• MediaElementAudioSourceNode to pipe the audio from <audio> or <video> in an AudioContext
(beware of CORS)
• MediaStreamAudioSourceNode, from PeerConnection, getUserMedia.

• GainNode: Change the volume
• DelayNode: Delays the input, in time. Needed for cycles.
• ScriptProcessorNode (with both input and output connected): arbitrary processing (but beware, it's broken)
• PannerNode: Position a source and a listener in 3D and get the sound panned accordingly
• StereoPannerNode: Pan a source in 2D, better than PannerNode for musical applications. and get the sound panned accordingly

• Channel{Splitter,Merger}Node: Merge or Split multi-channel audio from/to mono
• ConvolverNode: Perform one-dimensional convolution between an AudioBuffer and the input (e.g. reverb)
• WaveShaperNode: Non-linear wave shaping (e.g. guitar distortion)
• BiquadFilter: low-pass, high-pass, band-pass, all-pass, etc.
• DynamicsCompressorNode: adjust audio dynamics

• MediaStreamAudioDestinationNode: Outputs to a MediaStream(send to e.g. a WebRTC PeerConnection or a MediaRecorder to encode)
• AudioDestinationNode: Outputs to speakers/headphones
• ScriptProcessorNode: arbitrary processing on input audio
  (still broken!)
• AnalyserNode: Get time-domain or frequency-domain audio data, in ArrayBuffers

• Almost every AudioNode parameter is an AudioParam
• Set directly: g.gain.value = 0.5;
• Use automation curve functions:
  • g.gain.setValueAtTime(0.5, ctx.currentTime + 1.0);
  • g.gain.linearRampToValueAtTime(0.5, ct + 1.0);
  • g.gain.exponentialRampToValueAtTime(0.0, ct + 1.0);
  • g.gain.setTargetAtTime(0.0, ct + 1.0, 0.66 /*exp constant*/);
  • g.gain.setValueCurveAtTime(arrayBuffer, ct + 1.0, arrayBuffer.length / ctx.sampleRate);
  • g.gain.cancelScheduledValues (ct);
• De-ziperring

             var osc = ctx.createOscillator(); // default: sine 
             osc.frequency.value = 4; // Hz 
             var lpf = ctx.createBiquadFilter(); // default: low-pass 
             var gain = ctx.createGain() 
             gain.gain.value = 1000; // [0; 1.0] to [0; 1000] 
             osc.connect(gain.gain); // osc -> AudioParam 
             gain.connect(lpf.frequency); 
             // dubstep 
             osc.start(0);
          

A fancy jsbin to play with the Web Audio API.

Example code at the bottom of the "output" panel, a drum loop and a reverb impulse are included, feel free to add your own files!

(Fork it before using it, to keep your changes)

• The ScriptProcessorNode is broken by design
• Sometimes Float32 are too memory-heavy (OfflineAudioContext trick to save memory)
• No FFT/IFFT exposed other than in the AnalyserNode
• Does not work (yet) in WebWorkers
• No way to enqueue buffers and have them play (but you have MediaSource for that)

• webkitAudioContext vs. AudioContext
• Chrome dropped the prefix (Safari won't) (Monkeypatching library, for easy compatibility)
• Old vs. new names (Our porting guide)
• Blink/Webkit do automatic de-zippering, Firefox doesn't (for now, it might change)
• MediaElementAudioSourceNode output is silence when it is from another server and CORS header is not there
• Use Chrome Canary (Win/OSX only) or Firefox Nightly (Win/Linux/OSX/Android/FxOS) to get the last features (Web MIDI in Chrome !)

• It still exists !
• It can play a Blob (from IndexedDB, XMLHttpRequest, local file, ArrayBuffer)
• For when you don't care about perf, scheduling effects
• For when you don't want to have the data loaded in memory
• Think about using MediaElementAudioSourceNode, best of both worlds!

Cheat sheet for the Music Hack Day: Here !

Don't be afraid to go read the spec! (or to ask me question, I'll be around)

Open the devtools, cog icon, enable it, reload the page that has an AudioContext.

In heavy development (Made by Jordan Santell (@jsantell)),

• Paul Adenot
• padenot@mozilla.com, emails welcome
• @padenot on Twitter
• #media on irc.mozilla.org to complain about Firefox's implementation (also padenot there, Paris timezone)
• bugzilla.mozilla.org to complain about Firefox's implementation, and actually get things fixed
• https://github.com/WebAudio/web-audio-api/issues to complain about the spec (please do)
• Slides: http://padenot.github.io/web-audio-scotlandjs14