Current state and future of audio and video in Firefox

Current state and future of audio and video in Firefox

What we will cover in this talk

  • What did we do in the media team this year?
  • What projects are ongoing?
  • What can you expect in the (near or not too near) future?

What did we do this year?

  • We added support for new codecs
  • We changed quite a few internal things that matter
  • We wrote new features

A new audio codec, Opus

Opus vs. other codecs

Written by Xiph.Org, standardized as RFC 6716, required for WebRTC.

How do we decode video using hardware.

  • Software decoding on mobile ☹
  • But we have a chip that can decode video ☺
  • But the libraries are buggy ☹
  • But now it works ☺ (mostly)

Platform decoders

Main goal: support H.264, AAC, mp3 without having to get a license, because we lost this war.

  • Windows Vista+: WMF (Windows Media Foundation)
  • Linux, Mac: GStreamer
  • Mac: Core Video
  • WMF:
  • GStreamer: ac_add_option --enable-gstreamer
  • Core Video: not yet implemented

Platform decoders: sites should work

  • Main problem: UA sniffing
  • HTMLVideoElement.canPlayType("video/webm")!
  • The fact that it returns "maybe", "probably" or "" is not an excuse not to use the API

Massive code cleaning

  • Refactored massively content/media
    • Removed now useless class hierarchy
    • Clear interfaces, less strong-coupling
    • Removed unused code
  • Prepare the code for others projects to plug in (WebRTC, WebAudio)
  • It was a bit of a mess sometimes, the code is much more approcheable now

New audio backend library

  • Called libcubeb, not sure why
  • Written in C, straightforward to use in other projets
  • Abstracts away CoreAudio, WinMM, Alsa, PulseAudio, AudioTrack, sndio, OpenSL…
  • Pull-based (uses callbacks instead of blocking write), hence less threads needed
  • Opens the door for latency optimizations, needed for B2G, WebRTC, games
  • Needs a last backend to be able to remove the old Sydneyaudio library

New features

  • DASH (Dynamic Adaptative Streaming over HTTP)
  • Chained OGG support (at last!)
  • playbackRate with pitch preservation
  • Metadata access in Javascript
  • Media sniffing for people that don't set Content-Type
  • getUserMedia
  • MediaStream framework to handle and synchronize All The Things™

Ongoing projects

  • WebAudio
  • WebVTT


  • We lost at the spec game, and we now have to implement it
  • Underspecified to death, hard to implement
  • Behind a pref: media.webaudio.enabled
  • A lot of DOM/refactoring work for now, nothing shiny to show
  • We are close to actually outputing sound!

WebAudio: what for?

  • Basically, like <canvas>, but for audio
  • Games (spatial positionning of audio)
  • Rich audio applications (sampler, synthetizer)
  • Intended to be used with WebRTC, getUserMedia
  • Will probably be used to polyfill everything we don't have
WebAudio example

WebVTT (Video Text Tracks)

  • Synchronized text tracks for media elements, e.g. subtitles for a video
  • Uses a format very close to the the popular .srt files
  • Written by David Humphrey's students, at Seneca College (Toronto), with the help of Mozilla folks
  •           00:00.000 --> 01:24.000
              Some subtitle
              02:04.000 --> 02:44.000
              Some other text right after

The Future

  • MediaSource
  • Latency
  • (platform decoders)


  • Allows author to pass in compressed audio buffers using Javascript to a media element
  • Needed to implement custom buffering strategy, protocols, for media
  • Usable to implement DRM stuff (☹)
  • Essential for the success of Shumway
  • Useful for massive hackery, very low-level

Latency optimizations

  • Games
  • WebRTC (getUserMedia)
  • Lightstreams?
  • Tighter integration?

Questions, complaints?