TPAC 2016: Multi-device Timing CG

Multi-device Timing CG

Goal: define a common, multi-device, timing mechanism and a practical programming model for time-sensitive, multi-device Web applications

• Created in February 2015 following discussions in Web and TV IG
• Chair: Ingar M. Arntzen, Norut and Motion Corporation
• https://www.w3.org/community/webtiming/

Presentation URL

https://goo.gl/AlR9Nm

Outline

• Part 1 : Introduction
  • Status
  • What is multi-device timing?
  • Reality
  • Goals
• Part 2 : Proposal - Timing Object
• Part 3 : Applications
• Part 4 : Sync HTML5 Media
• Part 5 : Standardization
  • Priorities
  • Related initiatives

Demo URL

Live Demo URL removed after presentation.

Demo featured echoless synchronization of HTML video and audio across Web browsers of audience laptops.

Part 1: Introduction

Status CG

• by TPAC 2015
  • Draft standards proposal : Timing Object
  • Online Timing Provider : Motion Corporation
  • Multi-device Timing CG : Webpage
• by TPAC 2016
  • JS implementation : Timingsrc
  • NabShow 2016
  • MediaScape EU Project completes with excellent rating
  • IBC 2016
  • More Publications

Status Media Capabilities

• With unified media pipeline Chrome has become excellent for sync
• Safari with "bug" in variable playback rate making it rather useless
• Efforts in WebRTC leads to improvements in timing info in getUserMedia

Some fresh media sync measurements in part 4

What is Multi-device Timing?

• Timing
  • ensure time-sensitive stuff happens at the correct time
  • interactive control for temporal aspects
• Timing Scopes
  • page-internal (intra) timing and control : within browser context
  • cross-page (inter) timing and control : across browser contexts

• MTCG concerned with both scopes!
• MTCG does not prescribe a particular solution for cross-page timing and control!

Timing Challenges

Challenge	Use-case
across media sources	multi-angle video, ad-insertion
across media types	video + WebAudio + animated map
across iframes	video + timed ad-banner
across tabs, windows, devices	split content, interaction
across platforms	Web, native, broadcast
across people and groups	collaboration
across Internet	global media experiences

Realities : page-internal timing

• lots of timed stuff on the Web platform
• limited to single browser context
• heterogeneity in control API’s
• dependencies between components
• coarse and non-deterministic timing models
• unaccounted delays and unclear specs

Realities : cross-page timing

• no support : programmers left on their own
• synchronized system clocks not a sound assumption
• timing - hard
• distributed timing - even harder
• media control separate issue

• Goals
  1. Making multi-device timing dead simple for Web programmers!
  2. Temporal interoperability among timed components
  3. Make it possible for Web and native to interoperate
• Good news!
  • a concrete proposal
  • some new concepts, but no radical changes
  • feasible now!

Demo: Easy-peasy

https://youtu.be/D6cBn65KXTk

Part 2: Proposal

Timing Objects

• red circle moves along timeaxis --- (position, velocity)
• play, pause, jump, fast-forward --- modify (position, velocity)
• common representation of media control --- independent from media and purpose

Multi-media frameworks always built around similar concepts. No such object in JS context...

Demo Timing Object

API Timing Object

        var vector = {
          position: 12.0, // position (units)
          velocity: 1.0, // velocity (units/second)
          acceleration : 0.0, // acceleration (units/second/second)
          timestamp : 1365.234 // timestamp performance.now 
        };
      

API Timing Object

        var to = new TimingObject({range:[0,100]});
        var vector = to.query();
        to.update({position:14.0});
        to.update({position:14.0, velocity:1.3});
        to.on("change", handler);
        to.on("timeupdate", handler);
        to.off("change", handler);
      

Page-internal Timing Objects

• multiple media components observing and taking direction from timing object
• each component aligns itself with the timing object
• media control by updating the timing object, modifications apply to all components equally

Pass a timing object to all media components, and be done with it...

Online Timing Objects

• page-internal timing objects are proxies for online timing object
• kept in sync by online timing provider
• Web developers shielded from complexity of distributed sync

Distributed media components connected to a single online timing object...

Demo Online Timing Object

Approach summary

• temporal interoperability
  • page-internal and cross-page
• user space - JS
  • values availability over precision
  • programming model for timing
• online service
  • precision [0-5] ms
  • quick sync [<1] s
  • scope - global (IP and Web)
  • availability, reliability, scalability

Part 3: Applications

Social viewing

Multi-screen

Multi-device

Remote control

Music

Split content and interaction

Multi-device capture

Platform integration

Part 4: HTML5 Media Sync

Android Video

Nexus 6p

Android Audio

Nexus 6p

Laptop Chrome Video

Normal Chrome (not beta or dev). Chrome often works better on Webm

Laptop Chrome Audio

Normal Chrome (not beta or dev). Chrome often works better on Webm

Laptop Firefox Video

Firefox often works better on Webm

Laptop Firefox Audio

Firefox often works better on Webm

iPad Video

Safari iOS 10 on brand new iPad Air 2

iPad Audio

Safari iOS 10 on brand new iPad Air 2

iPad Video

Avoid playbackrate - skipping only

iPad Audio

Avoid playbackrate - skipping only

Part 5: Standardization

Standardization

• must have
  • improvements for JS-based HTML5 media sync --- playbackrate, seekTo and testing
• nice to have
  • timing object
  • internal HTML5 media sync
  • text tracks depending on timing object instead of HTML5 Media Elements
  • improvements setTimeout and setInterval
    • precision, reliability, internal clock of timing object
• ok in JS?
  • analogies to setTimeout/setInterval, based on timing object
  • sequencer (improved track element)

Some related initiatives

• Second Screen WG
• DVB TM-CSS (HbbTV2.0)
• MPEG Media Orchestration
• Ad-insertion
• Accessiblity

What's next?

• Standardization strategy?
• Attracting support?
• Group activities?

Summary

• With HTML5, the Web platform has taken important steps, catching-up with native apps, both desktop and mobile.
• Now, with multi-device timing the Web would get ahead.
• Multi-device timing also represents an excellent way of boosting cross platform integration, with Web and native components playing seamlessly together by virtue of shared timing and control.