With temporal interoperability established as a goal for the Web, this section
surveys current abilities and limitations of the Web with respect to
media synchronization. The Web platform\footnote{In this chapter, the Web is
seen through the eyes of an end user browsing the Web with his/her favorite
\emph{user agent} in 2017.} is composed of a series of technologies centered
around the \emph{Hypertext Markup Language (HTML)}. These technologies have
been developed over the years and have grown steadily since the advent of
\emph{HTML5}~\cite{html5}, allowing Web applications to access an  ever-increasing 
pool of features such as local storage, geo{\-}location, peer-to-peer
communications, notifications, background execution, media capture, and more.
This section focuses on Web technologies that produce or
consume timed data, and highlights issues that arise when these technologies
are used or combined with others for synchronization purposes. These issues
are classified and summarized at the end of the section. Please note that this
section is written early 2017. It references technologies that are still under
development.

\subsection {HTML}
\label{sec:web-html}

First versions of the HTML specification (including HTML3.2~\cite{html32}) were
targeting static documents and did not have any particular support for timed playback. 
HTML5 introduced the Audio and Video media elements to add support for
\mbox{audio} and video data playback. Web applications may control the playback of
media elements using commands such as \emph{play} or \emph{pause} as well as
properties such as \mbox{\emph{currentTime}} (the current media offset)
and \emph{playbackRate} (the playback speed). In theory, this should be enough
to harness media element playback to any synchronization logic that authors
may be willing to implement. However, there are practical issues:

\begin{enumerate}

\item{

The playback offset of the media element is measured against a media clock,
which the specification defines as: \emph{user-agent defined, and may be media
resource-dependent, but [which] should approximate the user's wall clock.} In
other words, HTML5 does not impose any particular clock for media playback.
One second on the wall clock may not correspond to one second of playback, and
the relationship between the two may not be linear. Two media elements playing
at once on the same page may also follow different clocks, and thus media
offset of these two media elements may diverge over time even if playback was
initiated at precisely the same time.

}

\item{

HTML5 gives no guarantee about the latency that the software and the hardware
may introduce when the play button is pressed, and no compensation is done to
resorb that time afterwards.

}

\item{

The media clock in HTML5 automatically pauses when the user agent needs to
fetch more data before it may resume playback. This behavior matches the
expectations of authors for most simple media use cases. However, more advanced
scenarios where media playback is just a part of a larger and potentially
cross-device orchestration would likely require that the media clock keeps
ticking no matter what.

}

\item{

The \emph{playbackRate} property was motivated by the fast forward and rewind
features of \emph{Digital Video Disc (DVD)} players and previously
\emph{Videocassette Recorders (VCR)}. It was not meant for precise control of
playback velocity on the media timeline.

}

\end{enumerate}

To address use cases that would require synchronized playback of media
elements within a single page, for instance to play a sign language track as
an overlay video on top of the video it describes, HTML5 introduced the
concept of a \emph{media controller}~\cite{mediacontroller}. Each media
element can be associated with a media controller and all the media elements
that share the same media controller use the same media clock, allowing
synchronized playback. In practice though, browser vendors did not implement
media controllers and the feature was dropped in HTML5.1~\cite{html51}. It is
also worth noting that this mechanism was restricted to media elements and
could not be used to orchestrate scenarios that involved other types of timed
data.

While sometimes incorrectly viewed as a property of the JavaScript language,
the \emph{setTimeout}, \emph{setInterval} and other related timer functions,
which allow apps to schedule timeouts, are actually methods of the
\emph{window} interface, defined in HTML5. These methods take a timeout
counter in milliseconds, but the specification only mandates that Web browsers
wait until at least this number of milliseconds have passed (and only provided
the Web page has had the focus during that time). In particular, Web browsers
may choose to wait a further arbitrary length of time. This allows browsers to
optimise power consumption on devices that are in low-power mode. Even if
browsers do not wait any further, the event loop may introduce further delays
(see Section~\ref{sec:eventloop}). Surprisingly, browsers also fire timers too
early on occasion. All in all, the precision of timeouts is not guaranteed on
the Web, although experience shows that timeouts are relatively reliable in
practice.

\subsection{SMIL and Animations}
\label{sec:smil}

Interestingly, one of the first specifications to have been published as a Web
standard after HTML3.2~\cite{html32}, and as early as 1998, was the
\emph{Synchronized Multimedia Integration Language (SMIL)} 1.0
specification~\cite{smil1}. SMIL allowed integrating a set of independent
multimedia objects into a synchronized multimedia presentation. SMIL 1.0 was
the first Web standard to embed a notion of timeline (although it was only
implicitly defined). The specification did not mandate precise synchronization
requirements: \emph{the accuracy of synchronization between the children in a
parallel group is implementation-dependent.} Support for precise timing has
improved in subsequent revisions of SMIL, now in version 3.0~\cite{smil3}.

No matter how close to HTML it may be, SMIL appears to Web application
developers as a format on its own. It cannot simply be added to an existing
Web application to synchronize some of its components. SMIL has also never
been properly supported by browsers, requiring plugins such as
RealPlayer~\cite{realplayer}. With the disappearance of plugins in Web
browsers, authors are left without any simple way to unleash the power of SMIL
in their Web applications.

That said, SMIL 1.0 sparked the SMIL Animation
specification~\cite{smilanimation} in 2001, which builds on the SMIL 1.0
timing model to describe an animation framework suitable for integration with
\emph{Extensible Markup Language (XML)} documents. SMIL Animation has notably been incorporated in the Scalable
Vector Graphics (SVG) 1.0 specification~\cite{svg}, published as a Web
standard immediately afterwards. It took many years for SVG to take over
Flash~\cite{flash} and become supported across browsers, with the notable
exception of SMIL animations, which \emph{Microsoft}~\cite{microsoft} never
implemented, and which \emph{Google}~\cite{google} now intends to drop in
favor of \emph{CSS Animations} and of the \emph{Web Animations specification}.

While still a draft when this book is written, Web Animations~\cite{webanimation} appears as
a good candidate specification to unite all Web animation frameworks into one,
with solid support from \emph{Mozilla}~\cite{mozilla}, Google and now Microsoft. It introduces the
notion of a \emph{global clock}:
\begin{quote} a source of monotonically increasing time values unaffected by
adjustments to the system clock. The time values produced by the global clock
represent wall-clock milliseconds from an unspecified historical moment.
\end{quote}
The specification also defines the notion of a \emph{document timeline}
that provides time values tied to the global clock for a particular document.
It is easy to relate the global clock of Web Animations with other clocks
available to a Web application (e.g. the \emph{High Resolution Time} clock mentioned
in Section~\ref{sec:hrt}). However, the specification acknowledges that the setup of some
animations \emph{may incur some setup overhead}, for instance when the user agent
delegates the animation to specialized graphics hardware. In other words, the
exact start time of an animation cannot be known a priori.



\subsection{DOM Events}
\label{sec:domevents}

The ability to use scripting to dynamically access and update the content,
structure and style of documents, was developed in parallel to HTML, with
\emph{ECMAScript} (commonly known as JavaScript), and the publication of the
\emph{Document Object Model (DOM) Level 1} standard in 1998~\cite{dom1}. This
first level did not define any event model for HTML documents, but was quickly
followed by \emph{DOM Level 2}~\cite{dom2} and in particular the DOM Level 2
Events standard~\cite{domevents} in 2000. This specification defines: \emph{a
platform- and language-neutral interface that gives to programs and scripts a
generic event system}.

DOM events feature a \emph{timeStamp} property used to specify the time relative to
the epoch at which the event was created. DOM Level 2 Events did not mandate
that property on all events. Nowadays, DOM Events, now defined in the DOM4
standard~\cite{dom4}, all have a timestamp value, evaluated against the system
clock.

The precision of the timestamp value is currently limited to milliseconds, but
Google has now switched to using higher resolution timestamps associated with
the \emph{high resolution clock} (see Section~\ref{sec:hrt}). On top of
improving the precision down to a few microseconds, this change also means
that the \emph{monotonicity} of timestamp values can now be guaranteed.
\emph{Monotonicity} means that clock values are never decreasing. This change
will hopefully be included in a future revision of the DOM standard and
implemented across browsers.



\subsection{The Event Loop}
\label{sec:eventloop}

On the Web, all activities (including \emph{events, user interactions,
scripts, rendering, networking}) are coordinated through the use of an
\emph{event loop}\footnote{There may be more than one event loop, more than
one queue of tasks per event loop, and event loops also have a micro-task
queue that helps prioritizing some of the tasks added by HTML algorithms, but
this does not change the gist of the comments contained in this section.},
composed of a queue of tasks that are run in sequence. For instance, when the
user clicks a button, the user agent queues a task on the event loop to
dispatch the \emph{click} event onto the document. The user agent cannot
interrupt a running task in particular, meaning that, on the Web, all scripts
run to completion before further tasks may be processed.

The event loop may explain why a task scheduled to run in 2 seconds from now
through a call to the \emph{setTimeout} function may actually run in 2.5 seconds
from now, depending on the number of tasks that need to run to completion
before this last task may run. In practice, HTML5 has been carefully designed
to optimize and prioritize the tasks added to the event loop, and the
scheduled task is unlikely to be delayed by much, unless the Web application
contains a script that needs to run for a long period of time, which would
effectively freeze the event loop.

Starting in 2009, the Web Workers specification~\cite{webworkers} was developed to allow
Web authors to run scripts in the background, in parallel with the scripts
attached to the main document page, and thus without blocking the user
interface and the main event loop. Coordination between the main page and its
workers uses message passing, which triggers a \emph{message} event on the event
loop.

Any synchronization scenario that involves timed data exposed by some script
or event logic will de facto be constrained by the event loop. In turn, this
probably restricts the maximum level of precision that may be achieved for
such scenarios. Roughly speaking, it does not seem possible to achieve less
than one millisecond precision on the Web today if the event loop is involved.


\subsection{High Resolution Time}
\label{sec:hrt}

In JavaScript, the \emph{Date} class exposes the system clock to Web
applications. An instance of this class represents a number of milliseconds
since January 1., 1970 UTC. In many cases, this clock is a good enough
reference. It has a couple of drawbacks though:

\begin{enumerate}

\item{

The system clock is not monotonic and it is subject to adjustments. There is
no guarantee that a further reading of the system clock will yield a greater
result than a previous one. Most synchronization scenarios need to rely on the
monotonicity of the clock. 

}
\item {

Sub-millisecond resolution may be needed in some
cases, e.g. to compute the frame rate of a script based animation, or to
precisely schedule audio cues at the right point in an animation.

}

\end{enumerate}

As focus on the Web platform shifted away from documents to applications and
as the need to improve and measure performance arose, a need for a better
clock for the Web that would not have these restrictions emerged. The High
Resolution Time specification~\cite{hrt1} defines a new clock,
\emph{Performance.now()}, that is both guaranteed to be monotonic and accurate
to 5 microseconds, unless the user agent cannot achieve that accuracy due to
software or hardware constraints. The specification defines the time origin of
the clock, which is basically the time when the \emph{browsing context} (i.e.
browser Window, tab or iFrame) is first created. The very recent High
Resolution Time Level 2 specification~\cite{hrt2} aims to expose a similar
clock to background workers, and provide a mechanism to relate times between
the browsing context and workers.

It seems useful to point out that the 5 microseconds accuracy was not chosen
because of hardware limitations. It was rather triggered by privacy concerns
as a way to mitigate so called cache attacks, whereby a malicious Web site
uses high resolution timing data to fingerprint a particular user. In
particular, this sets a hard limit to precision on the Web, that will likely
remain stable over time.


\subsection{Web Audio API}
\label{sec:webaudio}

At about the same time that people started to work on the High Resolution Time
specification, Mozilla and Google pushed for the development of an API for
processing and synthesizing audio in Web applications. The Web Audio API draft
specification~\cite{webaudio} is already available across browsers. It builds upon an
audio routing graph paradigm where audio nodes are connected to define the
audio rendering.

Sample frames exposed by the Web Audio API have a \emph{currentTime} property that
represents the position on the Audio timeline, according to the clock used to
generate the audio stream. In other words, the hardware clock of the underlying
sound card. As alluded to in the specification, this clock \emph{may not be
synchronized with other clocks in the system}. In particular, there is little
chance that this clock be synchronized with the High Resolution Time clock,
the global clock of Web Animations, or the media clock of a media element.

The group that develops the Web Audio API at W3C investigated technical
solutions to overcome these limitations. The API now exposes the relationship
between the audio clock and the high resolution clock, coupled with the
latency introduced by the software and hardware, so that Web applications may
compute the exact times at which a sound will be heard. This is particularly
valuable for cross-device audio scenarios, but also allows audio to
be output on multiple sound cards at once on a single device.




\subsection{Media Capture}
\label{sec:capture}

W3C started to work on the Media Capture and Streams
specification~\cite{capture} in 2011. This specification defines the notions
of \emph{MediaStreamTrack}, which \emph{represents media of a single type that
originates from one media source} (typically video produced by a local camera)
and of \emph{MediaStream}, which is a group of loosely synchronized
MediaStreamTracks. The specification also describes an API to generate
MediaStreams and make them available for rendering in a media element in
HTML5.

The production of a MediaStreamTrack depends on the underlying hardware and
software, which may introduce some latency between the time when the data is
detected to the time when it is made available to the Web application. The
specification requires user agents to expose the target latency for each
track.

The playback of a MediaStream is subject to the same considerations as those
raised above when discussing media support in HTML5. The media clock is
implementation-dependent in particular. Moreover, a MediaStream is a
\emph{live} element and is not seekable. The \emph{currentTime} and
\emph{playbackRate} properties of the media element that renders a MediaStream
are \emph{read-only} (i.e. media controls do not apply), and thus cannot be
adjusted for synchronization\footnote{In the future, it may be possible to re-create 
a seekable stream out of a MediaStream, thanks to the
\emph{MediaRecorder} interface defined in the MediaStream Recording
specification~\cite{mediastreamrecording}. This specification is not yet stable when this book is
written.}.


\subsection{WebRTC}
\label{sec:webrtc}

Work on \emph{Web Real-Time Communication (WebRTC)} and its first specification, 
the WebRTC 1.0: Real-time Communication Between Browsers
specification~\cite{webrtc}, started at the same time as the work on media
capture, in 2011. As the name suggests, the specification allows media and
data to be sent to and received from another browser. There is no fixed timing
defined, and the goal is to minimize latency. How this is achieved in practice
is up to the underlying protocols, which have been designed to reduce latency
and allow peer-to-peer communications between devices.

The WebRTC API builds on top of the Media Capture and Streams specification
and allows the exchange of MediaStreams. On top of the synchronization
restrictions noted above, a remote peer does not have any way to relate the
media timeline of the MediaStream it receives with the clock of the local
peer that sent it. The \mbox{WebRTC} API does not expose synchronization primitives.
This is up to Web applications, which may for instance exchange
synchronization parameters over a peer-to-peer data channel. Also, the
MediaStreamTracks that compose a MediaStream are essentially treated
independently and re-aligned for rendering on the remote peer, when possible.
In case of transmission errors or delays, loss of synchronization, e.g.
between audio and video tracks, is often preferred in WebRTC scenarios to
avoid accumulation of delays and glitches.

\subsection{Summary}

While the High Resolution Time clock is a step in the right direction, the
adoption is still incomplete. As of early 2017, given an arbitrary set of
timed data composed of audio/video content, animations, synthesized audio,
events, and more there are several issues Web developers need to face to
synchronize the presentation:


\begin{enumerate}

\item{
Clocks used by media components or media subsystems may be different and may
not follow the system clock. This is typically the case for media elements in
HTML5 and for the Web Audio API.
}

\item{
The clock used by a media component or a media subsystem may not be monotonic
or sufficiently precise.
}

\item{
Additionally, specifications may leave some leeway to implementers on the
accuracy of timed operations, leading to notable differences in behavior
across browsers.
}

\item{
Operations may introduce latencies that cannot easily be accounted for. This
includes running Web Animations, playing/resuming/capturing media, or
scheduling events on the event loop.
}

\item{
Standards may require browsers to pause a motion for buffering, as typically
happens for media playback in HTML5. This behavior does not play well with the
orchestration of video with other types of timed data that do not pause for
buffering.
}

\item{ The ability to relate clocks is often lost during the transmission of
timestamps from one place to another, either because different time origins
are used, as happens between an application and its workers, or because the
latency of the transmission is not accounted for, e.g. between WebRTC peers.
At best, applications developers need to use an out-of-band mechanism to
convert timestamps and account for the transport latency. }

\item{
When they exist, controls exposed to harness media components may not be
sufficiently fine-grained. For example, the \emph{playbackRate} property of media
elements in HTML5 was not designed for precise adjustments, and setting the
start time of a Web animation to a specific time value may result in a
significant jump between the first and second frames of the animation.
}

\end{enumerate}

Small improvements to Web technologies should resolve some of these issues,
and discussions are underway in relevant standardization groups at W3C when
this book is written. For example, timestamps in DOM Events may
switch to using the same \emph{Performance.now()} clock. This is all good news for
media synchronization, although it may still take time before the situation
improves.

We believe that a shift of paradigm is also needed. The Web is all about
modularity, composition and interoperability. Temporal aspects have remained
an internal issue specific to each technology until now. In the rest of this
chapter, a programming model is presented to work around the restrictions
mentioned above, allowing media to be precisely orchestrated on the Web, even
across devices.




