# StreamOchestrator | Consideration | Status | | ----------------------- | --------------------------------- | | Preferred import style | Directory-only _[1]_ | | Multithread environment | Should be runnable in a WebWorker | _[1]_ Only the `orchestrator` directory itself should be imported and relied on by the rest of the code, not its inner files (thus `./index.ts` should export everything that may be imported by outside code). ## Overview To be able to play a content, the player has to be able to download chunks of media data - called segments - and has to push them to media buffers, called `SegmentSinks` in the RxPlayer code. In the RxPlayer, the _StreamOrchestrator_ is the entry point for performing all those tasks. Basically, the _StreamOrchestrator_: - dynamically creates various `SegmentSinks` depending on the needs of the given content - orchestrates segment downloading and "pushing" to allow the content to play in the best conditions possible. ## Multiple types handling More often than not, content are divised into multiple "types": "audio", "video" and "text" segments, for example. They are often completely distinct in a Manifest and as such, have to be downloaded and decoded separately. Each type has its own media buffer. For "audio"/"video" contents, we use regular browser-defined _MSE_ SourceBuffers. For any other types, such as "text", those buffers are entirely defined in the code of the RxPlayer. We then create a different Stream for each type. Each will progressively download and push content of their respective type to their respective media buffer: ``` - AUDIO STREAM - |====================== | - VIDEO STREAM - |================== | - TEXT STREAM - |======================== | ``` _(the `|` sign delimits the temporal start and end of the buffer linked to a given Stream, the `=` sign represent a pushed segment in the corresponding buffer)_ ## A special case: SourceBuffers Media buffers created for the audio and/or video types rely on a _native_ (implemented by the browser) `SourceBuffer` Object implementation. Media buffers relying on native SourceBuffers have several differences with the other "custom" (entirely defined in the RxPlayer) types of media buffers: - They are managed by the browser where custom ones are implemented in JS. As such, they must obey to various browser rules, among which: 1. They cannot be lazily created as the content plays. We have to initialize all of them beforehand. 2. They have to be linked to a specific codec. 3. The `SourceBuffer` has to be linked to the MediaSource, and they have to the MediaSource after the media HTMLElement has been linked to the MediaSource - They are in a way more "important" than custom ones. If a problem happens with a SourceBuffer, we interrupt playback. For a custom media buffer, we can just deactivate it for the rest of the content. For example, a problem with subtitles would just disable those with a warning. For a video problem however, we fail immediately. - They affect buffering when custom media buffers do not (no text segment for a part of the content means we will just not have subtitles, no audio segment means we will completely stop, to await them) - They affect various API linked to the media element in the DOM. Such as `HTMLMediaElement.prototype.buffered`. Custom media buffers do not. Due to these differences, media buffers relying on SourceBuffers are often managed in a less permissive way than custom ones: - They will be created at the very start of the content - An error coming from one of them will lead us to completely stop the content on a fatal error ## PeriodStreams The _DASH_ streaming technology has a concept called _Period_. Simply put, it allows to set various types of content successively in the same manifest. For example, let's take a manifest describing a live content with chronologically: 1. an english TV Show 2. an old italian film with subtitles 3. an American film with closed captions. Example: ``` 08h05 09h00 10h30 now |==================|===========================|===================| TV Show Italian Film American film ``` Those contents are drastically different (they have different languages, the american film might have more available bitrates than the old italian one). Moreover, even a library user might want to be able to know when the italian film is finished, to report about it immediately in a graphical interface. As such, they have to be considered separately - in a different Period: ``` Period 1 Period 2 Period 3 08h05 09h00 10h30 now |==================|===========================|===================| TV Show Italian Film American film ``` In the RxPlayer, we create one _PeriodStream_ per Period **and** per type. _PeriodStreams_ are automatically created/destroyed during playback. The job of a single _PeriodStream_ is to process and download optimally the content linked to a single _Period_ and to a single type: ``` - VIDEO BUFFER - PERIOD STREAM 08h05 09h00 |========= | TV Show - AUDIO BUFFER - PERIOD STREAM 08h05 09h00 |============= | TV Show - TEXT BUFFER - PERIOD STREAM 08h05 09h00 |====== | TV Show ``` To allow smooth transitions between them, we also might want to preload content defined by a subsequent _Period_ once we lean towards the end of the content described by the previous one. Thus, multiple _PeriodStreams_ might be active at the same time: ``` +---------------------------- AUDIO ----------------------------------+ | | | PERIOD STREAM 1 PERIOD STREAM 2 PERIOD STREAM 3 | | 08h05 09h00 10h30 now | | |============= |================= |================ | | | TV Show Italian Film American film | +-------------------------------------------------------------------------+ +------------------------------ VIDEO --------------------------------+ | | | PERIOD STREAM 1 PERIOD STREAM 2 PERIOD STREAM 3 | | 08h05 09h00 10h30 now | | |===== |=== |=== | | | TV Show Italian Film American film | +-------------------------------------------------------------------------+ +------------------------------ TEXT ---------------------------------+ | | | PERIOD STREAM 1 PERIOD STREAM 2 PERIOD STREAM 3 | | 08h05 09h00 10h30 now | | (NO SUBTITLES) |========================= |================= | | | TV Show Italian Film American film | +-------------------------------------------------------------------------+ ``` ### Multi-Period management The creation/destruction of _PeriodStreams_ is actually done in a very precize and optimal way, which gives a higher priority to immediate content. To better grasp how it works, let's imagine a regular use-case, with two periods for a single type of buffer: --- Let's say that the _PeriodStream_ for the first _Period_ (named P1) is currently actively downloading segments (the "^" sign is the current position): ``` P1 |==== | ^ ``` Once P1 is full (it has no segment left to download): ``` P1 |======| ^ ``` We will be able to create a new _PeriodStream_, P2, for the second _Period_: ``` P1 P2 |======| | ^ ``` Which will then also download segments: ``` P1 P2 |======|== | ^ ``` If P1 needs segments again however (e.g. when the bitrate or the language is changed): ``` P1 P2 |=== |== | ^ ``` Then we will destroy P2, to keep it from downloading segments: ``` P1 |=== | ^ ``` --- Once P1, goes full again, we re-create P2: ``` P1 P2 |======|== | ^ ``` _Note that we still have the segment pushed to P2 available in the corresponding media buffer_ When the current position go ahead of a _PeriodStream_ (here ahead of P1): ``` P1 P2 |======|=== | ^ ``` This _PeriodStream_ is destroyed to free up ressources: ``` P2 |=== | ^ ``` --- When the current position goes behind the first currently defined _PeriodStream_: ``` P2 |=== | ^ ``` Then we destroy all previous _PeriodStreams_ and [re-]create the one needed: ``` P1 |======| ^ ``` In this example, P1 is full (as we already downloaded its segments) so we also can re-create P2, which will also keep its already-pushed segments: ``` P1 P2 |======|=== | ^ ``` --- For multiple types of buffers (example: _audio_ and _video_) the same logic is repeated (and separated) as many times. An _audio_ _PeriodStream_ will not influence a _video_ one: ``` --------------------------- AUDIO -------------------------------- Period 1 Period 2 |================|============= | ^ --------------------------- VIDEO -------------------------------- Period 1 |======= | ^ ---------------------------- TEXT -------------------------------- Period 1 Period 2 Period 3 |================| (NO SUBTITLES) |================ | ^ ``` At the end, we should only have _PeriodStream[s]_ for consecutive Period[s]: - The first chronological one is the one currently seen by the user. - The last chronological one is the only one downloading content. - In between, we only have full consecutive _PeriodStreams_. ### Communication with the API Any "Stream" communicates to the API about creations and destructions of _PeriodStreams_ respectively through `"periodStreamReady"` and `"periodStreamCleared"` callbacks.