This page is documenting a work in progress feature do not assume it is available on the mainline branch.

Rationale

MVC is an extension documented in the H.264 standard, specifically described in Annex H. It describes how to encode multiple views sharing the same temporal reference.

It is used in the Blu-Ray specification, in a DVB standard and in various research projects.

Decoder extension

Each view requires a separate decoder context and each decoder must take in account the fact the reference list could reference frames from different views.

Additional NAL Units

Three new NAL Units are introduced: PREFIX, SUBSET_SPS, EXTENDED_SLICE.

PREFIX
Extend the immediately next NAL adding view-specific information. Only NAL videos are affected.
SUBSET_SPS
Replaces the normal SPS for the specific view
EXTENDED_SLICE

Video SLICE for any layer besides the first, to be decoded in an extended AVFrame

AVFrame should have AV_FRAME_MULTIVIEW set and priv should contain a pointer to all extra frames.

Structures

Only after parsing SUBSET_SPS the number of views is known, if the number changes a reinit/reallocation is needed. To preserve compatibility with existing h264 decoder, new contextes should be allocated on demand.

NAL Parser

The NAL Parser should split each EXTENDED_SLICE since they could refer to different views and not skip NALs > 13.

PREFIX and EXTENDED_SLICE NALUs carry an header extension of three additional bytes.

Decoder

The slice threading should be disabled at least initially. Frame threading should work correctly as long the contextes get propagated correctly.

SPS livecycle

The SUBSET_SPS interaction is described in H.7.4.2. It is required to keep a separate list of SPS for this specific kind (shared across all the views) and use it only when decoding EXTENDED_SLICE.

The separate getter and setter must be introduce to abstract the interaction.

VOIdx <-> ViewId matching

According H.7.4.1.2.1 and JM, we have two identifiers one for the extended slice and one for the sps. The match-list depends on the current active sps.

Current challenges

Slice multithreading is not possible when decoding MVC, but the structure of context initialization for different layers is very similar.

In slice threading a global reference to the SPS is kept across different contexts. I have to do a reinit when dimensions are wrong, but this has to be done only for the first context, all the other will reuse that reference, or error out.

So when you decode the slice header you have to consider using the same context twice, and not two different ones. We should split the decode_slice_header function in two, one for normal and slice threading, the other for mvc decoding.

Paragraph

Use alerts to point to other blueprints

// code snippets here
...

Architecture Design

How it is implemented internally. Put the information useful to implement it or extend it here.


CategoryBlueprint CategoryWIP CategoryBlueprintActive