This page is documenting a work in progress feature do not assume it is available on the mainline branch.


There are a few codecs that are structured in a "base layer" and one or more "enhancement layers". These codecs are designed to preserve compatibility with an existing codec and then add additional features not present in the the original codec or video itself.

For example one could have a video with two layers, each one representing a view of a stereo pair, or one could have a video with multiple layers that expand resolution, bitrate, framerate of the base layer, or one could have a video with a base layer in one codec and an enhancement layer of a completely different codec.

API Design

AVFrames are designed to hold only one frame worth of data, so the question arises when you have to deal with videos described above.

Compatibility with existing software has to be guaranteed, so decoding of additional video layers has to be explicitly opt-in. It is suggested to add AVOptions that allow decoding a single layer or that allow decoding all layers:

  • decode_view: "0,1,2" - provide only frames from the listed layers, default to 0

  • decode_all_views: boolean - provide frames from all layers, default to false

Architecture Design

Currently several possible implementation arise:

  1. modify the decode function so that an array of AVFrames are provided
    • pro: no changes to AVFrame

    • con: new API to handle decoding and possible downstream bugs

  2. extend AVFrame width and height so that full frames are in the same frame
    • pro: downstream applications receive all frames transparently and can get the original frame by changing the frame stride

    • con: will not work when base layer and enhancement layer have different resolution

  3. host a list of AVFrame inside the AVFrame of the base layer
    • pro: will not work when base layer and enhancement layer have different framerates

    • con: list management might be problematic

  4. provide two AVFrame sequentially with same timestamp
    • pro: no changes to AVFrame

    • con: downstream applications have hard time dealing with this

CategoryBlueprint CategoryWIP CategoryBlueprintActive