This page is documenting a work in progress feature do not assume it is available on the mainline branch. You may find it developed on a topic branch

This development has a fundraiser open.

Rationale

The project aims to replace the software scaling and color conversion library in Libav with an easier to use and easier to extend replacement that leverages the actual hardware available and solves current color conversion problems.

You plan to take out a relic from the original MPlayer and replace it with something fitting the current century.
Color conversion and scaling, when done in software may account to the biggest share of the cpu when decoding a video. With 4K and cinema oriented pixel formats such as XYZ, it gets more and more important to have better and possibly hardware backed scalers .

Current status

The software scaling library currently provided originates as a tightly coupled part of MPlayer, thus having a quite peculiar API and a quite convoluted internal structure.
It was written with the assumption that YUV420 would be the most used pixel format, thus making it the default intermediate. The support for additional input and output formats had been stitched together working around this.
The API itself was originally prepared to work tightly coupled with MPlayer internals, making it a bit hard to follow and sporting a large number of now-unused parameters.
Non x86 (powerpc, arm, aarch64) architecture specific optimization could not be fit easily within the current infrastructure.
AVScale aims to provide:
  • high quality conversion of high-bitdepth color formats (e.g. 10-bit YUV422 or 12-bit XYZ)
  • an easy way to use hardware accelerated scaling
  • support for every theoretically possible format (like YUV colorspace with different subsampling for chroma components and uncommon bitdepth) and more.

API Design

  • AVFrame based API

int avscale_process_frame(AVScaleContext *c, AVFrame *dst, AVFrame *src);
  • Optional lazy configuration/reconfiguration

AVScaleContext *c = avscale_alloc_context();

...

if (c)
    ret = avscale_process_frame(c, dst, src);

...

avscale_free_context(&c);
  • Simplified before-conversion configuration

ret = avscale_config_context(c, dst, src);

while(...) {
...
    ret = avscale_convert_frame(c, dst, src);
... 
}

avscale_free_context(&c);

The conversion context can be configured from the **AVFrame** formaton structure either using a stand-alone function or just passing the freshly allocated context and input and output frames.

  • AVOption based settings

int ret;
AVScaleContext *c = avscale_alloc_context();

av_opt_set_int(c, "scaling-algorithm", AVSCALE_BICUBIC);

ret = avscale_init_context(c);

while (...) {
    ...
    ret = avscale_convert_frame(c, dst, src);
}

avscale_free_context(&c);

It is possible to provide specific settings using the AVOption system. The context itself remains an opaque structure.

  • Support for hardware accelerated scaling through HWContext-awareness.

Architecture Design

libswscale by default tend to merge the scaling and color conversion in a single step and work on a per-slice basis.

libavscale will provide initially only a per-frame external api and the colorspace conversion and scaling will be implemented using a kernel-chain approach.

Conversion steps

AVScale can be modeled as a pipeline of kernels running over the frame data.

Fastpaths will be made available for commonly used conversion the following description is for the generic code.

The process is divided in steps: input layout conversion, input colorspace conversion , scaling, output colorspace conversion, output layout conversion.

Depending on the input and output the layout and colorspace conversion would be skipped.

Layout Conversion

Convert the input data layout to an valid intermediate representation.

Formats such as bayer or nv12 are not well suited for manipulation, thus they will be converted to a per-component representation.

Colorspace Conversion

Convert to an intermediate format fitting the input format gamut.

This step involves non-linear or otherwise formats not suited to a per-component processing.

Scaling

Resize to the required dimension.

Skipped completely if the dimension requested remain the same.

Color Conversion

Color Conversion works on full-frames. The conversion routine is implemented using functions working on tiles.

Scaling

Scaling is implemented per-component and its output should be as close to the target output format as possible.

Accelerated paths

The portable implementation will be written in C, since the process is decoupled and each step has its specific worker-context, tile and frame threading support can be provided easily.

Arch-specific variants can be supported by replacing the C-kernel in the pipeline with the arch-specific one.

Assembly tuned implementation

SWScale structure makes hard to support non-x86 accelerated variants.

AVScale makes as easy as for the rest of Libav leveraging the same framework used by AVCodec.

HWAccel2 integration

HWAccel1 specified that AVFrame could store opaque surfaces that would be up to the user to manage properly its allocation, rendering and disposal.

HWAccel1.2 provides implementation specific setup functions.

HWAccel2 provides default functions, hiding all the complexity by default and providing a default renderer.

AVScale can provide specialized backends to have avoid moving back to system memory if possible (e.g. MediaSDK-decoded AVFrames can be processed using the MediaSDK scaler component)

OpenCL assisted rendering

Most of the libavscale can be modeled through the application of kernels to surfaces, a natural application of OpenCL. Depending on the runtime an opencl implementation would be efficient and straightforward to write thanks to the vector datatypes provided.


CategoryBlueprintActive CategoryBlueprint