Rationale

The project aims to replace the software scaling and color conversion library in Libav with an easier to use and easier to extend replacement that leverages the actual hardware available and solves current color conversion problems.

You plan to take out a relic from the original MPlayer and replace it with something fitting the current century.

Current status

API Design

• AVFrame based API

int avscale_process_frame(AVScaleContext *c, AVFrame *dst, AVFrame *src);

• Optional lazy configuration/reconfiguration

AVScaleContext *c = avscale_alloc_context();

...

if (c)
    ret = avscale_process_frame(c, dst, src);

...

avscale_free_context(&c);

• Simplified before-conversion configuration

ret = avscale_config_context(c, dst, src);

while(...) {
...
    ret = avscale_convert_frame(c, dst, src);
... 
}

avscale_free_context(&c);

The conversion context can be configured from the **AVFrame** formaton structure either using a stand-alone function or just passing the freshly allocated context and input and output frames.

• AVOption based settings

int ret;
AVScaleContext *c = avscale_alloc_context();

av_opt_set_int(c, "scaling-algorithm", AVSCALE_BICUBIC);

ret = avscale_init_context(c);

while (...) {
    ...
    ret = avscale_convert_frame(c, dst, src);
}

avscale_free_context(&c);

It is possible to provide specific settings using the AVOption system. The context itself remains an opaque structure.

• Support for hardware accelerated scaling through HWContext-awareness.

Architecture Design

libswscale by default tend to merge the scaling and color conversion in a single step and work on a per-slice basis.

libavscale will provide initially only a per-frame external api and the colorspace conversion and scaling will be implemented using a kernel-chain approach.

Conversion steps

AVScale can be modeled as a pipeline of kernels running over the frame data.

The process is divided in steps: input layout conversion, input colorspace conversion , scaling, output colorspace conversion, output layout conversion.

Depending on the input and output the layout and colorspace conversion would be skipped.

Layout Conversion

Convert the input data layout to an valid intermediate representation.

Formats such as bayer or nv12 are not well suited for manipulation, thus they will be converted to a per-component representation.

Colorspace Conversion

Convert to an intermediate format fitting the input format gamut.

This step involves non-linear or otherwise formats not suited to a per-component processing.

Scaling

Resize to the required dimension.

Skipped completely if the dimension requested remain the same.

Color Conversion

Color Conversion works on full-frames. The conversion routine is implemented using functions working on tiles.

Scaling

Scaling is implemented per-component and its output should be as close to the target output format as possible.

Accelerated paths

The portable implementation will be written in C, since the process is decoupled and each step has its specific worker-context, tile and frame threading support can be provided easily.

Arch-specific variants can be supported by replacing the C-kernel in the pipeline with the arch-specific one.

Assembly tuned implementation

SWScale structure makes hard to support non-x86 accelerated variants.

AVScale makes as easy as for the rest of Libav leveraging the same framework used by AVCodec.

HWAccel2 integration

HWAccel1 specified that AVFrame could store opaque surfaces that would be up to the user to manage properly its allocation, rendering and disposal.

HWAccel1.2 provides implementation specific setup functions.

HWAccel2 provides default functions, hiding all the complexity by default and providing a default renderer.

AVScale can provide specialized backends to have avoid moving back to system memory if possible (e.g. MediaSDK-decoded AVFrames can be processed using the MediaSDK scaler component)

OpenCL assisted rendering

Most of the libavscale can be modeled through the application of kernels to surfaces, a natural application of OpenCL. Depending on the runtime an opencl implementation would be efficient and straightforward to write thanks to the vector datatypes provided.

CategoryBlueprintActive CategoryBlueprint