This page is documenting a work in progress feature do not assume it is available on the mainline branch. You may find it developed on a topic branch
This development has a fundraiser open.
Rationale
The project aims to replace the software scaling and color conversion library in Libav with an easier to use and easier to extend replacement that leverages the actual hardware available and solves current color conversion problems.
You plan to take out a relic from the original MPlayer and replace it with something fitting the current century.
Current status
The software scaling library currently provided originates as a tightly coupled part of MPlayer, thus having a quite peculiar API and a quite convoluted internal structure.- high quality conversion of high-bitdepth color formats (e.g. 10-bit YUV422 or 12-bit XYZ)
- an easy way to use hardware accelerated scaling
- support for every theoretically possible format (like YUV colorspace with different subsampling for chroma components and uncommon bitdepth) and more.
API Design
- AVFrame based API
int avscale_process_frame(AVScaleContext *c, AVFrame *dst, AVFrame *src);
- Optional lazy configuration/reconfiguration
AVScaleContext *c = avscale_alloc_context();
...
if (c)
ret = avscale_process_frame(c, dst, src);
...
avscale_free_context(&c);
- Simplified before-conversion configuration
ret = avscale_config_context(c, dst, src);
while(...) {
...
ret = avscale_convert_frame(c, dst, src);
...
}
avscale_free_context(&c);
The conversion context can be configured from the **AVFrame** formaton structure either using a stand-alone function or just passing the freshly allocated context and input and output frames.
- AVOption based settings
int ret;
AVScaleContext *c = avscale_alloc_context();
av_opt_set_int(c, "scaling-algorithm", AVSCALE_BICUBIC);
ret = avscale_init_context(c);
while (...) {
...
ret = avscale_convert_frame(c, dst, src);
}
avscale_free_context(&c);
It is possible to provide specific settings using the AVOption system. The context itself remains an opaque structure.
- Support for hardware accelerated scaling through HWContext-awareness.
Architecture Design
libswscale by default tend to merge the scaling and color conversion in a single step and work on a per-slice basis.
libavscale will provide initially only a per-frame external api and the colorspace conversion and scaling will be implemented using a kernel-chain approach.
Conversion steps
AVScale can be modeled as a pipeline of kernels running over the frame data.
Fastpaths will be made available for commonly used conversion the following description is for the generic code.
The process is divided in steps: input layout conversion, input colorspace conversion , scaling, output colorspace conversion, output layout conversion.
Depending on the input and output the layout and colorspace conversion would be skipped.
Layout Conversion
Convert the input data layout to an valid intermediate representation.
Formats such as bayer or nv12 are not well suited for manipulation, thus they will be converted to a per-component representation.
Colorspace Conversion
Convert to an intermediate format fitting the input format gamut.
This step involves non-linear or otherwise formats not suited to a per-component processing.
Scaling
Resize to the required dimension.
Skipped completely if the dimension requested remain the same.
Color Conversion
Color Conversion works on full-frames. The conversion routine is implemented using functions working on tiles.
Scaling
Scaling is implemented per-component and its output should be as close to the target output format as possible.
Accelerated paths
The portable implementation will be written in C, since the process is decoupled and each step has its specific worker-context, tile and frame threading support can be provided easily.
Arch-specific variants can be supported by replacing the C-kernel in the pipeline with the arch-specific one.
Assembly tuned implementation
SWScale structure makes hard to support non-x86 accelerated variants.
AVScale makes as easy as for the rest of Libav leveraging the same framework used by AVCodec.
HWAccel2 integration
HWAccel1 specified that AVFrame could store opaque surfaces that would be up to the user to manage properly its allocation, rendering and disposal.
HWAccel1.2 provides implementation specific setup functions.
HWAccel2 provides default functions, hiding all the complexity by default and providing a default renderer.
AVScale can provide specialized backends to have avoid moving back to system memory if possible (e.g. MediaSDK-decoded AVFrames can be processed using the MediaSDK scaler component)
OpenCL assisted rendering
Most of the libavscale can be modeled through the application of kernels to surfaces, a natural application of OpenCL. Depending on the runtime an opencl implementation would be efficient and straightforward to write thanks to the vector datatypes provided.
CategoryBlueprintActive CategoryBlueprint
Contents