This page is documenting a work in progress feature do not assume it is available on the mainline branch.

You may find it developed on a topic branch


Our current Hardware Acceleration support is lacking and unwieldy:

  • Assumes slice-based decoding
    • Many hw decoders are bitstream based, reformatting twice is a bit of a waste but makes falling back simpler.
  • Requires special setup, glue and render code written from scratch by the user
    • All this boilerplate can be done without in most cases.
  • Uses context struct that might change requiring an ABI break.
    • Being solved currently by providing a separate allocator per hwaccel implementation.
  • Use special pixel formats to signal that the data in the AVFrame is opaque, but its semantics are not well defined nor documented.
    • Currently only the third pointer is defined usually, in some cases it isn't even refcounted properly.
  • Requires to override directly get_buffer2 callback.

The following proposal aims at providing a simpler access to hardware acceleration, a more regular interface and provide "under the hood" access ONLY when needed.

API Design

The new api tries to stay fully compatible with the previous one while moving in Libav the boilerplate code needed to setup the global context and render the opaque frames back to memory.

Simplified usage

Some users do not have the need to use opaque AVFrames and render/scale/deinterlace on screen. In that case setting the codec AVOption hwaccel to the specific method and hwaccel-device to the specific device is all that's needed:

  • No specific in-application setup is required
  • No additional knowledge about the backend-specific buffer structure is needed
  • The resulting frames from avcodec_decode_video2 get rendered to system memory automatically.

  • The speed/power efficiency gain is limited.


  • A new get_hw_surface callback is provided to provide backend-specific hardware surfaces, get_buffer2 does not change semantics depending if hwaccel is in use or not.

  • Functions to hand over the ownership of memory buffers and device are provided to let advanced user have full control over the decode->rendering chain without having to keep the AVFrame and AVCodecContext wrapping in their application specific logic.

  • avcodec_decode* will return a specific AVERROR to signal when the hwaccel is not supporting a format and a reconfiguration is required.

Utility functions

  • A mean to enumerate the available hwaccel backends and devices will be provided

There is no need to provide custom callbacks nor initialize by hand the backend-specific context, yet is possible to override the defaults after avcodec_open2.

AVScale will be able manipulate directly opaque AVFrames using hw accelerated scaling/conversion if available.

// allocate an empty hwaccel context
c->hwctx = avhwaccel_alloc();

// common options
av_opt_set(c->hwctx, "hwaccel", "mfx", 0);
av_opt_set(c->hwctx, "hwaccel-device", "/dev/drm/card0", 0);
av_opt_set_int(c->hwctx, "hwaccel-render", 1, 0); // If left as 0 the AVFrame will use a specific custom pixel format

//  initialize the hwaccel global context using the options stored in hwctx and the codec parameters
if (avcodec_open2(c, codec, opts) < 0) 

Functions in detail



CPU memory rendering

GPU buffer handover

Device handover

Architecture Design


Internal helpers

AVScale Interaction

AVScale should able to accept opaque frames and render them to system memory leveraging the hardware accelerated scalers if present.

This way the frame could ideally not move from the video memory at all.

CategoryBlueprint CategoryBlueprintActive