FATE is a Fully Automated Test Enviroment test suite for Libav.

Introduction

Due to the complexity of code in Libav it is necessary to continuously check that new code does not break existing behaviour and to do so a whole set of videos, filters, formats and other operations are feed into integrated testing programs.

Any substantial code submitted to Libav should bear some test capabilities.

Using FATE

After compiling Libav, the fate tests can be run with make fate, which runs the tests that do not need external samples. To run all tests, use

make fate-rsync SAMPLES=/path/to/the/samples
make fate SAMPLES=/path/to/the/samples
make fate-list # list all the targets
make fate-h264 # run all the h264 fate
make -j 18 fate-h264 # run the tests in parallel

See the FATE documentation for more details.

FATE samples upload

Developers have write access to fate-suite through sftp or rsync with the same SSH key as for git push access. The location of fate-suite is horace@fate.libav.org on port 2222 in the folder fate-suite/

When using rsync to upload samples it's advised to sync only a partial tree and check with -n or --dry-run that only the intended files are uploaded. Files should have 644 permissions and directories 755.

If you do not have write access and need to have some samples uploaded the quickest way is to get in touch with developer who can, usually identified by the @ in the IRC channel.

Decoder testing

Depending on the complexity of the decoder you might want to add a specific Makefile fragment file named after your decoder. This file should list all the samples under test and all the options needed for decoding: don't forget to submit your samples to FATE server 24h before applying the patch so that mirrors are able to pick them up.

After setting up the main reference file (tests/fate/NAME.mak) and instructed the FATE Makefile (tests/Makefile) to use it, you have to add the reference file for each of your sample (under tests/ref/fate/).

Example commit e1c058dba930c1f6f180e04d9e26dbda261cedbc

In order to make the tests on big endian architectures return the same data you must pin the output format.


Encoder testing


Container testing


Filter testing

For a filter test you only need to check the result of your filter given a known input, and there is no need to submit any sample.

Normally you have to edit tests/fate/filter-video.mak and add the right configuration options for your filter, as well as adding a reference file in tests/ref/fate/filter-NAME

Example commit: 157dc0b489d472e4fc6eca7a2eb92b847f5b88dd

Note

Be very careful when swscale is involved (e.g. for a pixel format conversion). Remember to always add -sws_flags +accurate_rnd+bitexact otherwise tests may fail on a different architecture.


Updating tests

Sometimes code changes and results of the test are legitimately different. When this happens it is sufficient to run the following command to amend the ref files.

make GEN=1 fate

Some test with large reference files e.g. audio resampling tests, keep them in the rsync directory. Make sure to generate a second version of the reference and not rsync overwriting the previous.

Types of comparisons

There are multiple to compare results, sometimes a test might give different valid results across different arches. The make variable to specify the comparator is CMP.

The other types of CMP are oneline, stddev, and null. Please document them.

Known failures

There's a list of in certain configurations, see KnownFateFailures.

CategoryHowTo CategoryWIP