FATE is a Fully Automated Test Enviroment test suite for Libav.
Introduction
Due to the complexity of code in Libav it is necessary to continuously check that new code does not break existing behaviour and to do so a whole set of videos, filters, formats and other operations are feed into integrated testing programs.
Any substantial code submitted to Libav should bear some test capabilities.
Using FATE
After compiling Libav, the fate tests can be run with make fate, which runs the tests that do not need external samples. To run all tests, use
make fate-rsync SAMPLES=/path/to/the/samples
make fate SAMPLES=/path/to/the/samples
make fate-list # list all the targets
make fate-h264 # run all the h264 fate
make -j 18 fate-h264 # run the tests in parallel
See the FATE documentation for more details.
FATE samples upload
Developers have write access to fate-suite through sftp or rsync with the same SSH key as for git push access. The location of fate-suite is horace@fate.libav.org on port 2222 in the folder fate-suite/
When using rsync to upload samples it's advised to sync only a partial tree and check with -n or --dry-run that only the intended files are uploaded. Files should have 644 permissions and directories 755.
If you do not have write access and need to have some samples uploaded the quickest way is to get in touch with developer who can, usually identified by the @ in the IRC channel.
Decoder testing
Depending on the complexity of the decoder you might want to add a specific Makefile fragment file named after your decoder. This file should list all the samples under test and all the options needed for decoding: don't forget to submit your samples to FATE server 24h before applying the patch so that mirrors are able to pick them up.
After setting up the main reference file (tests/fate/NAME.mak) and instructed the FATE Makefile (tests/Makefile) to use it, you have to add the reference file for each of your sample (under tests/ref/fate/).
Example commit e1c058dba930c1f6f180e04d9e26dbda261cedbc
In order to make the tests on big endian architectures return the same data you must pin the output format.
Encoder testing
Container testing
Filter testing
For a filter test you only need to check the result of your filter given a known input, and there is no need to submit any sample.
Normally you have to edit tests/fate/filter-video.mak and add the right configuration options for your filter, as well as adding a reference file in tests/ref/fate/filter-NAME
Example commit: 157dc0b489d472e4fc6eca7a2eb92b847f5b88dd
Note
Be very careful when swscale is involved (e.g. for a pixel format conversion). Remember to always add -sws_flags +accurate_rnd+bitexact otherwise tests may fail on a different architecture.
Updating tests
Sometimes code changes and results of the test are legitimately different. When this happens it is sufficient to run the following command to amend the ref files.
make GEN=1 fate
Some test with large reference files e.g. audio resampling tests, keep them in the rsync directory. Make sure to generate a second version of the reference and not rsync overwriting the previous.
Types of comparisons
There are multiple to compare results, sometimes a test might give different valid results across different arches. The make variable to specify the comparator is CMP.
- 'diff', the default comparator, check for bit-exactness the test output.
'oneoff' accepts as valid results that are off by a specific amount signaled using the uses the FUZZ variable. Tests involving floating point should use it since, the results will not be identical across all platforms.
The other types of CMP are oneline, stddev, and null. Please document them.
Known failures
There's a list of in certain configurations, see KnownFateFailures.