Testing
This file describes the testing methodology used in SVS and includes documentation on how update reference results for integration tests.
The test suite defined in the tests
directory are broken into two categories: unit tests that test individual components and integration tests which exercise more cross-unit functionality.
The latter includes index searching and index construction.
Now, we cannot necessarily make our integration tests exhaustive for a number of reasons.
First, integration tests should have high coverage of the many different combinations of indexes, distance functions, compile time optimizations, etc., each of which takes time.
Second, these integration tests must all reference the same binary dataset to minimize the repository size.
This dataset may not be appropriate for all distance functions.
Hence, integration tests compare against reference behavior (i.e., recall for a given configuration generated at some point in the past) and fail if the observed behavior deviates too far. These reference recalls are generated by executables in the benchmarking framework.
See the section on building the library for CMake options related to building integration tests into the test executable (SVS_FORCE_INTEGRATION_TESTS
) and building reference result test generators into the benchmarking executable (SVS_BUILD_BENCHMARK_TEST_GENERRATORS
).
Generating Test Inputs
Complete scripts for compiling, executing, and updating reference inputs are collected here.
Vamana
SVS_NUM_THREADS=10
CC=gcc-11
C++=g++-11
mkdir build && cd build
cmake .. -DCMAKE_BUILD_TYPE=Release \
-DSVS_BUILD_BENCHMARK_TEST_GENERATORS=YES
make -j
# Generate the expected results.
./benchmark/svs_benchmark vamana_test_generator \
../tools/benchmark_inputs/vamana-test-generator.toml \
./vamana_reference.toml \
${SVS_NUM_THREADS} \
../data/test_dataset
# After checking that the results look good, update the reference file.
cp ./vamana_reference.toml ../data/test_dataset/vamana_reference.toml