Skip to content

Comments

Run latest Wycheproof tests every night#5269

Open
reneme wants to merge 9 commits intorandombit:masterfrom
Rohde-Schwarz:test/wycheproof_nightly
Open

Run latest Wycheproof tests every night#5269
reneme wants to merge 9 commits intorandombit:masterfrom
Rohde-Schwarz:test/wycheproof_nightly

Conversation

@reneme
Copy link
Collaborator

@reneme reneme commented Jan 28, 2026

These are some (fairly heavily vibe-coded) test scripts that download the latest test vectors from https://github.com/C2SP/wycheproof every night and run our implementations against it (through the Python wrapper). This complements the wycheproof test data that we already pulled into our unit test harness by running the full vectors from upstream.

Apart from extending the scope of tested algorithms, it would be really neat if we could make this CPUID aware so that the test vectors are run against all available accelerator implementations.

Currently this is manually formatted using ruff to be compliant once #5096 lands.

TODO

  • AEAD
  • KeyWrap
  • ECDH
  • ECDSA
  • EdDSA
  • HKDF
  • MAC
  • ML-DSA
  • ML-KEM
  • RSA Encryption
  • RSA Signing
  • X25519/448

Not necessarily all in this PR. Also potentially more, see: testvectors_v1.

These tests are fast. On my local machine they currently run in about 5 seconds (when the test data vectors are cached).

Usage

# build the shared object
./configure.py --without-documentation --compiler-cache=ccache --build-targets=shared --build-tool=ninja
ninja libs

# run the wycheproof tests
cd src/scripts/wycheproof
LD_LIBRARY_PATH=../../.. \
PYTHONPATH=../../python \
WYCHEPROOF_TESTDATA_URL="https://raw.githubusercontent.com/C2SP/wycheproof/refs/heads/main/testvectors_v1" \
WYCHEPROOF_TESTDATA_CACHE_DIR=/tmp/wycheproof_cache \
python3 -m unittest

The WYCHEPROOF_TESTDATA_CACHE_DIR is optional and should be used only for local testing/debugging. That way, the tests aren't always re-downloading the test vectors from GitHub which makes the execution much faster. In CI we don't want to have this cache, because we want to always pull in the latest vectors from Wycheproof.

Pull Request dependencies

@reneme reneme requested a review from randombit January 28, 2026 11:47
@reneme reneme self-assigned this Jan 28, 2026
@coveralls
Copy link

coveralls commented Jan 28, 2026

Coverage Status

coverage: 90.073% (+0.002%) from 90.071%
when pulling 2c8a799 on Rohde-Schwarz:test/wycheproof_nightly
into 28378fa on randombit:master.

@randombit
Copy link
Owner

Don't have time to review right now but good idea. A not entirely current set of wycheproof test vectors are tested in botan-rs against wycheproof-rs but CI runs there is a bit sporadic...

@reneme reneme force-pushed the test/wycheproof_nightly branch 3 times, most recently from a1f0b9d to 5bcaed6 Compare February 4, 2026 12:53
@reneme reneme marked this pull request as ready for review February 4, 2026 12:54
Copilot AI review requested due to automatic review settings February 4, 2026 12:54
@reneme
Copy link
Collaborator Author

reneme commented Feb 4, 2026

Marked this as ready-for-review so that Copilot has a look. We could work on extending this further in this PR or merge and then do follow-ups. Both are fine from my perspective, because it's just adding scripts anyway.

This comment was marked as resolved.

This comment was marked as resolved.

This comment was marked as resolved.

@reneme reneme force-pushed the test/wycheproof_nightly branch from 9ad00db to 2c8a799 Compare February 5, 2026 10:04
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants