Run latest Wycheproof tests every night#5269
Open
reneme wants to merge 9 commits intorandombit:masterfrom
Open
Run latest Wycheproof tests every night#5269reneme wants to merge 9 commits intorandombit:masterfrom
reneme wants to merge 9 commits intorandombit:masterfrom
Conversation
Owner
|
Don't have time to review right now but good idea. A not entirely current set of wycheproof test vectors are tested in botan-rs against wycheproof-rs but CI runs there is a bit sporadic... |
a1f0b9d to
5bcaed6
Compare
Collaborator
Author
|
Marked this as ready-for-review so that Copilot has a look. We could work on extending this further in this PR or merge and then do follow-ups. Both are fine from my perspective, because it's just adding scripts anyway. |
5bcaed6 to
cc14586
Compare
cc14586 to
9ad00db
Compare
This downloads the latest wycheproof test vectors from C2SP/wycheproof and uses Botan's python wrapper to feed the test vectors into our implementations.
9ad00db to
2c8a799
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
These are some (fairly heavily vibe-coded) test scripts that download the latest test vectors from https://github.com/C2SP/wycheproof every night and run our implementations against it (through the Python wrapper). This complements the wycheproof test data that we already pulled into our unit test harness by running the full vectors from upstream.
Apart from extending the scope of tested algorithms, it would be really neat if we could make this CPUID aware so that the test vectors are run against all available accelerator implementations.
Currently this is manually formatted using ruff to be compliant once #5096 lands.
TODO
Not necessarily all in this PR. Also potentially more, see: testvectors_v1.
These tests are fast. On my local machine they currently run in about 5 seconds (when the test data vectors are cached).
Usage
The
WYCHEPROOF_TESTDATA_CACHE_DIRis optional and should be used only for local testing/debugging. That way, the tests aren't always re-downloading the test vectors from GitHub which makes the execution much faster. In CI we don't want to have this cache, because we want to always pull in the latest vectors from Wycheproof.Pull Request dependencies
PublicKey.used_explicit_encoding()in python wrapper #5282