-
Notifications
You must be signed in to change notification settings - Fork 0
spec: update constraint ID system #570
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Open
erik-3milabs
wants to merge
19
commits into
spec/main
Choose a base branch
from
spec/constraint_id
base: spec/main
Could not load branches
Branch not found: {{ refName }}
Loading
Could not load tags
Nothing to show
Loading
Are you sure you want to change the base?
Some commits from the old base branch may be removed from the timeline,
and old review comments may become outdated.
Open
Changes from all commits
Commits
Show all changes
19 commits
Select commit
Hold shift + click to select a range
2074516
spec/constraint_id: introduce FNV-1a hash
erik-3milabs a3e5950
spec/constraint_id: compute constraint ID upon load
erik-3milabs acddf3a
spec/constraint_id: attempt at new id
erik-3milabs 58ac836
spec: drop "constraint" from "polynomial constraint"
erik-3milabs b635dbd
spec/constraint_id: refactor ID
erik-3milabs 8903751
spec: completely hide unused constraint table columns
erik-3milabs 37468f1
spec: squish column table iter notation
erik-3milabs 08fab18
spec: reduce some constraint table column widths
erik-3milabs 65bd648
spec: introduce thin lines between constraints
erik-3milabs d4680af
spec: use chip codes as shorthand in constraint notation
erik-3milabs 331547f
spec: fix tooling
erik-3milabs 2d989fe
spec: update assumption naming
erik-3milabs b0f2694
spec/chip: fix missing sum defaults
erik-3milabs 35bc74b
spec: fix z-fill bug
erik-3milabs 764c8d7
spec/constraint_id: use \x00 as domain separator
erik-3milabs 9f6a43a
spec/chip: fix reading "code"
erik-3milabs b824b4b
spec/constraint_id: include `tag` in constraint-ID derivation
erik-3milabs 95216ec
spec/constraint_id: support indexing for chips beyond 99 constraints
erik-3milabs a27e78f
spec/constraint_id: drop bytes-to-hex
erik-3milabs File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -115,12 +115,158 @@ | |
| } | ||
| } | ||
|
|
||
| /// Fowler-Noll-Vo (FNV) hash function, version 1a | ||
| /// Src: https://en.wikipedia.org/wiki/Fowler-Noll-Vo_hash_function | ||
| /// | ||
| /// Note: this is a non-cryptographic hash function; it is optimized | ||
| /// for speed at the expense of unpredictability. | ||
| /// | ||
| /// This implementation operates on two 32-bit limbs, rather than a single | ||
| /// 64-bit limb, since Typst does not support u64s. | ||
| #let FNV-1a(bytes) = { | ||
| // FNV_prime := 0x00000100000001B3 | ||
| let prime = (0x000001B3, 0x00000100) | ||
|
|
||
| // hash := FNV_offset_basis = 0xCBF29CE484222325 | ||
| let hash = (0x84222325, 0xCBF29CE4) | ||
| for b in bytes { | ||
| // hash := hash XOR byte_of_data | ||
| hash.at(0) = hash.at(0).bit-xor(b) | ||
|
|
||
| // hash := hash × FNV_prime | ||
| let lo = hash.at(0) * prime.at(0) | ||
| let hi = hash.at(0) * prime.at(1) + hash.at(1) * prime.at(0) | ||
|
|
||
| // Carry result | ||
| let carry = lo.bit-rshift(32) | ||
| let lo = lo.bit-and(0xFFFFFFFF) | ||
| let hi = (hi + carry).bit-and(0xFFFFFFFF) | ||
| hash = (lo, hi) | ||
| } | ||
|
|
||
| hash.map(int.to-bytes).join() | ||
| } | ||
|
|
||
| /// Converts a byte array to a hexadecimal string | ||
| #let bytes-to-hex(bytes) = { | ||
|
Collaborator
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Is there much need to carry hex around instead of going straight from bytes to base 32?
Collaborator
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. not really. dropped it |
||
| /// Pads a string with 0s on the left to reach a certain length | ||
| let z-fill(str) = "0" * calc.max(2 - str.len(), 0) + str | ||
|
|
||
| array(bytes) | ||
| .map(b => str(b, base: 16)) | ||
| .map(z-fill) | ||
| .sum() | ||
| } | ||
|
|
||
| /// Tag constraints with an identifier | ||
| #let _add_constraint_ids(chip) = { | ||
|
|
||
| /// A NON-CRYPTOGRAPHIC hash function. | ||
| let nchf(str) = FNV-1a(bytes(str)) | ||
|
|
||
| // number of characters in constraint ID | ||
| let CONSTRAINT_ID_CHAR_COUNT = 4; | ||
|
|
||
| // Map hash digest to ID | ||
| let digest_to_id(hash_bytes) = { | ||
| // Character set used to represent ID | ||
| let CHARS = "123456789ABDEFGHJKLMNPQRSTUVWXYZ".codepoints() | ||
| assert(CHARS.len() == 32, message: "invalid CHARS length") | ||
|
|
||
| let min_bytes_len = 2 * CONSTRAINT_ID_CHAR_COUNT | ||
| assert( | ||
| hash_bytes.len() >= min_bytes_len, | ||
| message: "too few bytes to digest: " + repr(hash_bytes) + " has " + str(hash_bytes.len()) + " where " + str(min_bytes_len) + " is required." | ||
| ) | ||
|
|
||
| let int = int.from-bytes(hash_bytes.slice(0, count: min_bytes_len)) | ||
| for _ in range(CONSTRAINT_ID_CHAR_COUNT) { | ||
| let idx = int.bit-and(31) | ||
| int = int.bit-rshift(5) | ||
| (CHARS.at(idx), ) | ||
| }.sum() | ||
| } | ||
|
|
||
| /// Digests a variable based on its location and type. | ||
| let digest_variable(chip, group, idx, var) = { | ||
| /// Flatten the type of a variable into a string | ||
| let flatten_vartype(typ) = { | ||
| if type(typ) == array { | ||
| "(" + typ.map(flatten_vartype).join(",") + ")" | ||
| } else { | ||
| str(typ) | ||
| } | ||
| } | ||
|
|
||
| let flattened_type = lower(flatten_vartype(var.type)) | ||
| let input = (chip, group, str(idx), flattened_type).join("\x00") | ||
| digest_to_id(nchf(input)) | ||
| } | ||
|
|
||
| // Map variables to their ID | ||
| let variable_to_ID = chip | ||
| .variables | ||
| .pairs() | ||
| .map(((group, variables)) => { | ||
| variables | ||
| .enumerate() | ||
| .map(((idx, var)) => { | ||
| (var.name: digest_variable(chip.name, group, idx, var)) | ||
| }).sum(default: (:)) | ||
| }).sum(default: (:)) | ||
|
|
||
| // replace variable with ID in LISP | ||
| let replace_variable_with_ID(lisp) = { | ||
| if type(lisp) == array { | ||
| "(" + lisp.map(replace_variable_with_ID).join(",") + ")" | ||
| } else { | ||
| variable_to_ID.at(str(lisp), default: str(lisp)) | ||
| } | ||
| } | ||
|
|
||
| // Replace variable names with their ID | ||
| let digestable_constraint(c) = { | ||
| let CONSTRAINT_CAT_TO_SCOPE = ( | ||
|
erik-3milabs marked this conversation as resolved.
|
||
| "interaction": ("tag", "iter", "input", "output", "multiplicity"), | ||
| "template": ("tag", "iter", "input", "output", "cond"), | ||
| "arith": ("iter", "poly") | ||
| ) | ||
|
|
||
| assert(c.kind in CONSTRAINT_CAT_TO_SCOPE) | ||
| let id_tagged = CONSTRAINT_CAT_TO_SCOPE | ||
| .at(c.kind) | ||
| .filter(cat => cat in c.keys()) | ||
| .map(cat => (str(cat): replace_variable_with_ID(c.at(cat)))) | ||
| .sum(default: (:)) | ||
|
|
||
| repr(id_tagged) | ||
| .replace("\n", "") | ||
| .replace(" ", "") | ||
| } | ||
|
|
||
| // Add an ID to each constraint | ||
| chip.constraints = chip.at("constraints", default: (:)) | ||
| .pairs() | ||
| .map(((group, constraints)) => { | ||
| ( | ||
| str(group): | ||
| constraints | ||
| .map(c => { | ||
| c.id = digest_to_id(nchf(digestable_constraint(c))) | ||
| c | ||
| }) | ||
| ) | ||
| }).sum(default: (:)) | ||
|
|
||
| chip | ||
| } | ||
|
|
||
| /// Load a chip object from file | ||
| /// | ||
| /// - path(str): path to file containing chip data | ||
| /// - config: configuration data this chip needs to match with | ||
| #let load_chip(path, config) = { | ||
| let chip = toml(path) | ||
| _check_chip(chip, config) | ||
| return chip | ||
| return _add_constraint_ids(chip) | ||
| } | ||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -1,4 +1,5 @@ | ||
| name = "BRANCH" | ||
| code = "BRH" | ||
|
|
||
|
|
||
| # Input | ||
|
|
||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -1,4 +1,5 @@ | ||
| name = "COMMIT" | ||
| code = "CMT" | ||
|
|
||
| # Variables | ||
|
|
||
|
|
||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -1,4 +1,5 @@ | ||
| name = "KECCAK" | ||
| code = "KCK" | ||
|
|
||
| [[variables.input]] | ||
| name = "timestamp" | ||
|
|
||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -1,4 +1,5 @@ | ||
| name = "KECCAK_RC" | ||
| code = "KCC" | ||
|
|
||
| [[variables.input]] | ||
| name = "round" | ||
|
|
||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -1,4 +1,5 @@ | ||
| name = "KECCAK_RND" | ||
| code = "KCR" | ||
|
|
||
| [[variables.input]] | ||
| name = "timestamp" | ||
|
|
||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -1,4 +1,5 @@ | ||
| name = "LOAD" | ||
| code = "LD" | ||
|
|
||
| # Input | ||
|
|
||
|
|
||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -1,4 +1,5 @@ | ||
| name = "MEMW" | ||
| code = "MMW" | ||
|
|
||
| # Input | ||
|
|
||
|
|
||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -1,4 +1,5 @@ | ||
| name = "MEMW_A" | ||
| code = "MWA" | ||
|
|
||
| # Input | ||
|
|
||
|
|
||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -1,4 +1,5 @@ | ||
| name = "MEMW_R" | ||
| code = "MWR" | ||
|
|
||
| # Variables | ||
|
|
||
|
|
||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -1,4 +1,5 @@ | ||
| name = "ROTXOR" | ||
| code = "RTXR" | ||
|
|
||
| [[variables.input]] | ||
| name = "a" | ||
|
|
||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -1,4 +1,5 @@ | ||
| name = "SHA256" | ||
| code = "SHA" | ||
|
|
||
| [[variables.input]] | ||
| name = "timestamp" | ||
|
|
||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -1,4 +1,5 @@ | ||
| name = "SHA256_K" | ||
| code = "SHK" | ||
|
|
||
| [[variables.input]] | ||
| name = "index" | ||
|
|
||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -1,4 +1,5 @@ | ||
| name = "SHA256MSGSCHED" | ||
| code = "SHM" | ||
|
|
||
| [[variables.input]] | ||
| name = "timestamp" | ||
|
|
||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -1,4 +1,5 @@ | ||
| name = "SHA256ROUND" | ||
| code = "SHR" | ||
|
|
||
| [[variables.input]] | ||
| name = "timestamp" | ||
|
|
||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -1,4 +1,5 @@ | ||
| name = "SHIFT" | ||
| code = "SHF" | ||
|
|
||
| # Input | ||
|
|
||
|
|
||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -1,4 +1,5 @@ | ||
| name = "SIGN" | ||
| code = "SGN" | ||
|
|
||
| [[variables.input]] | ||
| name = "X" | ||
|
|
||
Oops, something went wrong.
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Any reasoning for not using a typst package for hashing?
e.g. digestify should be running in wasm, so speed should not be a massive concern (probably)
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
digestifyis available under the MIT license. When I skimmed the license last week, I thought that using it might require our spec to also be released under the MIT (or compatible) license. Having a closer look now, this might not actually be the case. Still, I'm not a legal expert, so I decided to avoid it just in case.jumble) is slower than this implementation.