Skip to content

feat!: use IDL types in parser#625

Closed
ilbertt wants to merge 43 commits intomasterfrom
luca/idl-types-in-parser
Closed

feat!: use IDL types in parser#625
ilbertt wants to merge 43 commits intomasterfrom
luca/idl-types-in-parser

Conversation

@ilbertt
Copy link
Copy Markdown
Contributor

@ilbertt ilbertt commented Jun 20, 2025

Overview
Refactor the candid_parser and the candid_derive crates to use IDL* types. This allows us to enrich the IDL* types e.g. with comments later on, without affecting serialization and deserialization performance.

Requirements
Blocked by:

Considerations
Breaking change.

@ilbertt ilbertt changed the base branch from master to luca/candid-idl-types June 22, 2025 10:11
@github-actions
Copy link
Copy Markdown

github-actions Bot commented Jun 22, 2025

Name Max Mem (Kb) Encode Decode
blob 4_224 4_206_417 ($\textcolor{green}{-0.00\%}$) 2_121_046 ($\textcolor{green}{-0.00\%}$)
btreemap 75_456 4_261_360_414 ($\textcolor{green}{-0.00\%}$) 15_191_359_407 ($\textcolor{green}{-0.33\%}$)
nns 256 ($\textcolor{red}{100.00\%}$) 1_962_853 ($\textcolor{green}{-0.20\%}$) 5_451_551 ($\textcolor{green}{-0.32\%}$)
nns_list_proposal 1_088 6_793_988 ($\textcolor{green}{-0.30\%}$) 66_773_097 ($\textcolor{green}{-1.62\%}$)
option_list 128 7_474_582 ($\textcolor{green}{-0.01\%}$) 25_414_428 ($\textcolor{green}{-1.41\%}$)
text 6_336 4_203_377 ($\textcolor{green}{-0.00\%}$) 7_877_221 ($\textcolor{green}{-0.00\%}$)
variant_list 128 7_530_885 ($\textcolor{green}{-0.00\%}$) 23_996_664 ($\textcolor{green}{-1.08\%}$)
vec_int16 16_704 123_692_744 ($\textcolor{green}{-0.00\%}$) 1_019_238_900 ($\textcolor{green}{-4.71\%}$)
  • Parser cost: 15_816_101 ($\textcolor{red}{5.43\%}$)
  • Extra args: 3_147_769 ($\textcolor{red}{0.03\%}$)
Click to see raw report
---------------------------------------------------

Benchmark: blob
  total:
    instructions: 6.33 M (-0.00%) (change within noise threshold)
    heap_increase: 66 pages (no change)
    stable_memory_increase: 0 pages (no change)

  1. Encoding (scope):
    calls: 1 (no change)
    instructions: 4.21 M (-0.00%) (change within noise threshold)
    heap_increase: 66 pages (no change)
    stable_memory_increase: 0 pages (no change)

  2. Decoding (scope):
    calls: 1 (no change)
    instructions: 2.12 M (-0.00%) (change within noise threshold)
    heap_increase: 0 pages (no change)
    stable_memory_increase: 0 pages (no change)

---------------------------------------------------

Benchmark: text
  total:
    instructions: 12.08 M (-0.00%) (change within noise threshold)
    heap_increase: 99 pages (no change)
    stable_memory_increase: 0 pages (no change)

  1. Encoding (scope):
    calls: 1 (no change)
    instructions: 4.20 M (-0.00%) (change within noise threshold)
    heap_increase: 66 pages (no change)
    stable_memory_increase: 0 pages (no change)

  2. Decoding (scope):
    calls: 1 (no change)
    instructions: 7.88 M (-0.00%) (change within noise threshold)
    heap_increase: 33 pages (no change)
    stable_memory_increase: 0 pages (no change)

---------------------------------------------------

Benchmark: vec_int16
  total:
    instructions: 1.14 B (improved by 4.22%)
    heap_increase: 261 pages (no change)
    stable_memory_increase: 0 pages (no change)

  1. Encoding (scope):
    calls: 1 (no change)
    instructions: 123.69 M (-0.00%) (change within noise threshold)
    heap_increase: 261 pages (no change)
    stable_memory_increase: 0 pages (no change)

  2. Decoding (scope):
    calls: 1 (no change)
    instructions: 1.02 B (improved by 4.71%)
    heap_increase: 0 pages (no change)
    stable_memory_increase: 0 pages (no change)

---------------------------------------------------

Benchmark: btreemap
  total:
    instructions: 19.45 B (-0.26%) (change within noise threshold)
    heap_increase: 1179 pages (no change)
    stable_memory_increase: 0 pages (no change)

  1. Encoding (scope):
    calls: 1 (no change)
    instructions: 4.26 B (-0.00%) (change within noise threshold)
    heap_increase: 159 pages (no change)
    stable_memory_increase: 0 pages (no change)

  2. Decoding (scope):
    calls: 1 (no change)
    instructions: 15.19 B (-0.33%) (change within noise threshold)
    heap_increase: 1020 pages (no change)
    stable_memory_increase: 0 pages (no change)

---------------------------------------------------

Benchmark: option_list
  total:
    instructions: 32.89 M (-1.09%) (change within noise threshold)
    heap_increase: 2 pages (no change)
    stable_memory_increase: 0 pages (no change)

  1. Encoding (scope):
    calls: 1 (no change)
    instructions: 7.47 M (-0.01%) (change within noise threshold)
    heap_increase: 0 pages (no change)
    stable_memory_increase: 0 pages (no change)

  2. Decoding (scope):
    calls: 1 (no change)
    instructions: 25.41 M (-1.41%) (change within noise threshold)
    heap_increase: 2 pages (no change)
    stable_memory_increase: 0 pages (no change)

---------------------------------------------------

Benchmark: variant_list
  total:
    instructions: 31.53 M (-0.82%) (change within noise threshold)
    heap_increase: 2 pages (no change)
    stable_memory_increase: 0 pages (no change)

  1. Encoding (scope):
    calls: 1 (no change)
    instructions: 7.53 M (-0.00%) (change within noise threshold)
    heap_increase: 0 pages (no change)
    stable_memory_increase: 0 pages (no change)

  2. Decoding (scope):
    calls: 1 (no change)
    instructions: 24.00 M (-1.08%) (change within noise threshold)
    heap_increase: 2 pages (no change)
    stable_memory_increase: 0 pages (no change)

---------------------------------------------------

Benchmark: nns
  total:
    instructions: 24.07 M (regressed by 3.41%)
    heap_increase: 4 pages (regressed by 100.00%)
    stable_memory_increase: 0 pages (no change)

  0. Parsing (scope):
    calls: 1 (no change)
    instructions: 15.82 M (regressed by 5.43%)
    heap_increase: 4 pages (regressed by 100.00%)
    stable_memory_increase: 0 pages (no change)

  1. Encoding (scope):
    calls: 1 (no change)
    instructions: 1.96 M (-0.20%) (change within noise threshold)
    heap_increase: 0 pages (no change)
    stable_memory_increase: 0 pages (no change)

  2. Decoding (scope):
    calls: 1 (no change)
    instructions: 5.45 M (-0.32%) (change within noise threshold)
    heap_increase: 0 pages (no change)
    stable_memory_increase: 0 pages (no change)

---------------------------------------------------

Benchmark: nns_list_proposal
  total:
    instructions: 73.57 M (-1.50%) (change within noise threshold)
    heap_increase: 17 pages (no change)
    stable_memory_increase: 0 pages (no change)

  1. Encoding (scope):
    calls: 1 (no change)
    instructions: 6.79 M (-0.30%) (change within noise threshold)
    heap_increase: 3 pages (no change)
    stable_memory_increase: 0 pages (no change)

  2. Decoding (scope):
    calls: 1 (no change)
    instructions: 66.77 M (-1.62%) (change within noise threshold)
    heap_increase: 14 pages (no change)
    stable_memory_increase: 0 pages (no change)

---------------------------------------------------

Benchmark: extra_args
  total:
    instructions: 3.15 M (0.03%) (change within noise threshold)
    heap_increase: 0 pages (no change)
    stable_memory_increase: 0 pages (no change)

---------------------------------------------------

Summary:
  instructions:
    status:   Regressions and improvements 🔴🟢
    counts:   [total 9 | regressed 1 | improved 1 | new 0 | unchanged 7]
    change:   [max +793.03K | p75 -137 | median -262.23K | p25 -1.12M | min -50.33M]
    change %: [max +3.41% | p75 -0.00% | median -0.26% | p25 -1.09% | min -4.22%]

  heap_increase:
    status:   Regressions detected 🔴
    counts:   [total 9 | regressed 1 | improved 0 | new 0 | unchanged 8]
    change:   [max +2 | p75 0 | median 0 | p25 0 | min 0]
    change %: [max +100.00% | p75 0.00% | median 0.00% | p25 0.00% | min 0.00%]

  stable_memory_increase:
    status:   No significant changes 👍
    counts:   [total 9 | regressed 0 | improved 0 | new 0 | unchanged 9]
    change:   [max 0 | p75 0 | median 0 | p25 0 | min 0]
    change %: [max 0.00% | p75 0.00% | median 0.00% | p25 0.00% | min 0.00%]

---------------------------------------------------

Only significant changes:
| status | name                   | calls |    ins |  ins Δ% |  HI |    HI Δ% | SMI |  SMI Δ% |
|--------|------------------------|-------|--------|---------|-----|----------|-----|---------|
|   +    | nns::0. Parsing        |     1 | 15.82M |  +5.43% |   4 | +100.00% |   0 |   0.00% |
|   +    | nns                    |       | 24.07M |  +3.41% |   4 | +100.00% |   0 |   0.00% |
|   -    | vec_int16              |       |  1.14B |  -4.22% | 261 |    0.00% |   0 |   0.00% |
|   -    | vec_int16::2. Decoding |     1 |  1.02B |  -4.71% |   0 |    0.00% |   0 |   0.00% |

ins = instructions, HI = heap_increase, SMI = stable_memory_increase, Δ% = percent change

---------------------------------------------------
Successfully persisted results to canbench_results.yml

Comment thread rust/candid/src/types/syntax.rs Outdated
@christoph-dfinity
Copy link
Copy Markdown
Contributor

Notes from our 1n1/pairing session earlier:

  • We shouldn't have a KnotT constructor on the syntax IDLType
  • The check_* functions should stay as they are in candid_parser::typing, and we'll want to produce a merged IDLProg by "concatenating" the imported did files (not as text, but by concatenating the defs and service methods). That should also let us not need an extra IDLEnv
  • Let's try to have the derive code operate on the TypeContainer as it used to, and only convert to IDLType at the last moment (attaching the comments we collected along the way).

Copy link
Copy Markdown
Contributor

@christoph-dfinity christoph-dfinity left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looking very good! Spotted a couple places that might need changes.

Comment thread rust/candid_parser/src/bindings/javascript.rs Outdated
Comment thread rust/candid_parser/src/bindings/javascript.rs Outdated
Comment thread rust/candid_parser/src/bindings/javascript.rs Outdated
@ilbertt ilbertt marked this pull request as ready for review June 25, 2025 18:27
@ilbertt ilbertt requested a review from a team as a code owner June 25, 2025 18:27
Base automatically changed from luca/candid-idl-types to master June 25, 2025 18:38
Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This struct and enum were not used

Comment on lines -154 to -176
fn check_cycle(env: &TypeEnv) -> Result<()> {
fn has_cycle<'a>(seen: &mut BTreeSet<&'a str>, env: &'a TypeEnv, t: &'a Type) -> Result<bool> {
match t.as_ref() {
TypeInner::Var(id) => {
if seen.insert(id) {
let ty = env.find_type(id)?;
has_cycle(seen, env, ty)
} else {
Ok(true)
}
}
_ => Ok(false),
}
}
for (id, ty) in env.0.iter() {
let mut seen = BTreeSet::new();
if has_cycle(&mut seen, env, ty)? {
return Err(Error::msg(format!("{id} has cyclic type definition")));
}
}
Ok(())
}

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Moved into a TypeEnv method

Copy link
Copy Markdown
Contributor

@lwshang lwshang left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@ilbertt ilbertt requested a review from lwshang June 30, 2025 19:29
Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We should not change this code, because if we incur in Future or Unknown, this explodes now.

@ilbertt ilbertt marked this pull request as draft July 1, 2025 13:48
christoph-dfinity added a commit that referenced this pull request Jul 8, 2025
**Overview**
Adds support for reflecting doc comments from Candid declaration files
to the Motoko generated bindings.

Internally, it adds the `IDLMergedProg` which is constructed when
parsing the Candid declarations and is passed to the Motoko bindings
generator. It containts the syntax types along with the associated
comments, and it's traversed in parallel with the inner types by the
binding generator.

Follow-up PRs will add support for doc comments in the other bindings
generators.

Closes #625.

---------

Co-authored-by: ilbertt <luca.bertelli@dfinity.org>
@ilbertt ilbertt deleted the luca/idl-types-in-parser branch July 10, 2025 08:11
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants