📝 Add docstrings to codex/build-differential-datalog-tokenizer#8
Open
coderabbitai[bot] wants to merge 3 commits intomainfrom
Open
Conversation
Docstrings generation was requested by @leynos. * #7 (comment) The following files were modified: * `src/tokenizer.rs` * `tests/tokenizer.rs`
Contributor
Reviewer's GuideAdds comprehensive docstrings to both the tokenizer implementation and its tests, clarifying function behavior, expected inputs/outputs, and examples for fixtures, test cases, and core API functions. File-Level Changes
Tips and commandsInteracting with Sourcery
Customizing Your ExperienceAccess your dashboard to:
Getting Help
|
Contributor
Author
|
Important Review skippedCodeRabbit bot authored PR detected. To trigger a single review, invoke the You can disable this status message by setting the 🪧 TipsChatThere are 3 ways to chat with CodeRabbit:
SupportNeed help? Create a ticket on our support page for assistance with any issues or questions. Note: Be mindful of the bot's finite context window. It's strongly recommended to break down tasks such as reading entire modules into smaller chunks. For a focused discussion, use review comments to chat about specific files and their changes, instead of using the PR comments. CodeRabbit Commands (Invoked using PR comments)
Other keywords and placeholders
CodeRabbit Configuration File (
|
Base automatically changed from
codex/build-differential-datalog-tokenizer
to
main
June 25, 2025 02:07
leynos
added a commit
that referenced
this pull request
Feb 28, 2026
This change closes conformance gap #8 by explicitly marking top-level `for` loops as unsupported in the parser. When a `for` keyword appears at the top-level (i.e., at line-start), the span scanner emits a clear diagnostic error "top-level `for` is not supported; use `for` inside rule bodies instead" and consumes the token without producing a rule span. Rule-body `for` loops remain fully supported unchanged as `Expr::ForLoop`. Documentation in section 6.5 of the syntax spec, parser conformance register item 8, parser implementation notes, and roadmap items 2.5.4 & 2.6.1 are updated to reflect this decision. New unit tests and behavioural tests verify that the diagnostic is emitted correctly and no erroneous rule spans are produced for top-level `for`. Quality gates pass. This decision defers implementing `convertStatement` desugaring due to incomplete specification to avoid semantic divergence and can be revisited in future parser generations. Co-authored-by: devboxerhub[bot] <devboxerhub[bot]@users.noreply.github.com>
leynos
added a commit
that referenced
this pull request
Mar 4, 2026
) * feat(parser): mark top-level `for` as unsupported and emit diagnostic This change closes conformance gap #8 by explicitly marking top-level `for` loops as unsupported in the parser. When a `for` keyword appears at the top-level (i.e., at line-start), the span scanner emits a clear diagnostic error "top-level `for` is not supported; use `for` inside rule bodies instead" and consumes the token without producing a rule span. Rule-body `for` loops remain fully supported unchanged as `Expr::ForLoop`. Documentation in section 6.5 of the syntax spec, parser conformance register item 8, parser implementation notes, and roadmap items 2.5.4 & 2.6.1 are updated to reflect this decision. New unit tests and behavioural tests verify that the diagnostic is emitted correctly and no erroneous rule spans are produced for top-level `for`. Quality gates pass. This decision defers implementing `convertStatement` desugaring due to incomplete specification to avoid semantic divergence and can be revisited in future parser generations. Co-authored-by: devboxerhub[bot] <devboxerhub[bot]@users.noreply.github.com> * fix(parser): consume full top-level for statement after diagnostic After emitting the diagnostic for unsupported top-level `for`, the parser now consumes the entire top-level `for` statement including balanced parentheses/braces/brackets and trailing dot. This prevents tokens in the body of a multiline `for` statement from leaking into the rule scanner and being misinterpreted as standalone rules. Additionally, the `UNSUPPORTED_TOP_LEVEL_FOR` constant was made `pub(crate)` and reused in unit tests to avoid test message duplication drift. A new helper, `find_matching_error`, was introduced to aid in structured multi-error pattern matching. Includes new unit and behavioral tests verifying the multiline behavior and updated documentation on the change. Closes issues related to incorrect parsing of multiline top-level `for` statements. Co-authored-by: devboxerhub[bot] <devboxerhub[bot]@users.noreply.github.com> * test(parser): add tests for top-level for loop parsing errors and acceptance Add tests to verify that multiline top-level `for` loops produce the expected UNSUPPORTED_TOP_LEVEL_FOR diagnostic and do not produce rules, while rule-body `for` loops still produce the expected rule. These tests ensure correct parsing behavior and error reporting for top-level `for` constructs. Co-authored-by: devboxerhub[bot] <devboxerhub[bot]@users.noreply.github.com> * feat(parser): reject top-level `for` with diagnostic and skip tokens Add a new constant `UNSUPPORTED_TOP_LEVEL_FOR` for the diagnostic message and implement proper handling in `collect_rule_spans` to reject top-level `for` statements. The new `K_FOR` arm calls `skip_rejected_top_level_for` which emits an appropriate error and consumes the entire rejected statement to prevent parsing errors downstream. Unit and behavioral tests were added to cover this logic. Co-authored-by: devboxerhub[bot] <devboxerhub[bot]@users.noreply.github.com> * docs(execplans): correct typo 'hand-written' to 'handwritten' in markdown doc Co-authored-by: devboxerhub[bot] <devboxerhub[bot]@users.noreply.github.com> * docs(execplans): update tolerances and risks for top-level for statement - Increased implementation scope tolerance from 9 to 11 files due to added test utilities during review. - Clarified risk related to scanner state after top-level 'for' diagnostics with updated mitigation. - Added Decision log entry for scope tolerance increase. - Improved wording and details in Tolerances and Risks sections. - Minor editorial refinements and housekeeping in the ExecPlan document. Co-authored-by: devboxerhub[bot] <devboxerhub[bot]@users.noreply.github.com> * docs(execplans): clarify scope for net-new-line tolerance in Tolerances section - Scoped 200 net-new-line threshold to code-only changes, excluding documentation and the ExecPlan itself. - Added rationale explaining acceptance of minor overshoot beyond threshold due to review additions. - Updated relevant text for precision and aligned with Decision Log. This prevents documentation bloating from affecting feature creep tolerance limits. Co-authored-by: devboxerhub[bot] <devboxerhub[bot]@users.noreply.github.com> --------- Co-authored-by: devboxerhub[bot] <devboxerhub[bot]@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Docstrings generation was requested by @leynos.
The following files were modified:
src/tokenizer.rstests/tokenizer.rsThese file types are not supported
Cargo.tomldocs/parser-plan.mdℹ️ Note
Summary by Sourcery
Add comprehensive Rustdoc comments to core tokenizer functions and related test utilities
Documentation:
tokenizefunction with usage examples and behavior detailskeyword_kindhelper explaining keyword mapping