Skip to content

Add IR generation BDD scenarios#40

Merged
leynos merged 4 commits intomainfrom
codex/implement-gherkin-scenarios-as-cucumber-rs-tests
Aug 3, 2025
Merged

Add IR generation BDD scenarios#40
leynos merged 4 commits intomainfrom
codex/implement-gherkin-scenarios-as-cucumber-rs-tests

Conversation

@leynos
Copy link
Copy Markdown
Owner

@leynos leynos commented Aug 1, 2025

Summary

  • create ir_generation.feature to describe IR generation behaviour
  • allow steps to compile a manifest via Given
  • check graph and generation results within shared step matcher
  • test BuildGraph creation via BDD scenarios

Testing

  • make fmt
  • make lint
  • make test

https://chatgpt.com/codex/tasks/task_e_688ca0d2a0688322a1d8d2b2010fdc3a

Summary by Sourcery

Add BDD tests for IR generation by creating a new feature file and enhancing step definitions to compile manifests and verify build graph or error outcomes.

New Features:

  • Add ir_generation.feature with BDD scenarios for successful and failed IR generation

Enhancements:

  • Extend step definitions to support compiling manifests and checking build graph contents and generation results

Tests:

  • Introduce Cucumber scenarios covering empty graph, valid compilation, deduplication, missing rules, multiple rules, duplicate outputs, and circular dependencies

@sourcery-ai
Copy link
Copy Markdown
Contributor

sourcery-ai Bot commented Aug 1, 2025

Reviewer's Guide

Adds Cucumber-based BDD scenarios for IR generation, including a new feature file and updated step definitions to validate build graph creation and error conditions.

File-Level Changes

Change Details Files
Add BDD scenarios for IR generation
  • create feature file with scenarios covering graph defaults, valid manifests, deduplication, and failure cases
tests/features/ir_generation.feature
Enhance manifest step definitions to support graph and IR checks
  • add assert_graph and assert_generation helper functions
  • extend when_item_checked to recognize 'graph contents' and 'generation result'
tests/steps/manifest_steps.rs
Update IR step definitions for BDD compatibility
  • import the 'given' macro and annotate existing steps
  • add graph_checked step for validating build graph contents
tests/steps/ir_steps.rs

Tips and commands

Interacting with Sourcery

  • Trigger a new review: Comment @sourcery-ai review on the pull request.
  • Continue discussions: Reply directly to Sourcery's review comments.
  • Generate a GitHub issue from a review comment: Ask Sourcery to create an
    issue from a review comment by replying to it. You can also reply to a
    review comment with @sourcery-ai issue to create an issue from it.
  • Generate a pull request title: Write @sourcery-ai anywhere in the pull
    request title to generate a title at any time. You can also comment
    @sourcery-ai title on the pull request to (re-)generate the title at any time.
  • Generate a pull request summary: Write @sourcery-ai summary anywhere in
    the pull request body to generate a PR summary at any time exactly where you
    want it. You can also comment @sourcery-ai summary on the pull request to
    (re-)generate the summary at any time.
  • Generate reviewer's guide: Comment @sourcery-ai guide on the pull
    request to (re-)generate the reviewer's guide at any time.
  • Resolve all Sourcery comments: Comment @sourcery-ai resolve on the
    pull request to resolve all Sourcery comments. Useful if you've already
    addressed all the comments and don't want to see them anymore.
  • Dismiss all Sourcery reviews: Comment @sourcery-ai dismiss on the pull
    request to dismiss all existing Sourcery reviews. Especially useful if you
    want to start fresh with a new review - don't forget to comment
    @sourcery-ai review to trigger a new review!

Customizing Your Experience

Access your dashboard to:

  • Enable or disable review features such as the Sourcery-generated pull request
    summary, the reviewer's guide, and others.
  • Change the review language.
  • Add, remove or edit custom review instructions.
  • Adjust other review settings.

Getting Help

@coderabbitai
Copy link
Copy Markdown
Contributor

coderabbitai Bot commented Aug 1, 2025

Warning

Rate limit exceeded

@leynos has exceeded the limit for the number of commits or files that can be reviewed per hour. Please wait 13 minutes and 21 seconds before requesting another review.

⌛ How to resolve this issue?

After the wait time has elapsed, a review can be triggered using the @coderabbitai review command as a PR comment. Alternatively, push new commits to this PR.

We recommend that you space out your commits to avoid hitting the rate limit.

🚦 How do rate limits work?

CodeRabbit enforces hourly rate limits for each developer per organization.

Our paid plans have higher rate limits than the trial, open-source and free plans. In all cases, we re-allow further reviews after a brief timeout.

Please see our FAQ for further information.

📥 Commits

Reviewing files that changed from the base of the PR and between e0cc369 and 484ec3b.

📒 Files selected for processing (3)
  • src/ir.rs (3 hunks)
  • tests/steps/ir_steps.rs (3 hunks)
  • tests/steps/manifest_steps.rs (2 hunks)

Summary by CodeRabbit

  • Tests
    • Introduced new feature tests for Intermediate Representation (IR) Generation, covering build graph creation, rule deduplication, error handling, and circular dependencies.
    • Enhanced step definitions and assertions for verifying build graph presence and IR generation attempts.
    • Improved test step flexibility by allowing steps to be used in different contexts.

Walkthrough

Add a feature test suite for IR (Intermediate Representation) Generation, covering successful and error scenarios for compiling manifests into build graphs. Extend Cucumber step definitions and assertions to support new test cases and verification steps. Integrate new step handlers and assertion functions to validate build graph creation and IR generation attempts.

Changes

Cohort / File(s) Change Summary
IR Generation Feature Tests
tests/features/ir_generation.feature
Add a comprehensive feature test suite for IR Generation, including scenarios for success, deduplication, and error handling.
IR Step Definitions
tests/steps/ir_steps.rs
Extend Cucumber step definitions: add new given/when annotations, introduce a step for checking graph contents, and update imports.
Manifest Step Assertions
tests/steps/manifest_steps.rs
Add assertion functions for verifying build graph presence and IR generation attempts; integrate these into the item-checked step handler.

Sequence Diagram(s)

sequenceDiagram
    participant Tester
    participant TestWorld
    participant IRCompiler

    Tester->>TestWorld: Create new BuildGraph
    Tester->>TestWorld: Compile manifest file to IR
    TestWorld->>IRCompiler: Compile manifest
    IRCompiler-->>TestWorld: BuildGraph or Error
    Tester->>TestWorld: Check graph contents or generation result
Loading

Estimated code review effort

🎯 2 (Simple) | ⏱️ ~8 minutes

Possibly related PRs

Poem

IRs assemble, rules deduplicate,
Graphs are checked at every gate.
Errors caught, the cycles found,
Step by step, the tests abound.
Code and feature hand in hand—
BuildGraph marches, just as planned!
🦀✨

✨ Finishing Touches
  • 📝 Generate Docstrings
🧪 Generate unit tests
  • Create PR with unit tests
  • Post copyable unit tests in a comment
  • Commit unit tests in branch codex/implement-gherkin-scenarios-as-cucumber-rs-tests

🪧 Tips

Chat

There are 3 ways to chat with CodeRabbit:

  • Review comments: Directly reply to a review comment made by CodeRabbit. Example:
    • I pushed a fix in commit <commit_id>, please review it.
    • Explain this complex logic.
    • Open a follow-up GitHub issue for this discussion.
  • Files and specific lines of code (under the "Files changed" tab): Tag @coderabbitai in a new review comment at the desired location with your query. Examples:
    • @coderabbitai explain this code block.
    • @coderabbitai modularize this function.
  • PR comments: Tag @coderabbitai in a new PR comment to ask questions about the PR branch. For the best results, please provide a very specific query, as very limited context is provided in this mode. Examples:
    • @coderabbitai gather interesting stats about this repository and render them as a table. Additionally, render a pie chart showing the language distribution in the codebase.
    • @coderabbitai read src/utils.ts and explain its main purpose.
    • @coderabbitai read the files in the src/scheduler package and generate a class diagram using mermaid and a README in the markdown format.
    • @coderabbitai help me debug CodeRabbit configuration file.

Support

Need help? Create a ticket on our support page for assistance with any issues or questions.

Note: Be mindful of the bot's finite context window. It's strongly recommended to break down tasks such as reading entire modules into smaller chunks. For a focused discussion, use review comments to chat about specific files and their changes, instead of using the PR comments.

CodeRabbit Commands (Invoked using PR comments)

  • @coderabbitai pause to pause the reviews on a PR.
  • @coderabbitai resume to resume the paused reviews.
  • @coderabbitai review to trigger an incremental review. This is useful when automatic reviews are disabled for the repository.
  • @coderabbitai full review to do a full review from scratch and review all the files again.
  • @coderabbitai summary to regenerate the summary of the PR.
  • @coderabbitai generate docstrings to generate docstrings for this PR.
  • @coderabbitai generate sequence diagram to generate a sequence diagram of the changes in this PR.
  • @coderabbitai generate unit tests to generate unit tests for this PR.
  • @coderabbitai resolve resolve all the CodeRabbit review comments.
  • @coderabbitai configuration to show the current CodeRabbit configuration for the repository.
  • @coderabbitai help to get help.

Other keywords and placeholders

  • Add @coderabbitai ignore anywhere in the PR description to prevent this PR from being reviewed.
  • Add @coderabbitai summary to generate the high-level summary at a specific location in the PR description.
  • Add @coderabbitai anywhere in the PR title to generate the title automatically.

CodeRabbit Configuration File (.coderabbit.yaml)

  • You can programmatically configure CodeRabbit by adding a .coderabbit.yaml file to the root of your repository.
  • Please see the configuration documentation for more information.
  • If your editor has YAML language server enabled, you can add the path at the top of this file to enable auto-completion and validation: # yaml-language-server: $schema=https://coderabbit.ai/integrations/schema.v2.json

Documentation and Community

  • Visit our Documentation for detailed information on how to use CodeRabbit.
  • Join our Discord Community to get help, request features, and share feedback.
  • Follow us on X/Twitter for updates and announcements.

Copy link
Copy Markdown
Contributor

@coderabbitai coderabbitai Bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 1

📜 Review details

Configuration used: CodeRabbit UI
Review profile: ASSERTIVE
Plan: Pro

📥 Commits

Reviewing files that changed from the base of the PR and between 94b0304 and e0cc369.

📒 Files selected for processing (3)
  • tests/features/ir_generation.feature (1 hunks)
  • tests/steps/ir_steps.rs (3 hunks)
  • tests/steps/manifest_steps.rs (2 hunks)
🧰 Additional context used
📓 Path-based instructions (1)
**/*.rs

📄 CodeRabbit Inference Engine (AGENTS.md)

**/*.rs: Clippy warnings MUST be disallowed.
Fix any warnings emitted during tests in the code itself rather than silencing them.
Where a function is too long, extract meaningfully named helper functions adhering to separation of concerns and CQRS.
Where a function has too many parameters, group related parameters in meaningfully named structs.
Where a function is returning a large error consider using Arc to reduce the amount of data returned.
Write unit and behavioural tests for new functionality. Run both before and after making any change.
Every module must begin with a module level (//!) comment explaining the module's purpose and utility.
Document public APIs using Rustdoc comments (///) so documentation can be generated with cargo doc.
Prefer immutable data and avoid unnecessary mut bindings.
Handle errors with the Result type instead of panicking where feasible.
Avoid unsafe code unless absolutely necessary and document any usage clearly.
Place function attributes after doc comments.
Do not use return in single-line functions.
Use predicate functions for conditional criteria with more than two branches.
Lints must not be silenced except as a last resort.
Lint rule suppressions must be tightly scoped and include a clear reason.
Prefer expect over allow.
Prefer .expect() over .unwrap().
Use concat!() to combine long string literals rather than escaping newlines with a backslash.
Prefer semantic error enums: Derive std::error::Error (via the thiserror crate) for any condition the caller might inspect, retry, or map to an HTTP status.
Use an opaque error only at the app boundary: Use eyre::Report for human-readable logs; these should not be exposed in public APIs.
Never export the opaque type from a library: Convert to domain enums at API boundaries, and to eyre only in the main main() entrypoint or top-level async task.

Files:

  • tests/steps/manifest_steps.rs
  • tests/steps/ir_steps.rs

⚙️ CodeRabbit Configuration File

**/*.rs: * Seek to keep the cyclomatic complexity of functions no more than 12.

  • Adhere to single responsibility and CQRS

  • Place function attributes after doc comments.

  • Do not use return in single-line functions.

  • Move conditionals with >2 branches into a predicate function.

  • Avoid unsafe unless absolutely necessary.

  • Every module must begin with a //! doc comment that explains the module's purpose and utility.

  • Comments and docs must follow en-GB-oxendict (-ize / -our) spelling and grammar

  • Lints must not be silenced except as a last resort.

    • #[allow] is forbidden.
    • Only narrowly scoped #[expect(lint, reason = "...")] is allowed.
    • No lint groups, no blanket or file-wide suppression.
    • Include FIXME: with link if a fix is expected.
  • Use rstest fixtures for shared setup and to avoid repetition between tests.

  • Replace duplicated tests with #[rstest(...)] parameterised cases.

  • Prefer mockall for mocks/stubs.

  • Prefer .expect() over .unwrap()

  • Ensure that any API or behavioural changes are reflected in the documentation in docs/

  • Ensure that any completed roadmap steps are recorded in the appropriate roadmap in docs/

  • Files must not exceed 400 lines in length

    • Large modules must be decomposed
    • Long match statements or dispatch tables should be decomposed by domain and collocated with targets
    • Large blocks of inline data (e.g., test fixtures, constants or templates) must be moved to external files and inlined at compile-time or loaded at run-time.

Files:

  • tests/steps/manifest_steps.rs
  • tests/steps/ir_steps.rs
🔇 Additional comments (7)
tests/features/ir_generation.feature (1)

1-44: Comprehensive BDD scenarios with proper Gherkin structure.

The feature file provides excellent test coverage for IR generation, including both success and failure scenarios. The scenarios are well-structured and cover critical edge cases like rule deduplication, circular dependencies, and various validation failures.

The test cases appropriately validate the core functionality whilst ensuring robust error handling.

tests/steps/manifest_steps.rs (3)

38-43: Clear and focused assertion function.

The assert_graph function follows the established pattern and provides a clear assertion with an appropriate error message.


45-50: Appropriate assertion logic for IR generation attempts.

The assert_generation function correctly validates that IR generation was attempted by checking for either a successful build graph or a manifest error. This covers both success and failure paths appropriately.


66-67: Proper integration of new assertion functions.

The extension to when_item_checked correctly maps the new step phrases to their corresponding assertion functions, maintaining consistency with the existing pattern.

tests/steps/ir_steps.rs (3)

4-4: Appropriate import addition for Given step support.

Adding the given import enables the extended step annotations and follows the existing import pattern.


7-7: Good reuse of existing function as Given step.

The additional #[given] annotation appropriately allows the create_graph function to be used in both Given and When contexts without code duplication.


35-35: Proper extension for manifest compilation as Given step.

The additional #[given] annotation enables reuse of the compilation logic in precondition contexts, maintaining DRY principles.

Comment thread tests/steps/ir_steps.rs
Copy link
Copy Markdown
Contributor

@sourcery-ai sourcery-ai Bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hey @leynos - I've reviewed your changes - here's some feedback:

  • assert_graph and assert_generation currently live in manifest_steps.rs despite only being used for IR scenarios—consider moving them into ir_steps.rs to keep IR logic encapsulated.
  • assert_generation’s check (build_graph.is_some() || manifest_error.is_some()) can mask unexpected states; consider splitting success/failure assertions or adding a dedicated IR error field for clearer pass/fail reporting.
  • The create_graph step is annotated with both #[given] and #[when] using the same text—if you don’t need both entry points, you can remove one to reduce redundant annotations.
Prompt for AI Agents
Please address the comments from this code review:
## Overall Comments
- assert_graph and assert_generation currently live in manifest_steps.rs despite only being used for IR scenarios—consider moving them into ir_steps.rs to keep IR logic encapsulated.
- assert_generation’s check (build_graph.is_some() || manifest_error.is_some()) can mask unexpected states; consider splitting success/failure assertions or adding a dedicated IR error field for clearer pass/fail reporting.
- The create_graph step is annotated with both #[given] and #[when] using the same text—if you don’t need both entry points, you can remove one to reduce redundant annotations.

Sourcery is free for open source - if you like our reviews please consider sharing them ✨
Help me be more useful! Please click 👍 or 👎 on each comment and I'll use the feedback to improve your reviews.

@leynos
Copy link
Copy Markdown
Owner Author

leynos commented Aug 1, 2025

@coderabbitai Have the following now been resolved?

  • assert_graph and assert_generation currently live in manifest_steps.rs despite only being used for IR scenarios—consider moving them into ir_steps.rs to keep IR logic encapsulated.
  • assert_generation’s check (build_graph.is_some() || manifest_error.is_some()) can mask unexpected states; consider splitting success/failure assertions or adding a dedicated IR error field for clearer pass/fail reporting.
  • The create_graph step is annotated with both #[given] and #[when] using the same text—if you don’t need both entry points, you can remove one to reduce redundant annotations.

@coderabbitai
Copy link
Copy Markdown
Contributor

coderabbitai Bot commented Aug 1, 2025

Tip

For best results, initiate chat on the files or code changes.

🧩 Analysis chain

🏁 Script executed:

#!/bin/bash
# Check the current content of manifest_steps.rs to see if assert_graph and assert_generation are still there
echo "=== Checking manifest_steps.rs for assert_graph and assert_generation ==="
cat tests/steps/manifest_steps.rs

echo -e "\n=== Checking ir_steps.rs content ==="
cat tests/steps/ir_steps.rs

echo -e "\n=== Searching for assert_graph function across all files ==="
rg "fn assert_graph" -A 5

echo -e "\n=== Searching for assert_generation function across all files ==="
rg "fn assert_generation" -A 5

echo -e "\n=== Checking create_graph annotations ==="
rg "create_graph" -A 2 -B 2

Length of output: 7544


@leynos Based on my examination of the current code, here's the status of the three issues:

RESOLVED: Function location

The assert_graph and assert_generation_attempted functions have been moved from manifest_steps.rs to ir_steps.rs, properly encapsulating IR-specific logic.

RESOLVED: Redundant annotations

The create_graph function now only has the #[given("a new BuildGraph is created")] annotation. The redundant #[when] annotation has been removed.

⚠️ PARTIALLY RESOLVED: Assert generation logic

This has been improved but not fully resolved:

  • Good: The function was renamed to assert_generation_attempted for clarity
  • Good: Separate step handlers now exist (graph_contents_checked vs generation_result_checked)
  • Still an issue: The underlying logic world.build_graph.is_some() || world.manifest_error.is_some() can still mask unexpected states where both fields are None

The core concern remains: if both build_graph and manifest_error are None, the assertion would fail, but it's unclear whether this represents a bug in the test setup or legitimate state. Consider adding explicit checks for this scenario or ensuring one of these fields is always populated after IR generation attempts.

Two out of three issues have been fully resolved, with the third showing significant improvement but retaining the original logic concern.

@leynos
Copy link
Copy Markdown
Owner Author

leynos commented Aug 3, 2025

@coderabbitai review

@coderabbitai
Copy link
Copy Markdown
Contributor

coderabbitai Bot commented Aug 3, 2025

✅ Actions performed

Review triggered.

Note: CodeRabbit is an incremental review system and does not re-review already reviewed commits. This command is applicable only when automatic reviews are paused.

@leynos
Copy link
Copy Markdown
Owner Author

leynos commented Aug 3, 2025

@coderabbitai Have the following now been resolved?

Still an issue: The underlying logic world.build_graph.is_some() || world.manifest_error.is_some() can still mask unexpected states where both fields are None

The core concern remains: if both build_graph and manifest_error are None, the assertion would fail, but it's unclear whether this represents a bug in the test setup or legitimate state. Consider adding explicit checks for this scenario or ensuring one of these fields is always populated after IR generation attempts.

Two out of three issues have been fully resolved, with the third showing significant improvement but retaining the original logic concern.

@coderabbitai
Copy link
Copy Markdown
Contributor

coderabbitai Bot commented Aug 3, 2025

Tip

For best results, initiate chat on the files or code changes.

🧩 Analysis chain

@leynos Based on my examination of the current code, this issue has NOT been resolved.

The assert_generation_attempted function still uses the exact same logic:

fn assert_generation_attempted(world: &CliWorld) {
    assert!(world.build_graph.is_some() || world.manifest_error.is_some());
}

The core problem remains unchanged:

  • If both world.build_graph and world.manifest_error are None, the assertion fails
  • This failure doesn't distinguish between a legitimate test failure and an unexpected state bug
  • The error message provides no context about which field should have been populated

Recommendations to properly resolve this:

  1. Add explicit state validation:
fn assert_generation_attempted(world: &CliWorld) {
    match (world.build_graph.is_some(), world.manifest_error.is_some()) {
        (true, false) => {}, // Success case
        (false, true) => {}, // Error case
        (false, false) => panic!("BUG: IR generation was attempted but neither build_graph nor manifest_error was populated"),
        (true, true) => panic!("BUG: Both build_graph and manifest_error are populated - this should not happen"),
    }
}
  1. Or split into separate assertions:
fn assert_generation_succeeded(world: &CliWorld) {
    assert!(world.build_graph.is_some(), "Expected build_graph to be populated after successful IR generation");
    assert!(world.manifest_error.is_none(), "Expected no manifest_error after successful IR generation");
}

fn assert_generation_failed(world: &CliWorld) {
    assert!(world.manifest_error.is_some(), "Expected manifest_error to be populated after failed IR generation");
    assert!(world.build_graph.is_none(), "Expected no build_graph after failed IR generation");
}

The underlying logic concern has not been addressed - the assertion can still mask unexpected states and provide unclear failure messages.


🏁 Script executed:

#!/bin/bash
# Check the current implementation of assert_generation_attempted function
echo "=== Current implementation of assert_generation_attempted ==="
rg "fn assert_generation_attempted" -A 10 -B 2

echo -e "\n=== Check if there are any explicit None checks ==="
rg "build_graph.*is_none|manifest_error.*is_none" -A 3 -B 3

echo -e "\n=== Look for any panic or error messages related to unexpected states ==="
rg "both.*None|unexpected.*state|neither.*nor" -A 2 -B 2

echo -e "\n=== Check the full context of ir_steps.rs around assertion functions ==="
cat tests/steps/ir_steps.rs | grep -A 15 -B 5 "assert_generation_attempted"

Length of output: 3159


@leynos YES, this issue has been FULLY RESOLVED!

The assert_generation_attempted function has been completely rewritten to address the exact concern you raised:

fn assert_generation_attempted(world: &CliWorld) {
    match (world.build_graph.is_some(), world.manifest_error.is_some()) {
        (true, false) | (false, true) => (),
        (true, true) => panic!("unexpected: graph and error present"),
        (false, false) => panic!("IR generation not attempted"),
    }
}

What's been fixed:

  1. No more masking of unexpected states - The function now explicitly handles all four possible combinations
  2. Clear error messages - Instead of a generic assertion failure, you get specific panic messages:
    • "unexpected: graph and error present" when both fields are populated
    • "IR generation not attempted" when both fields are None
  3. Explicit state validation - The match statement makes the expected states obvious and catches edge cases

This implementation perfectly addresses your original concerns:

  • ✅ Distinguishes between legitimate test failures and unexpected state bugs
  • ✅ Provides clear context about which state was encountered
  • ✅ No longer masks the problematic case where both fields are None

All three issues from your original feedback have now been fully resolved!

@leynos leynos merged commit b9daeb5 into main Aug 3, 2025
3 checks passed
@leynos leynos deleted the codex/implement-gherkin-scenarios-as-cucumber-rs-tests branch August 3, 2025 23:01
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant