Refactor Lexer to be infallible for LSP support#1428
Conversation
- Changed `Lexer::scan_tokens` to return `Vec<Token>` instead of `Result`. - Added `TokenKind::Error` to represent syntax errors (e.g., unterminated strings, invalid chars) without aborting the token stream. - Updated `Interpreter::interpret` to check for `TokenKind::Error` and return early errors, preserving runtime behavior. - Updated `Interpreter::complete` to ignore error tokens, allowing autocompletion to work on partially invalid code. - Updated tests to match the new infallible Lexer API.
|
👋 Jules, reporting for duty! I'm here to lend a hand with this pull request. When you start a review, I'll add a 👀 emoji to each comment to let you know I've read it. I'll focus on feedback directed at me and will do my best to stay out of conversations between you and other bots or reviewers to keep the noise down. I'll push a commit with your requested changes shortly after. Please note there might be a delay between these steps, but rest assured I'm on the job! For more direct control, you can switch me to Reactive Mode. When this mode is on, I will only act on comments where you specifically mention me with For security, I will only act on instructions from the user who triggered this task. New to Jules? Learn more at jules.google/docs. |
Summary
Previous Results
Insights
Slowest Tests
🎉 No failed tests in this run. | 🍂 No flaky tests in this run. Github Test Reporter by CTRF 💚 🔄 This comment has been updated |
|
Jules run cargo fmt |
Running cargo fmt now. |
- Updated `eldritch-core/src/token.rs` to handle `TokenKind::Error` in `fmt::Display`. - Updated `eldritch-repl/src/core.rs` to handle `Vec<Token>` return type from `Lexer::scan_tokens` and correctly process error tokens for REPL logic. - Verified that all tests pass in `implants/`. Co-authored-by: google-labs-jules[bot] <161369871+google-labs-jules[bot]@users.noreply.github.com>
2 participants
Refactored the
Lexerineldritch-coreto returnVec<Token>directly, ensuring it never panics or returnsErron syntax errors. This allows tools (like an LSP) to obtain a partial token stream even when the code is incomplete or contains errors. Syntax errors are now captured asTokenKind::Errortokens. The Interpreter was updated to handle these error tokens gracefully.PR created automatically by Jules for task 11738474779397551328 started by @KCarretto