Skip to content

refactor: use token-based terminator detection in parse_command#329

Merged
ldayton merged 1 commit intomainfrom
feat/token-based-terminators
Jan 17, 2026
Merged

refactor: use token-based terminator detection in parse_command#329
ldayton merged 1 commit intomainfrom
feat/token-based-terminators

Conversation

@ldayton
Copy link
Copy Markdown
Owner

@ldayton ldayton commented Jan 17, 2026

Summary

  • Add _lex_is_command_terminator() method that uses the Lexer to detect command terminators via token types
  • Update parse_command() to use token-based terminator detection instead of character-level checks
  • Remove the now-unused _is_list_terminator() function
  • Update PLAN.md to document the hybrid approach progress

Motivation

This change improves the Parser's architectural alignment with bash's tokenizer-parser separation. By using token-based terminator detection, the EOF token mechanism (implemented in #327) now works at the command level, not just the list level.

Key Insight

The hybrid approach works because:

  • _lex_peek_token() doesn't advance Parser position (only _lex_next_token() does)
  • This allows token-based terminator checks followed by character-based word parsing
  • Position syncing via _sync_lexer() bridges the two modes

Changes

# Before (character-based):
if self.at_end():
    break
ch = self.peek()
if _is_list_terminator(ch):
    break
if ch == "&" and not (self.pos + 1 < self.length and self.source[self.pos + 1] == ">"):
    break

# After (token-based):
if self._lex_is_command_terminator():
    break

All 4515 tests pass in both Python and JavaScript.

Replace character-based terminator checks with token-based detection
using the Lexer. This enables the EOF token mechanism to work at the
command level and improves architectural alignment.

Changes:
- Add _lex_is_command_terminator() method for token-based checking
- Update parse_command() to use token-based terminator detection
- Remove unused _is_list_terminator() function
- Update PLAN.md to document hybrid approach progress
@ldayton ldayton merged commit 9f4b728 into main Jan 17, 2026
1 check passed
@ldayton ldayton deleted the feat/token-based-terminators branch January 17, 2026 12:43
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant