Skip to content

Conversation

@michal-pawlik
Copy link
Contributor

When lexer is unable to find matching token, LexingError raises. This error contains source position for further debugging, but in current implementation it only points valid index (character number), while the line and colum numbers equal -1.

This PR fixes this issue by tracking column number in next() method of LexerStream and providing both self._lineno and self._colno to SourcePosition in LexingError.


stream = l.lex("\nfail")
with raises(LexingError) as excinfo:
stream.next()
Copy link
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please move this line outside the raises block so it's clear which line raises an exception.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@alex good point, fixed

l = lg.build()
stream = l.lex("1+2+fail")
with raises(LexingError) as excinfo:
stream.next()
Copy link
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Same here.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed

@alex alex merged commit 6e16262 into alex:master Oct 16, 2019
@nobodxbodon
Copy link
Contributor

Seems there's edge case not covered here:

    def test_error_reset_column_number(self):
        lg = LexerGenerator()
        lg.add("NEW_LINE", r"\n")
        lg.add("NUMBER", r"\d+")
        l = lg.build()
        stream = l.lex("12\nfail")
        stream.next()
        stream.next()
        with raises(LexingError) as excinfo:
            stream.next()

        assert excinfo.value.source_pos.colno == 1   # it fails with 3 instead

The same assertion works if added in test_error_line_number.

nobodxbodon pushed a commit to MulanRevive/mulan-rework that referenced this pull request Feb 21, 2021
@nobodxbodon
Copy link
Contributor

FYI here's a fix for colno when there's lexing error: nobodxbodon@4afbd7f , covering the edge case above.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants