Remove imprecise T_IDENTIFIER capture groups, leading to simpler token sequences#258
Conversation
A `T_IDENTIFIER` represents: * a constant * a function name * a function name prefixed with `\` * a class name * a class name prefixed with `\` * a class constant * a class constant, with class name prefixed with `\` The current capturing regular expression is too simplistic, and it leads to edge cases where `Foo\42.5` are captured as separate tokens. In this patch, we deprecate `DocLexer::T_NAMESPACE_SEPARATOR` (which shouldn't be looked up directly anymore), and tell the parser to work with wider `T_IDENTIFIER` tokens instead. Note that a test is also being removed. That is intentional, since the test verifies a parser issue when parsing `@Foo\3.42`: * previously, the tokens would be `[@ Foo\3.42]` * now they are `[@, Foo, \, 3.42]` This also means that the above will be parsed as `@Foo`.
nicolas-grekas
left a comment
nicolas-grekas
left a comment
There was a problem hiding this comment.
Makes sense (+ would fix a deprecation that blocks Symfony's CI on PHP 7.4 :) )
|
|
||
| $className = $this->lexer->token['value']; | ||
|
|
||
| while ($this->lexer->lookahead['position'] === ($this->lexer->token['position'] + strlen($this->lexer->token['value'])) |
There was a problem hiding this comment.
because lookahead can be null here, this line triggers a deprecation on 7.4
that's one of the main remaining blockers we have to make Symfony's tests pass on PHP 7.4 :)
There was a problem hiding this comment.
@guilhermeblanco let's separate the concern of fixing 7.4 compat and the issue found with this patch: we should check for the lookahead to exist in this while(), and have a test that triggers a deprecation warning on 7.4
|
Fixes #273 |
| * @expectedException \Doctrine\Common\Annotations\AnnotationException | ||
| * @expectedExceptionMessage [Syntax Error] Expected Doctrine\Common\Annotations\DocLexer::T_IDENTIFIER or Doctrine\Common\Annotations\DocLexer::T_TRUE or Doctrine\Common\Annotations\DocLexer::T_FALSE or Doctrine\Common\Annotations\DocLexer::T_NULL, got '3.42' at position 5. | ||
| */ | ||
| public function testInvalidIdentifierInAnnotation() |
There was a problem hiding this comment.
Urgh, I shouldn't have removed this test. It should at least have a test declaring another sort of failure here.
|
As agreed, we decided to take a simplified approach and only resolve the 7.4 notice instead. |
6 participants
Note: I just need a review - will forward-port after that.
Discovered while working on #257
A
T_IDENTIFIERrepresents:\\\The current capturing regular expression is too simplistic, and it
leads to edge cases where
Foo\42.5are captured as separate tokens.In this patch, we deprecate
DocLexer::T_NAMESPACE_SEPARATOR(whichshouldn't be looked up directly anymore), and tell the parser to work
with wider
T_IDENTIFIERtokens instead.Note that a test is also being removed. That is intentional, since the
test verifies a parser issue when parsing
@Foo\3.42:[@ Foo\3.42][@, Foo, \, 3.42]This also means that the above will be parsed as
@Foo.