Fix RubyLex.ripper_lex_without_warning with heredoc and embexpr

[tompng]

Description

RubyLex.ripper_lex_without_warning(code) cannot tokenize all part of the code.

code = <<~'RUBY'
  s = <<A+1
  #{2+3
RUBY
tokens = RubyLex.ripper_lex_without_warning(code)
p tokens.map(&:tok).join, code
# should be equal. Some tokens are missing
# "s = <<A\#{2+3\n"
# "s = <<A+1\n\#{2+3\n"

I Fixed RubyLex.ripper_lex_without_warning(code) to return

[#<Ripper::Lexer::Elem: on_ident@1:0 CMDARG token: "s">,
 #<Ripper::Lexer::Elem: on_sp@1:1 CMDARG token: " ">,
 #<Ripper::Lexer::Elem: on_op@1:2 BEG token: "=">,
 #<Ripper::Lexer::Elem: on_sp@1:3 BEG token: " ">,
 #<Ripper::Lexer::Elem: on_heredoc_beg@1:4 BEG token: "<<A">,
 #<Ripper::Lexer::Elem: on_ignored_by_ripper@1:7 NONE token: "+1\n">, ← Inserts this dummy token
 #<Ripper::Lexer::Elem: on_embexpr_beg@2:0 BEG token: "\#{">,
 #<Ripper::Lexer::Elem: on_int@2:2 END token: "2">,
 #<Ripper::Lexer::Elem: on_op@2:3 BEG token: "+">,
 #<Ripper::Lexer::Elem: on_int@2:4 END token: "3">,
 #<Ripper::Lexer::Elem: on_nl@2:5 BEG token: "\n">]

Problem of missing tokens

When the code has heredoc and unclosed embexpr inside it, IRB::NestingParser.parse_by_line(tokens) will return fewer lines.

It will cause wrong indent

irb(main):001:0" <<A
irb(main):002:1" #{
irb(main):003:1" if 1
irb(main):004:0"   2
irb(main):005:1"   end
irb(main):005:0"   3

Also, IRB will crash with such code in #607

Related pull-request

I've fixed the same problem only for IRB::Color in #391

[st0012]

Thank you for the fix and the problem is well explained 👍
Do we know why those tokens are ignored by Ripper? Is it intentional or it's a bug?

[tompng]

Is it intentional or it's a bug?

I think it's a bug but I don't know. I've opened a ticket https://bugs.ruby-lang.org/issues/19736