Tokenizer for syntax highlighting using Prism

[tompng]

To replace RDoc::Parser::RipperStateLex

[matzbot]

🚀 Preview deployment available at: https://1dae1d57.rdoc-6cd.pages.dev (commit: 46cc1f8)

[st0012]

Why don't we make Prism a dependency and just remove RipperStateLex?

[tompng]

Why don't we make Prism a dependency and just remove RipperStateLex?

RipperStateLex is still used in parser/ruby.rb for parsing, so we can't remove it now.
This new tokenizer doesn't generate state bits required in parser/ruby.rb, so we can't replace RipperStateLex.

This pull request makes parser/prism_ruby.rb not to depend on RipperStateLex. Tokenize to a compatible token stream so that the same syntax highlighter (TokenStream.to_html) can be used, while trying to make colorization unchanged as possible.
For this constraint, tokenizer logic is a bit complicated than it needs to be:

• Have Prism token name to Ripper token name conversion
• Have token squashing which is generally impossible if there is a heredoc. (≒ buggy)

We can change this, but it will also change syntax highlight result. It may also be a relatively large change.

[st0012]

So at the moment we have

• 1 tokenizer using Ripper
• 1 parser using Ripper
• 1 parser using Prism

And this PR will add another tokenizer using Prism. Is this correct?

Have Prism token name to Ripper token name conversion

Will we avoid this if we fully migrate to Prism parser?

I think my main concern is that after this we'll have 2 tokenizers and 2 parsers but it's not clear when we'll be able to drop the old ones.
Do we know:

• Will this change make migrating to Prism easier
• Will migrating to Prism make this or a similar change simpler