Investigate emitting `:on_sp` event for ripper lex

[kddnewton]

IRB needs this for calculating leading whitespace so it knows how to indent multi statements.

If this can be done on the Ruby side, we should do it there, so keep the simplicity of the C parser.

[kddnewton]

Closing this for now. It's documented, and as long as all of our consumers can work around it, we're going to avoid implementing it. I really hope no one ends up needing it. If they do we'll revisit.

[schneems]

My only worry is that the original source couldn’t be constructed using the lex information and position information if we are dropping tokens.

I don’t know the theory behind different lex output, but I somewhat assumed that I could convert between source code and lex output and back with no loss.

Syntax suggest doesn’t rely on that property, but it’s something I’ve considered using before.

[kddnewton]

The only tokens we drop are on_sp. We don't emit them because it's a lot of tokens and I didn't think anyone actually relied on them because they don't have any semantic meaning.

If you want to reconstruct the source, you still can. The only thing that's missing is spaces, so if you put all of the tokens back in their exact positions and slice out all of the spaces from the source, you could definitely do that.

I just figured that's not something anyone actually needs to do. But if they do we can do it in Ruby instead of slowing down the overall parser.