Add option to display pre-lexfilter tokens

[ErikSchierboom]

In this video, it was suggested that it might be useful to be able to add an option to the compiler to output the tokens pre-lexfilter. This PR adds that option. It follows the naming suggested in the PR, but let me know if you'd like to use different naming.

Here is the source code that I've tested with:

open System

let x = 2 + 4

If I run .\fsc.exe --pre-lexfilter-tokenize .\test.fs, it outputs:

C:\Code\fsharp\artifacts\bin\fsc\Debug\netcoreapp3.1 [pre-lexfilter-tokenize-option +0 ~4 -0 !]> .\fsc.exe --pre-lexfilter-tokenize .\test.fs
Microsoft (R) F# Compiler version 11.0.0.0 for F# 5.0
Copyright (c) Microsoft Corporation. All Rights Reserved.

warning FS0075: The command-line option '--pre-lexfilter-tokenize' is for test purposes only
tokenize - getting one token from test.fs
tokenize - got OPEN @ C:\Code\fsharp\artifacts\bin\fsc\Debug\netcoreapp3.1\.\test.fs(1,0)-(1,4)
tokenize - getting one token from test.fs
tokenize - got IDENT @ C:\Code\fsharp\artifacts\bin\fsc\Debug\netcoreapp3.1\.\test.fs(1,5)-(1,11)
tokenize - getting one token from test.fs
tokenize - got LET @ C:\Code\fsharp\artifacts\bin\fsc\Debug\netcoreapp3.1\.\test.fs(3,0)-(3,3)
tokenize - getting one token from test.fs
tokenize - got IDENT @ C:\Code\fsharp\artifacts\bin\fsc\Debug\netcoreapp3.1\.\test.fs(3,4)-(3,5)
tokenize - getting one token from test.fs
tokenize - got EQUALS @ C:\Code\fsharp\artifacts\bin\fsc\Debug\netcoreapp3.1\.\test.fs(3,6)-(3,7)
tokenize - getting one token from test.fs
tokenize - got INT32 @ C:\Code\fsharp\artifacts\bin\fsc\Debug\netcoreapp3.1\.\test.fs(3,8)-(3,9)
tokenize - getting one token from test.fs
tokenize - got PLUS_MINUS_OP @ C:\Code\fsharp\artifacts\bin\fsc\Debug\netcoreapp3.1\.\test.fs(3,10)-(3,11)
tokenize - getting one token from test.fs
tokenize - got INT32 @ C:\Code\fsharp\artifacts\bin\fsc\Debug\netcoreapp3.1\.\test.fs(3,12)-(3,13)
tokenize - getting one token from test.fs
tokenize - got EOF @ C:\Code\fsharp\artifacts\bin\fsc\Debug\netcoreapp3.1\.\test.fs(3,13)-(3,13)

Conversely, running .\fsc.exe --tokenize .\test.fs outputs:

C:\Code\fsharp\artifacts\bin\fsc\Debug\netcoreapp3.1 [pre-lexfilter-tokenize-option +0 ~4 -0 !]> .\fsc.exe --tokenize .\test.fs       
Microsoft (R) F# Compiler version 11.0.0.0 for F# 5.0
Copyright (c) Microsoft Corporation. All Rights Reserved.

warning FS0075: The command-line option '--tokenize' is for test purposes only
tokenize - getting one token from test.fs
tokenize - got OPEN @ C:\Code\fsharp\artifacts\bin\fsc\Debug\netcoreapp3.1\.\test.fs(1,0)-(1,4)
tokenize - getting one token from test.fs
tokenize - got IDENT @ C:\Code\fsharp\artifacts\bin\fsc\Debug\netcoreapp3.1\.\test.fs(1,5)-(1,11)
tokenize - getting one token from test.fs
tokenize - got OBLOCKSEP @ C:\Code\fsharp\artifacts\bin\fsc\Debug\netcoreapp3.1\.\test.fs(1,12)-(3,0)
tokenize - getting one token from test.fs
tokenize - got OLET @ C:\Code\fsharp\artifacts\bin\fsc\Debug\netcoreapp3.1\.\test.fs(3,0)-(3,3)
tokenize - getting one token from test.fs
tokenize - got IDENT @ C:\Code\fsharp\artifacts\bin\fsc\Debug\netcoreapp3.1\.\test.fs(3,4)-(3,5)
tokenize - getting one token from test.fs
tokenize - got EQUALS @ C:\Code\fsharp\artifacts\bin\fsc\Debug\netcoreapp3.1\.\test.fs(3,6)-(3,7)
tokenize - getting one token from test.fs
tokenize - got OBLOCKBEGIN @ C:\Code\fsharp\artifacts\bin\fsc\Debug\netcoreapp3.1\.\test.fs(3,8)-(3,9)
tokenize - getting one token from test.fs
tokenize - got INT32 @ C:\Code\fsharp\artifacts\bin\fsc\Debug\netcoreapp3.1\.\test.fs(3,8)-(3,9)
tokenize - getting one token from test.fs
tokenize - got PLUS_MINUS_OP @ C:\Code\fsharp\artifacts\bin\fsc\Debug\netcoreapp3.1\.\test.fs(3,10)-(3,11)
tokenize - getting one token from test.fs
tokenize - got INT32 @ C:\Code\fsharp\artifacts\bin\fsc\Debug\netcoreapp3.1\.\test.fs(3,12)-(3,13)
tokenize - getting one token from test.fs
tokenize - got OBLOCKEND_COMING_SOON @ C:\Code\fsharp\artifacts\bin\fsc\Debug\netcoreapp3.1\.\test.fs(3,0)-(3,13)
tokenize - getting one token from test.fs
tokenize - got OBLOCKEND_COMING_SOON @ C:\Code\fsharp\artifacts\bin\fsc\Debug\netcoreapp3.1\.\test.fs(3,0)-(3,13)
tokenize - getting one token from test.fs
tokenize - got OBLOCKEND_COMING_SOON @ C:\Code\fsharp\artifacts\bin\fsc\Debug\netcoreapp3.1\.\test.fs(3,0)-(3,13)
tokenize - getting one token from test.fs
tokenize - got OBLOCKEND_COMING_SOON @ C:\Code\fsharp\artifacts\bin\fsc\Debug\netcoreapp3.1\.\test.fs(3,0)-(3,13)
tokenize - got OBLOCKEND_COMING_SOON @ C:\Code\fsharp\artifacts\bin\fsc\Debug\netcoreapp3.1\.\test.fs(3,0)-(3,13)
tokenize - getting one token from test.fs
tokenize - got OBLOCKEND_COMING_SOON @ C:\Code\fsharp\artifacts\bin\fsc\Debug\netcoreapp3.1\.\test.fs(3,0)-(3,13)
tokenize - getting one token from test.fs
tokenize - got OBLOCKEND_IS_HERE @ C:\Code\fsharp\artifacts\bin\fsc\Debug\netcoreapp3.1\.\test.fs(3,0)-(3,13)
tokenize - getting one token from test.fs
tokenize - got ODECLEND @ C:\Code\fsharp\artifacts\bin\fsc\Debug\netcoreapp3.1\.\test.fs(3,0)-(3,13)
tokenize - getting one token from test.fs
tokenize - got EOF @ C:\Code\fsharp\artifacts\bin\fsc\Debug\netcoreapp3.1\.\test.fs(3,13)-(3,13)

[vzarytovskii]

@ErikSchierboom thanks for that, I'm afraid that it may be a duplicate PR.
There's an existing one for unfiltered tokenization: #10578

[ErikSchierboom]

@vzarytovskii! Ah whoops, totally missed that! I'll close this.

[vzarytovskii]

@vzarytovskii! Ah whoops, totally missed that! I'll close this.

Sorry that you've wasted time on that one :(
This is good improvement to have though!

[ErikSchierboom]

Yeah, no worries! It is my mistake that I didn't properly check for an open PR :) And it actually was relatively simple to implement, so hardly any time lost.