feat(core): start support for hex numbers

[hippietrail]

towards #543

based on the existing lex_number which cuts off when the digits finish, which is fine because a unit of measurement etc might immediately follow. But with a hex number a non-hex letter might follow and not be flagged if it's a single letter. Also if it's very wrong, the start won't be flagged if it's 0x and at least one hex digit. Examples will help illustrate:

So just a draft PR to solicit thoughts and mods.

[hippietrail]

I'm still figuring out GitHub. I made this PR a draft because it's not ready to be merged.

But I think "draft" also means "not ready for review" but I specifically want it reviewed to get feedback on what it should do with text "stuck to" the hex that is not hex. And perhaps how to change the code to handle comply to what's decided. Especially if we want to flag such "hex with appendages".

If my understanding of draft status for a PR is still wrong, please let me know.

[elijah-potter]

@hippietrail, the code looks good. Would you mind adding some test cases under harper-core/tests/test_sources?

I would add a couple Markdown files that should be considered "clean" with this lexer added. Make sure you actually import them in harper-core/tests/run_tests.rs.

[hippietrail]

@hippietrail, the code looks good. Would you mind adding some test cases under harper-core/tests/test_sources?

I would add a couple Markdown files that should be considered "clean" with this lexer added. Make sure you actually import them in harper-core/tests/run_tests.rs.

Are you sure? Did you see my concerns about what to do regarding non-hex text immediately after the hex with no space etc between?

---

After some thought I came to grok the problem more completely and refactored the code, now written in a more old-school way that I'm used to doing lexing which makes it easier to think about when it gets tricky.

I added positive and negative tests each in a loop. Let me know if this is a good or bad idea.

[ccoVeille]

I'm worried about possible reporting of leet speak like 0xf0rd

Then it's a matter of balance, what is the more likely to be found in text? Hexadecimal or leet speak ?

I updated #555 accordingly.

I would prefer hex rules to be merged after leet speak is handled #598

[ccoVeille]

I'm worried about possible reporting of leet speak like 0xf0rd

Then it's a matter of balance, what is the more likely to be found in text? Hexadecimal or leet speak ?

I updated #555 accordingly.

I would prefer hex rules to be merged after leet speak is handled #598

@elijah-potter you merged. It's OK, but can I get a feedback on what I explained?