New lexer for syntax-highlighting

[stengerh]

I have experimented with a new lexer which is based on a depth-first, left-to-right traversal of the PegDown AST. The lexer has the following properties:

• It does not generate empty tokens.
• The total length of the generated tokens equals the length of the input.
• It generates an empty token sequence for an empty input.
• It is able to handle nested styles, e.g. emphasis in headers, bold links, etc. Each peace of Markdown source is hihglighted according to innermost applicable style.

The last one is a change from the current behaviour but could be easily changed if desired by the users. Please see the example below which illustrates the highlighting in the editor.

Markdown source:

### Header with [link][example]
a *b **c** d* e

HTML as rendered by PegDown:

Header with link

a b c d e

Highlighting with old lexer:

## Header with [link][example]
a *b **c** d* e

Highlighting with new lexer:

### Header with [link][example]
a *b **c** d* e

[madflow]

Hi,

This is a huge step towards a mature lexer! Thanks for your effort! I only did a quick review - but I will already merge the changes and test if my work lets me ;).

[madflow]

@stengerh Until now - the master branch was always also the stable branch. With this merge it is more "experimental". Since we have the tags as releases I do not mind specifically to change this procedure. Do you prefer an extra "experimental" branch?

Also: Some things already work some don't. Should I create issues for my findings?

[stengerh]

I agree on the experimental nature of this change. While I have done enough testing to convince myself that the code is reasonably robust, it does change the user experience. I am not sure yet whether we should have an experimental branch. However I would like to have some kind of nightly or preview build to get feedback from the users about new features.

Yes, please create issues if you find anything strange or broken.