A MyST markdown pandoc extension?

[choldgraf]

I recently spoke with @arfon who mentioned that he knew of some interest in building support for MyST markdown in pandoc. I wanted to mention it here in case this would make it easier for people to port from rST to MyST. @mmcky @AakashGfude and @chrisjsewell in particular may be interested in this

[jstac]

Thanks @choldgraf. CCing @najuzilu who is also interested in this project.

[mmcky]

The pandoc filters providing python support sounds like the way to go here. I like the idea of building a robust converter and it seems to make sense to write one using pandoc.

@chrisjsewell do you think this would be the best technology pathway for this tool based on your experience?

[mmcky]

From reading the documentation for lua filters it supports items like block elements which will be essential for this tool.

It looks like the python pandoc filters approach is based on building json filters which may be more limiting than the lua option.

[chrisjsewell]

Yeh but use https://github.com/sergiocorreia/panflute

[mmcky]

Ah thanks @chrisjsewell that is the tool I couldn't remember when trying to document this issue. Thanks!

[chrisjsewell]

You can use https://github.com/chrisjsewell/ipypublish/tree/develop/ipypublish/filters_pandoc as an example

[mmcky]

@najuzilu I think this is probably the best technology choice for implementing this tool.

Let's put some cycles into using panflute and focus on this over the next couple of weeks.

http://scorreia.com/software/panflute/guide.html#

with examples above from @chrisjsewell

[arfon]

/ cc'ing @tarleb for visibility on this thread too.

[tarleb]

👋 Hi, thanks for the ping! I'd be happy to be part of this.

I agree that panflute is an excellent choice to get going. My hope would be to get MyST support into the main pandoc library some day – having a working filter would make that a lot easier, as we'd only have to translate that into Haskell.

How can I help?

[mmcky]

thanks @arfon @tarleb -- I am Currently working through some docs to get up to speed having never worked with the pandoc AST.

We will need:

1. Collection of panflute filters
2. Simple CLI interface using click

I am working through:

Panflute

https://github.com/sergiocorreia/panflute
http://scorreia.com/software/panflute/index.html

RST:

• reStructuredText Primer
• reStructuredText Directives

Myst:

• MyST Syntax Guide

Pandoc:

• Pandoc User Guide
• List of Pandoc Elements

In my discussions with @chrisjsewell this past week he made some really helpful pointers. The test suite for ipypublish project contains a lot of panflute filters and some testing infrastructure which we can use

https://github.com/chrisjsewell/ipypublish/tree/develop/ipypublish/filters_pandoc

I guess the first point to understand is the execution flow:

1. pandoc converts the input string to a JSON blob
2. Panflute converts that blob to a set of nested python classes
3. The filter(s) modify that python “document tree”
4. Panflute then converts it back to JSON and
5. finally pandoc converts the JSON to what ever the specified output format was

I will put an update together next week re: submission of a working branch with some infrastructure in place to run a collection of filters etc.

[mmcky]

One part of this approach I don't have my head wrapped around is how to update pandoc at the parser level? Sphinx provides a lot of directives such as .. code-block:: that includes a lot of configuration for showing code blocks. From what I have read pandoc implements base docutils rst.

Using the following test string:

s2 = """
    .. code-block:: python3
       :linenos:
       :emphasize-lines: 1
       :name: test-block

       import pandas as pd
    """

From pandoc I am getting the following json representation of this snippet

OrderedDict([('pandoc-api-version', (1, 17, 5, 1)), ('meta', OrderedDict()), ('blocks', [OrderedDict([('t', 'BlockQuote'), ('c', [OrderedDict([('t', 'CodeBlock'), ('c', [['', ['sourceCode', 'python3'], []], 'import pandas as pd'])])])])])])

while it picks up on CodeBlock it doesn't seem to pass through the directive options and config. Any ideas?

[najuzilu]

This is from the pandoc documentation here:

When pandoc is used with -t markdown to create a Markdown document, a YAML metadata block will be produced only if the -s/--standalone option is used. All of the metadata will appear in a single block at the beginning of the document.

I tried it and it only works if the options begin and end with --- or ....

[tarleb]

True, pandoc currently doesn't pass the block attributes on. Would you raise an issue for this on the pandoc issue tracker?

[mmcky]

@tarleb just checking -- do you know for sure that pandoc doesn't pass on the block attributes? I was just wondering if this is a panflute internal issue (not fetching information from pandoc)? Thanks.