Pandoc filters are not working anymore with quarto

[FabienSe]

Bug description

I was using quarto with a filter for pandoc, it was working great but it is not working anymore.
I am working on a custom filter developed with panflute to render plantuml diagram in image. A library used to develop pandoc filter with python.
A similar filter can be found here. I was trying to improve it.
I got the same error with this filter too.

I was using it with the filters parameter like this in a index.qmd file:

    ---
    execute:
      echo: false
    pagination_next: null
    filters:
      - custom-filter
    editor:
      render-on-save: true
    ---
    
    # Welcome for fun
    
    ```plantuml
    @startuml
    actor "User" as user
    @enduml
    ```

This file is in a quarto project to render it in a docusaurus website.

The code block was rendered into image by my custom filter. I assume the filter was passed to pandoc directly because the filter is working great with --filter of pandoc cli.

Now I got the following error:

> quarto preview /home/user/project/template_docs/docs/index.qmd --no-browser --no-watch-inputs
pandoc 
  to: >-
    markdown_strict+raw_html+all_symbols_escapable+backtick_code_blocks+fenced_code_blocks+space_in_atx_header+intraword_underscores+lists_without_preceding_blankline+shortcut_reference_links+autolink_bare_uris+emoji+footnotes+gfm_auto_identifiers+pipe_tables+strikeout+task_lists+tex_math_dollars+pipe_tables+tex_math_dollars+header_attributes+raw_html+all_symbols_escapable+backtick_code_blocks+fenced_code_blocks+space_in_atx_header+intraword_underscores+lists_without_preceding_blankline+shortcut_reference_links
  output-file: index.md
  standalone: true
  default-image-extension: png
  wrap: none
  html-math-method: webtex
  
metadata
  pagination_next: null
  editor:
    render-on-save: true
  
Error running filter /opt/quarto/share/filters/main.lua:
Error running filter /home/user/project/template_docs/docs/custom-filter:
Could not find executable /home/user/project/template_docs/docs/custom-filter
stack traceback:
        /opt/quarto/share/filters/main.lua:4026: in function </opt/quarto/share/filters/main.lua:4005>
        [C]: in ?
        [C]: in method 'walk'
        /opt/quarto/share/filters/main.lua:171: in function 'run_emulated_filter'
        /opt/quarto/share/filters/main.lua:449: in local 'callback'
        /opt/quarto/share/filters/main.lua:454: in upvalue 'run_emulated_filter_chain'
        /opt/quarto/share/filters/main.lua:495: in function </opt/quarto/share/filters/main.lua:476>
stack traceback:
        /opt/quarto/share/filters/main.lua:171: in function 'run_emulated_filter'
        /opt/quarto/share/filters/main.lua:449: in local 'callback'
        /opt/quarto/share/filters/main.lua:454: in upvalue 'run_emulated_filter_chain'
        /opt/quarto/share/filters/main.lua:495: in function </opt/quarto/share/filters/main.lua:476>

Operating system : WSL on Ubuntu 20.04.6
Quarto check:

> quarto check

[✓] Checking versions of quarto binary dependencies...
      Pandoc version 3.1.1: OK
      Dart Sass version 1.55.0: OK
[✓] Checking versions of quarto dependencies......OK
[✓] Checking Quarto installation......OK
      Version: 1.3.340
      Path: /opt/quarto/bin

[✓] Checking basic markdown render....OK

[✓] Checking Python 3 installation....OK
      Version: 3.11.3
      Path: /home/user/.pyenv/versions/3.11.3/bin/python3
      Jupyter: (None)

      Jupyter is not available in this Python installation.
      Install with python3 -m pip install jupyter

[✓] Checking R installation...........(None)

      Unable to locate an installed version of R.
      Install R from https://cloud.r-project.org/

Checklist

• Please include a minimal, fully reproducible example in a single .qmd file? Please provide the whole file rather than the snippet you believe is causing the issue.
• Please format your issue so it is easier for us to read the bug report.
• Please document the RStudio IDE version you're running (if applicable), by providing the value displayed in the "About RStudio" main menu dialog?
• Please document the operating system you're running. If on Linux, please provide the specific distribution.
• Please provide the output of quarto check so we know which version of quarto and its dependencies you're running.

[cderv]

Thanks for the report. custom-filter is a python script right ? Did you try adding an extension ?

I pinged @cscheid who knows more about our filter execution layer.

[FabienSe]

custom-filter is a python package which is installed on my computer.

Test can be also made with pandoc-plantuml-filter which I based my custom-filter.

> pip list
Package                Version
---------------------- -------
argcomplete            3.0.5
click                  8.1.3
packaging              23.0
pandoc-plantuml-filter 0.1.2
pandocfilters          1.5.0
panflute               2.3.0
pip                    23.0.1
pipx                   1.2.0
PyYAML                 6.0
setuptools             65.5.0
userpath               1.8.0

This file was working on previous version of quarto.

    ---
    execute:
      echo: false
    pagination_next: null
    filters:
      - pandoc-plantuml
    editor:
      render-on-save: true
    ---
    
    # Welcome for fun
    
    ```plantuml
    @startuml
    actor "User" as user
    @enduml
    ```

Got the same error with it.

But the command following command is working:

> pandoc -s docs/index.qmd -o docs/index.html --filter pandoc-plantuml
[WARNING] Could not deduce format from file extension .qmd
  Defaulting to markdown
Created directory plantuml-images
Created image plantuml-images/8e377f6639c577b8eec950fe2b906a90812ee344.svg
[WARNING] This document format requires a nonempty <title> element.
  Defaulting to 'index' as the title.
  To specify a title, use 'title' in metadata or --metadata title="...".

[cscheid]

Note the error message:

Could not find executable /home/user/project/template_docs/docs/custom-filter

Panflute should work (there are a number of caveats wrt custom AST nodes though); the issue here is that we are not finding the filter where we expect to, and we no longer resolve arbitrary executable files (like pandoc does).

We can reconsider that decision, but the easiest way for you to solve it right now is to add (eg) a python script to the path that then calls the files on your path.

[cderv]

and we no longer resolve arbitrary executable files (like pandoc does).

This was my missing part. I wasn't ware of that change. Thanks for the clarification

[cscheid]

This was my missing part. I wasn't ware of that change. Thanks for the clarification

We have to wrap user filters so they can handle custom AST nodes, and that requires special code. The last time I thought through this I came to the conclusion that it would be hard to do this portably in Lua, but I could be wrong.

I also think that allowing a filter from the user's PATH goes against quarto's philosophy of keeping everything important in the project source, so I'm actually happy with this new restriction. (Otherwise, the result of rendering this file would be non-obviously dependent on the surrounding environment.)

[cderv]

I also think that allowing a filter from the user's PATH goes against quarto's philosophy of keeping everything important in the project source, so I'm actually happy with this new restriction. (Otherwise, the result of rendering this file would be non-obviously dependent on the surrounding environment.)

Yes I agree - it is just a big difference with how it works with Pandoc itself. So I understand the user-side questioning on this for a "pandoc wrapper".

Our extension system support non-lua filter already right ? It seems to be case as in our doc.

Extensions seems to be the good way to user (any) filter with quarto. if it is not possible already, should be adapt our extension to resolve that

@FabienSe did you try using the Extension form to use your filter with a Quarto project ?

On Lua filter side, some new org is following the _extension organisation to be quarto compatible for example: https://github.com/pandoc-ext/abstract-section

[FabienSe]

Thanks @cderv and @cscheid for your answers.

the easiest way for you to solve it right now is to add (eg) a python script to the path that then calls the files on your path.

I succeed to do it with the following python script named custom-filter.py located at the called path:

from custom_filter.main import main
from panflute import Doc

def test(doc: Doc = None) -> Doc:
    return main(doc=doc)

if __name__ == "__main__":
    test()

The documentation metadata seems to be wrong. Indeed the provided document format is
$DocMetadata(doc_format='/opt/quarto/share/filters/customwriter/customwriter.lua') but it is not a problem for me now because I do not use it to do specific treatment according to the format of the document.

I also changed the filter definition to custom-filter.py in the qmd file.

did you try using the Extension form to use your filter with a Quarto project ?

I did not tested it yet.
I have to say I liked panflute solution to be able to write pandoc filter with python because I do not know how to use Lua.
I would like to use Quarto extension but I did not see how to use non-lua filter in the linked documentation.

[cderv]

I would like to use Quarto extension but I did not see how to use non-lua filter in the linked documentation.

Yes we don't document it with example, but we say

You can write Pandoc filters using Lua (via Pandoc’s built-in Lua interpreter) or using any other language using a JSON representation of the Pandoc AST piped to/from an external process. We strongly recommend using Lua Filters

So I am assuming we support already other type of filter. @dragonstyle do we support non lua filter in extension ? Do we have an example somewhere ?

[cscheid]

So I am assuming we support already other type of filter. @dragonstyle do we support non lua filter in extension ? Do we have an example somewhere ?

Extensions won't know the difference, this distinction happens entirely inside of the Lua filter chain now. (quartodoc used to work like that but I helped those folks convert it to a Lua filter exactly because of these limitations.)

[cderv]

I also think that allowing a filter from the user's PATH goes against quarto's philosophy of keeping everything important in the project source

I was thinking that using Extension would be easier to find non-lua filter as it would be relative to the project, and not from the user's PATH.

I understand now:

• Quarto does not resolve the filter in the PATH as pandoc would do, not non-lua filter needs to be loaded a bit differently
• Though they can be used with Quarto using a specific script
• However, using an extension to locally provide the filter will not help directly that much

Not sure it is all crystal clear to me yet. Anyway, as always you're precision helped me understand the changes better.

Thanks!

[cscheid]

The documentation metadata seems to be wrong. Indeed the provided document format is
$DocMetadata(doc_format='/opt/quarto/share/filters/customwriter/customwriter.lua') but it is not a problem for me now because I do not use it to do specific treatment according to the format of the document.

Unfortunately that metadata is not wrong, and the behavior is currently inevitable; this is the output format in quarto (because reasons). That's one of the problems with using non-Lua filters in quarto. We have Lua APIs for checking formats, but we don't yet have language-agnostic APIs in filters. We intend to do that, but not in the near future.