include does not inject code into a non-executable block

[aronatkins]

Bug description

Using Quarto 0.9.607 (contained in a recent RStudio build -- 2022.10.0 Build 9), but have also tested Quarto 0.9.629. The same rendered results occur with both versions, and within and outside the RStudio IDE.

Observed on macOS Monterey - 12.4

Given index.qmd defined as:

---
title: embedding code using includes
---

## in a code block

The `cats.R` file should be injected into the R code block and correctly 
formatted without being executed.

```r
{{< include cats.R >}}
```

## outside a code block

Here is an example showing how the R code can be injected, but it does not
have the desired code-block styling.

{{< include cats.R >}}

## expected

This is the expected result when formatted:

```r
cat("cats!\n")
```

and a cats.R file defined as:

cat("cats!\n")

The rendering should present the code from cats.R as formatted R code in the first code block (containing the include).

Instead, the rendered result has an empty code block and looks like:

Checklist

• formatted your issue so it is easier for us to read?
• included a minimal, self-contained, and reproducible example?
• documented the quarto version you're running, by providing the output produced by quarto check in a terminal in your issue?
• documented the RStudio IDE version you're running (if applicable), by providing the value displayed in the "About RStudio" main menu dialog?
• documented which operating system you're running? If on Linux, please provide the specific distribution as well.
• upgraded to the latest version, including your versions of R, the RStudio IDE, and relevant R packages?

[cscheid]

Thanks! I can repro this locally.

[cscheid]

The "issue" here is that shortcodes are currently only detected as "standalone" markdown constructs. This is admittedly bad, but I'm concerned about changing this ahead of v1.0.

It's not the only problem with shortcodes, either. A related problem happens, for example, if we try including from inside a bulleted list:

* Hello.

  This is a paragraph.

  Include?

  {{< include cats.R >}}

* Another item.

The result looks like so:

This happens because our detection of shortcodes is not syntax-aware, and doesn't know to expand the file by including the bulleted list indentation.

[aronatkins]

Codes like {{< meta ... >}} and {{< var ... >}} also cannot be used within code blocks, probably for similar reasons.

[jjallaire]

Those definitely should be working in code blocks (here we specifically look for them: https://github.com/quarto-dev/quarto-cli/blob/main/src/resources/filters/quarto-pre/shortcodes.lua#L14). If they don't work its a regression we need to fix for v1.0 (as we currently use that in the quarto website and RS Workbench docs)

[jjallaire]

@aronatkins These do work in static/display code blocks (i.e. non-executable) but they don't work for executable code blocks (and they will likely not ever work inside executable blocks).

[jjallaire]

Note that knitr already has a native feature for specifying the contents of code chunk from an external file (file option): https://quarto.org/docs/reference/cells/cells-knitr.html#include

[jjallaire]

One line we don't want to cross is allowing content in code chunks/cells that is not syntactically valid for the language (which in turn prevents cell-by-cell execution in RStudio, VS Code, Jupyter Lab, etc.)

[aronatkins]

Thanks @jjallaire - I've confirmed that shortcodes like env work in non-executable code chunks. Keep this issue only against include.

In case others have similar problems:

My workaround to have code examples included in plain code blocks is to use the knitr engine embed. For example, here is how I have included an R script within the document:

---
title: embedded example
engine: knitr
---

```{embed, file = "environ.R"}
```

To have values available throughout the document, I am currently using environment variables, as they can be accessed in code (using language-specific techniques), in prose, in YAML fields, etc.

---
title: "Going {{< env HOME >}}"
engine: knitr
---

It is so nice to go {{< env HOME >}}

Using environment variables in code blocks using language-specific references:

```{bash}
echo "${HOME}"
```

```{r}
cat(Sys.getenv("HOME"))
```

Environment variables in code blocks (not executable):

```r
echo "{{< env HOME >}}"
```