Allow preprocessing of NotebookNode

[kai-tub]

Description / Summary

I would like to be able to preprocess my notebooks before they are parsed.
My wish is to have similar behavior to nbconvert preprocessors`.

Value / benefit

The main motivation for such an extension is that it would be possible to easily include any notebook transformations and previously written nbconvert preprocessors.
An example of a custom preprocessor:
Instead of having to write cell metadata into the special metadata field, one could simply write a magic comment that would automatically be converted to the metadata:

# remove-output
print("hello world")

Could be preprocessed to:

print("hello word")

with the metadata tag: remove-output.

I know that it would be possible to simply write the metadata directly into the metadata field, but this is different for jupyterlab, VSCode, etc...
Also, it is often not easy to see what metadata has been written to each cell.
A different solution is to use a preprocessing script to modify the notebook before being parsed, but that requires this extra step outside of the normal pipeline and some preprocessors are not idempotent (for example, the metadata writer processor removes the metadata line to not pollute the output).

Maybe this is a bit far-fetched, but I hope that it makes a little bit of sense. 😅
A top-down motivation would be for the use inside of jupyter-book, where it would be nice to use the metadata preprocessor that I've shown above to select what inputs/outputs to show inside of the code cell and have this magic comment removed to not pollute the printed output.

A different, simpler use-case would be to use a RegexRemovePreprocessor that would remove all cells that match the regular expression.

Implementation details

I believe that this should be done after executing the notebook.
Then there is no difference between the cached view and the original view, which might introduce some issues.
Like, when we use the RegexRemovePreprocessor and it removes a cell that produces an error.

Specifically, I would execute the preprocessors here.

But then there might be an issue with the source-map/pseudo-line numeration when the preprocessor deletes entires cells.

PS: Maybe I am too focused/used to NotebookNode and should rather consider working on the SyntaxTree.

Tasks to complete

No response

[welcome]

Thanks for opening your first issue here! Engagement like this is essential for open source projects! 🤗

If you haven't done so already, check out EBP's Code of Conduct. Also, please try to follow the issue template as it helps other community members to contribute more effectively.

If your issue is a feature request, others may react to it, to raise its prominence (see Feature Voting).

Welcome to the EBP community! 🎉

[nishikantparmariam]

I would also find this feature useful. Related : jupyter-book/jupyter-book#1740

[maximlt]

Upvoting this feature request :)

[kai-tub]

Hey, this issue was silently fixed in MyST v0.14:

MyST-NB/myst_nb/core/read.py

Lines 65 to 66 in 876edf3

	# add the default reader
	readers.setdefault(".ipynb", (standard_nb_read, {}, False)) # type: ignore

Now, we can overwrite the default ipynb reader and provide wrapper functions to preprocess the notebooks. 🥳
I wasn't even aware of the change until recently.

I was able to write this nice little wrapper:
https://docs.kai-tub.tech/common-nb-preprocessors/usage.html

With it, there is no need to touch the tags editor.
Magic comments can be used to modify the parsing/rendering behavior of MyST-NB 👍

If anyone finds it useful, maybe there is a way to mention it in the docs?
Is there something like a 'community' page for MyST-NB?

[maximlt]

It seems like indeed it would be a good use case to document 👍