Strange spaces in output depending on execution time.

[mforbes]

Describe the bug

context
When I execute the following code cell...

```{code-cell}
import time
for n in range(10):
    print(n)
    time.sleep(0.02*n)
```

expectation
I expected to get continuous output in a single output block.

bug
But instead, I get spaces between outputs that depend on how long the cell executes:

problem
This is a minor problem, but significantly degrades the output experience.

Notes:

• This does not happen with mystmd execution.
• It is possible this is an upstream issue, but I don't yet know enough about how myst-nb processes code to trace this.

Reproduce the bug

1. Insert the following code cell into a jupyter book. (I just added this to mynewbook/markdown-notebook.md after following the quickstart tutorial:

```{code-cell}
import time
for n in range(10):
    print(n)
    time.sleep(0.02*n)
```

Complete Instructions

python3 -m venv env
./env/bin/python3 -m pip install --upgrade pip jupyter-book
bash --init-file <(echo ". ./env/bin/activate")
jupyter-book create mynewbook
emacs mynewbook/markdown-notebook.md   # Add code cell
jupyter-book build mynewbook
exit
open mynewbook/_build/html/markdown-notebooks.html 

List your environment

$ python --version
Python 3.10.10
$ jupyter-book --version      
Jupyter Book      : 1.0.3
External ToC      : 1.0.1
MyST-Parser       : 2.0.0
MyST-NB           : 1.2.0
Sphinx Book Theme : 1.1.3
Jupyter-Cache     : 1.0.1
NbClient          : 0.10.2
$ uname -a
Darwin Admins-MacBook-Pro-2.local 23.6.0 Darwin Kernel Version 23.6.0: Fri Nov 15 15:13:15 PST 2024; root:xnu-10063.141.1.702.7~1/RELEASE_ARM64_T6000 arm64

[maximlt]

It seems like you can workaround/fix this with setting nb_merge_streams=True.

---

Digging a bit into this, MyST-nb uses nbclient to run the notebook. Running the notebook directly with nbclient and this code:

import nbformat
from nbclient import NotebookClient

notebook_filename = 'notebook.ipynb'
nb = nbformat.read(notebook_filename, as_version=4)
client = NotebookClient(nb, timeout=600, kernel_name='python3')
client.execute()
nbformat.write(nb, 'executed_notebook.ipynb')

Generates the following notebook. Interestingly, it has multiple "stream" outputs that appear to be merged by front-ends like JupyterLab.

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Notebook "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2025-02-09T11:04:48.875585Z",
     "iopub.status.busy": "2025-02-09T11:04:48.875386Z",
     "iopub.status.idle": "2025-02-09T11:04:49.822854Z",
     "shell.execute_reply": "2025-02-09T11:04:49.822063Z"
    },
    "vscode": {
     "languageId": "plaintext"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0\n",
      "1\n",
      "2\n",
      "3\n",
      "4\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "5\n",
      "6\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "7\n",
      "8\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "9\n"
     ]
    }
   ],
   "source": [
    "import time\n",
    "for n in range(10):\n",
    "    print(n)\n",
    "    time.sleep(0.02*n)"
   ]
  }
 ],
 "metadata": {
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.12.7"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}

This behavior of nbclient was already reported jupyter/nbclient#278 and resolved somehow in jupyter/nbclient#279 by adding a new coalesce_streams option.

I'm not sure what's the outcome of all this :)

[mforbes]

@maximlt Perfect! That is exactly what I was looking for. The solution exists and is documented, but the docs need a little updating since they refer only to merging stdout and stderr whereas nbclient emits multiple output streams together.

So the solution is:

• Cell level: At the start of the code cell:

  ```{code-cell}
  ---
  mystnb:
    merge_streams: true
  ---
  import time
  for n in range(10):
      print(n)
      time.sleep(0.02*n)
  ```

• File level: At the top of the file:

   ---
   mystnb:
     merge_streams: true
   ---

• Global: In conf.py:

  nb_merge_streams = True

I supposed that someone might want to merge all stdout streams while keeping stderr separate. This is exactly what is done, so the only potential update needed is to clarify the documentation slightly: If you read the docs really carefully knowing what the outcome is, then they are actually technically correct. But if you stumble on this issue and look at the docs, it is easy to skip this option because this issue has nothing to do with stderr. I will submit a PR.

[maximlt]

The solution exists

Yes though in this case one could argue that MyST-NB (or nbclient directly) should merge the streams automatically since there's only one kind of stream type (stdout) in the cell output.

[mforbes]

I agree that nb_merge_streams = True is a more sensible default but don't know the history, or if this might break backwards compatibility for some people.

[maximlt]

Should this issue be kept open?

[mforbes]

Given that MyST-NB is in maintenance mode, I think there is no need. There is a clear workaround, and my documentation patch was pulled. While changing the defaults might be nice, I think there are more important issues.

[bsipocz]

While I also needed this opt in for my own projects, I would not be comfortable deciding on flipping the default as it may break things for others.