diff --git a/figures/lazy_benchmark_metric_time_epochs.png b/figures/lazy_benchmark_metric_time_epochs.png new file mode 100644 index 0000000000..23399a750a Binary files /dev/null and b/figures/lazy_benchmark_metric_time_epochs.png differ diff --git a/figures/lazy_benchmark_total_epoch_time_comparison.png b/figures/lazy_benchmark_total_epoch_time_comparison.png new file mode 100644 index 0000000000..92acf4d57a Binary files /dev/null and b/figures/lazy_benchmark_total_epoch_time_comparison.png differ diff --git a/modules/lazy_resampling_benchmark.ipynb b/modules/lazy_resampling_benchmark.ipynb new file mode 100644 index 0000000000..a907157213 --- /dev/null +++ b/modules/lazy_resampling_benchmark.ipynb @@ -0,0 +1,760 @@ +{ + "cells": [ + { + "cell_type": "markdown", + "id": "7a8e33c4", + "metadata": {}, + "source": [ + "Copyright (c) MONAI Consortium \n", + "Licensed under the Apache License, Version 2.0 (the \"License\"); \n", + "you may not use this file except in compliance with the License. \n", + "You may obtain a copy of the License at \n", + "    http://www.apache.org/licenses/LICENSE-2.0 \n", + "Unless required by applicable law or agreed to in writing, software \n", + "distributed under the License is distributed on an \"AS IS\" BASIS, \n", + "WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. \n", + "See the License for the specific language governing permissions and \n", + "limitations under the License." + ] + }, + { + "attachments": {}, + "cell_type": "markdown", + "id": "8a0aaced", + "metadata": {}, + "source": [ + "# Lazy resampling benchmark\n", + "\n", + "In this notebook, we used 3D spleen segmentation task to show our lazy-resampling benchmark. Our results include the following two main parts.\n", + "- Time spent on each transform and the total amount of data preparation in the lazy and non-lazy mode.\n", + "- End-to-end time comparison" + ] + }, + { + "cell_type": "markdown", + "id": "626a7bfc", + "metadata": {}, + "source": [ + "## Setup environment" + ] + }, + { + "cell_type": "code", + "execution_count": 16, + "id": "9beac76a", + "metadata": {}, + "outputs": [], + "source": [ + "!python -c \"import monai\" || pip install -q \"monai-weekly[nibabel, tqdm, gdown]\"" + ] + }, + { + "cell_type": "markdown", + "id": "1b74510e", + "metadata": {}, + "source": [ + "## Setup imports" + ] + }, + { + "cell_type": "code", + "execution_count": 2, + "id": "cdc814c0", + "metadata": {}, + "outputs": [ + { + "name": "stderr", + "output_type": "stream", + "text": [ + "/opt/conda/lib/python3.8/site-packages/tqdm/auto.py:22: TqdmWarning: IProgress not found. Please update jupyter and ipywidgets. See https://ipywidgets.readthedocs.io/en/stable/user_install.html\n", + " from .autonotebook import tqdm as notebook_tqdm\n" + ] + }, + { + "name": "stdout", + "output_type": "stream", + "text": [ + "MONAI version: 1.2.0rc4+9.ge18097d0\n", + "Numpy version: 1.22.2\n", + "Pytorch version: 1.13.0a0+d0d6b1f\n", + "MONAI flags: HAS_EXT = False, USE_COMPILED = False, USE_META_DICT = False\n", + "MONAI rev id: e18097d0a8ba323ee66aeb7830d296bf54f44bdd\n", + "MONAI __file__: /workspace/Code/MONAI/monai/__init__.py\n", + "\n", + "Optional dependencies:\n", + "Pytorch Ignite version: 0.4.10\n", + "ITK version: 5.3.0\n", + "Nibabel version: 4.0.2\n", + "scikit-image version: 0.19.3\n", + "Pillow version: 9.0.1\n", + "Tensorboard version: 2.10.1\n", + "gdown version: 4.6.0\n", + "TorchVision version: 0.14.0a0\n", + "tqdm version: 4.64.1\n", + "lmdb version: 1.3.0\n", + "psutil version: 5.9.2\n", + "pandas version: 1.4.4\n", + "einops version: 0.6.0\n", + "transformers version: 4.21.3\n", + "mlflow version: 2.0.1\n", + "pynrrd version: 1.0.0\n", + "\n", + "For details about installing the optional dependencies, please visit:\n", + " https://docs.monai.io/en/latest/installation.html#installing-the-recommended-dependencies\n", + "\n" + ] + } + ], + "source": [ + "import os\n", + "import time\n", + "import glob\n", + "import torch\n", + "import tempfile\n", + "import numpy as np\n", + "import monai\n", + "import monai.transforms as mt\n", + "from monai.utils import WorkflowProfiler\n", + "from monai.apps import download_and_extract\n", + "\n", + "from monai.config import print_config\n", + "\n", + "print_config()" + ] + }, + { + "attachments": {}, + "cell_type": "markdown", + "id": "80fd97bc", + "metadata": {}, + "source": [ + "# Setup data directory\n", + "You can specify a directory with the `MONAI_DATA_DIRECTORY` environment variable. \n", + "This allows you to save results and reuse downloads. \n", + "If not specified a temporary directory will be used." + ] + }, + { + "cell_type": "code", + "execution_count": 3, + "id": "6cd43b69", + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "/workspace/Data\n" + ] + } + ], + "source": [ + "directory = os.environ.get(\"MONAI_DATA_DIRECTORY\")\n", + "root_dir = tempfile.mkdtemp() if directory is None else directory\n", + "print(root_dir)" + ] + }, + { + "attachments": {}, + "cell_type": "markdown", + "id": "806db6d7", + "metadata": {}, + "source": [ + "## Download dataset and prepare the utilities\n", + "\n", + "This section downloads and extracts the dataset.\n", + "\n", + "The dataset comes from http://medicaldecathlon.com/.\n", + "\n", + "`TraceObjectsOnly` is used to narrow the scope of the trace to top-level transforms only." + ] + }, + { + "cell_type": "code", + "execution_count": 4, + "id": "4255c4b0", + "metadata": {}, + "outputs": [], + "source": [ + "class TraceObjectsOnly:\n", + " def __init__(self, objects):\n", + " self.objects = objects\n", + "\n", + " def __call__(self, frame):\n", + " self_obj = frame.f_locals.get(\"self\", None)\n", + " return frame.f_code.co_name == \"__call__\" and self_obj in self.objects\n", + "\n", + "\n", + "resource = \"https://msd-for-monai.s3-us-west-2.amazonaws.com/Task09_Spleen.tar\"\n", + "md5 = \"410d4a301da4e5b2f6f86ec3ddba524e\"\n", + "\n", + "compressed_file = os.path.join(root_dir, \"Task09_Spleen.tar\")\n", + "data_dir = os.path.join(root_dir, \"Task09_Spleen\")\n", + "if not os.path.exists(data_dir):\n", + " download_and_extract(resource, compressed_file, root_dir, md5)" + ] + }, + { + "attachments": {}, + "cell_type": "markdown", + "id": "6a734ea8", + "metadata": {}, + "source": [ + "## Set MSD Spleen dataset path" + ] + }, + { + "cell_type": "code", + "execution_count": 5, + "id": "07e4250a", + "metadata": {}, + "outputs": [], + "source": [ + "train_images = sorted(glob.glob(os.path.join(data_dir, \"imagesTr\", \"*.nii.gz\")))\n", + "train_labels = sorted(glob.glob(os.path.join(data_dir, \"labelsTr\", \"*.nii.gz\")))\n", + "data_dicts = [{\"image\": image_name, \"label\": label_name} for image_name, label_name in zip(train_images, train_labels)]\n", + "train_files, val_files = data_dicts[:-9], data_dicts[-9:]" + ] + }, + { + "attachments": {}, + "cell_type": "markdown", + "id": "49e8149b", + "metadata": {}, + "source": [ + "## Transform Profiling Comparison" + ] + }, + { + "attachments": {}, + "cell_type": "markdown", + "id": "fe5f7a0f", + "metadata": {}, + "source": [ + "`transform_list` shows the transform chain used in non-lazy mode.\n", + "- `LoadImaged` loads the spleen CT images and labels from NIfTI format files.\n", + "- `EnsureChannelFirstd` ensures the original data to construct \"channel first\" shape.\n", + "- `ScaleIntensityRanged` extracts intensity range [-57, 164] and scales to [0, 1].\n", + "- `CropForegroundd` removes all zero borders to focus on the valid body area of the images and labels.\n", + "- `Orientationd` unifies the data orientation based on the affine matrix.\n", + "- `Spacingd` adjusts the spacing by `pixdim=(1.5, 1.5, 2.)` based on the affine matrix.\n", + "- `RandCropByPosNegLabeld` randomly crop patch samples from big image based on pos / neg ratio. \n", + "The image centers of negative samples must be in valid body area.\n", + "- `RandAffined` efficiently performs `rotate`, `scale`, `shear`, `translate`, etc. together based on PyTorch affine transform." + ] + }, + { + "cell_type": "code", + "execution_count": 6, + "id": "e3fd7197", + "metadata": {}, + "outputs": [], + "source": [ + "monai.utils.set_determinism(24)\n", + "transform_list = [\n", + " mt.LoadImaged(keys=[\"image\", \"label\"], image_only=True),\n", + " mt.EnsureChannelFirstd(keys=[\"image\", \"label\"]),\n", + " mt.ScaleIntensityRanged(\n", + " keys=[\"image\"],\n", + " a_min=-57,\n", + " a_max=164,\n", + " b_min=0.0,\n", + " b_max=1.0,\n", + " clip=True,\n", + " ),\n", + " mt.CropForegroundd(keys=[\"image\", \"label\"], source_key=\"image\"),\n", + " mt.Orientationd(keys=[\"image\", \"label\"], axcodes=\"RAS\"),\n", + " mt.Spacingd(keys=[\"image\", \"label\"], pixdim=(1.5, 1.5, 2.0), mode=(\"bilinear\", \"nearest\")),\n", + " mt.RandCropByPosNegLabeld(\n", + " keys=[\"image\", \"label\"],\n", + " label_key=\"label\",\n", + " spatial_size=(96, 96, 96),\n", + " pos=1,\n", + " neg=1,\n", + " num_samples=4,\n", + " image_key=\"image\",\n", + " image_threshold=0,\n", + " ),\n", + " mt.RandAffined(\n", + " keys=[\"image\", \"label\"],\n", + " mode=(\"bilinear\", \"nearest\"),\n", + " prob=1.0,\n", + " spatial_size=(96, 96, 96),\n", + " rotate_range=(0, 0, np.pi / 15),\n", + " scale_range=(0.1, 0.1, 0.1),\n", + " ),\n", + "]\n", + "xform = mt.Compose(transform_list)" + ] + }, + { + "attachments": {}, + "cell_type": "markdown", + "id": "07c857fa", + "metadata": {}, + "source": [ + "`transform_list_lazy` shows the transform chain used in lazy mode.\n", + "\n", + "In this particular example, we need to modify the `transform_list` in two places to enable lazy resampling:\n", + "- because the `RandCropByPosNegLabeld` relies on the `label` input to compute foreground and background indices, so we add `Identityd` before it to applying pending operations on `label`.\n", + "- the final `Identityd` is added to calculate to final applying time, in the real case, we can comment this transform and it will be executed automatically at the end of the `Compose`." + ] + }, + { + "cell_type": "code", + "execution_count": 7, + "id": "dca092cf", + "metadata": {}, + "outputs": [], + "source": [ + "monai.utils.set_determinism(24)\n", + "transform_list_lazy = [\n", + " mt.LoadImaged(keys=[\"image\", \"label\"], image_only=True),\n", + " mt.EnsureChannelFirstd(keys=[\"image\", \"label\"]),\n", + " mt.ScaleIntensityRanged(\n", + " keys=[\"image\"],\n", + " a_min=-57,\n", + " a_max=164,\n", + " b_min=0.0,\n", + " b_max=1.0,\n", + " clip=True,\n", + " ),\n", + " mt.CropForegroundd(keys=[\"image\", \"label\"], source_key=\"image\"),\n", + " mt.Orientationd(keys=[\"image\", \"label\"], axcodes=\"RAS\"),\n", + " mt.Spacingd(keys=[\"image\", \"label\"], pixdim=(1.5, 1.5, 2.0), mode=(\"bilinear\", \"nearest\")),\n", + " mt.Identityd(keys=[\"label\"]),\n", + " mt.RandCropByPosNegLabeld(\n", + " keys=[\"image\", \"label\"],\n", + " label_key=\"label\",\n", + " spatial_size=(96, 96, 96),\n", + " pos=1,\n", + " neg=1,\n", + " num_samples=4,\n", + " ),\n", + " mt.RandAffined(\n", + " keys=[\"image\", \"label\"],\n", + " mode=(\"bilinear\", \"nearest\"),\n", + " prob=1.0,\n", + " spatial_size=(96, 96, 96),\n", + " rotate_range=(0, 0, np.pi / 15),\n", + " scale_range=(0.1, 0.1, 0.1),\n", + " ),\n", + " mt.Identityd(keys=[\"image\", \"label\"]),\n", + "]\n", + "\n", + "lazy_kwargs = {\"mode\": (\"bilinear\", 0), \"padding_mode\": (\"border\", \"nearest\"), \"dtype\": (torch.float32, torch.uint8)}\n", + "xform_lazy = mt.Compose(\n", + " transform_list_lazy, lazy_evaluation=True, override_keys=(\"image\", \"label\"), overrides=lazy_kwargs\n", + ")" + ] + }, + { + "attachments": {}, + "cell_type": "markdown", + "id": "e1b8d61a", + "metadata": {}, + "source": [ + "### The preprocessing pipeline evaluated non-lazily" + ] + }, + { + "cell_type": "code", + "execution_count": 8, + "id": "72e3c8f3", + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "non-lazy preprocessing time: 53.942039012908936\n" + ] + }, + { + "data": { + "text/html": [ + "
\n", + "\n", + "\n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + "
CountTotal Time (s)AvgStdMinMax
LoadImaged.__call__3223.6260850.7383150.3783560.2248791.628534
Spacingd.__call__3218.7585360.5862040.2304480.2059770.972495
RandAffined.__call__1287.5335780.0588560.0033410.0553220.081781
ScaleIntensityRanged.__call__321.7459780.0545620.0200780.0216890.099323
CropForegroundd.__call__321.0452090.0326630.0104430.0151150.053857
RandCropByPosNegLabeld.__call__320.8991530.0280990.0082100.0150040.045086
Orientationd.__call__320.0662120.0020690.0004170.0017740.003228
EnsureChannelFirstd.__call__320.0100260.0003130.0000440.0002680.000458
\n", + "
" + ], + "text/plain": [ + " Count Total Time (s) Avg Std \\\n", + "LoadImaged.__call__ 32 23.626085 0.738315 0.378356 \n", + "Spacingd.__call__ 32 18.758536 0.586204 0.230448 \n", + "RandAffined.__call__ 128 7.533578 0.058856 0.003341 \n", + "ScaleIntensityRanged.__call__ 32 1.745978 0.054562 0.020078 \n", + "CropForegroundd.__call__ 32 1.045209 0.032663 0.010443 \n", + "RandCropByPosNegLabeld.__call__ 32 0.899153 0.028099 0.008210 \n", + "Orientationd.__call__ 32 0.066212 0.002069 0.000417 \n", + "EnsureChannelFirstd.__call__ 32 0.010026 0.000313 0.000044 \n", + "\n", + " Min Max \n", + "LoadImaged.__call__ 0.224879 1.628534 \n", + "Spacingd.__call__ 0.205977 0.972495 \n", + "RandAffined.__call__ 0.055322 0.081781 \n", + "ScaleIntensityRanged.__call__ 0.021689 0.099323 \n", + "CropForegroundd.__call__ 0.015115 0.053857 \n", + "RandCropByPosNegLabeld.__call__ 0.015004 0.045086 \n", + "Orientationd.__call__ 0.001774 0.003228 \n", + "EnsureChannelFirstd.__call__ 0.000268 0.000458 " + ] + }, + "execution_count": 8, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "dataset = monai.data.Dataset(train_files, transform=xform)\n", + "data_loader = monai.data.DataLoader(dataset, batch_size=1)\n", + "\n", + "with WorkflowProfiler(TraceObjectsOnly(transform_list)) as wp:\n", + " time_start = time.time()\n", + " for _item_non_lazy in data_loader:\n", + " pass\n", + " time_process = time.time() - time_start\n", + "\n", + "print(f\"non-lazy preprocessing time: {time_process}\")\n", + "wp.get_times_summary_pd()" + ] + }, + { + "attachments": {}, + "cell_type": "markdown", + "id": "7de2e892", + "metadata": {}, + "source": [ + "### The preprocessing pipeline evaluated lazily" + ] + }, + { + "cell_type": "code", + "execution_count": 9, + "id": "116d6e84", + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "lazy preprocessing time: 40.177658557891846\n" + ] + }, + { + "data": { + "text/html": [ + "
\n", + "\n", + "\n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + "
CountTotal Time (s)AvgStdMinMax
LoadImaged.__call__3221.7221010.6788160.3534060.2291941.545509
ScaleIntensityRanged.__call__321.6907930.0528370.0201760.0219440.097047
CropForegroundd.__call__320.9679800.0302490.0088740.0145800.045386
RandCropByPosNegLabeld.__call__320.7248510.0226520.0071750.0111370.033919
RandAffined.__call__1280.1632680.0012760.0001310.0011040.001903
Spacingd.__call__320.1084660.0033900.0001240.0032320.003817
Orientationd.__call__320.0592400.0018510.0000640.0017370.002000
EnsureChannelFirstd.__call__320.0094140.0002940.0000250.0002610.000337
Identityd.__call__1600.0025940.0000160.0000100.0000070.000041
\n", + "
" + ], + "text/plain": [ + " Count Total Time (s) Avg Std \\\n", + "LoadImaged.__call__ 32 21.722101 0.678816 0.353406 \n", + "ScaleIntensityRanged.__call__ 32 1.690793 0.052837 0.020176 \n", + "CropForegroundd.__call__ 32 0.967980 0.030249 0.008874 \n", + "RandCropByPosNegLabeld.__call__ 32 0.724851 0.022652 0.007175 \n", + "RandAffined.__call__ 128 0.163268 0.001276 0.000131 \n", + "Spacingd.__call__ 32 0.108466 0.003390 0.000124 \n", + "Orientationd.__call__ 32 0.059240 0.001851 0.000064 \n", + "EnsureChannelFirstd.__call__ 32 0.009414 0.000294 0.000025 \n", + "Identityd.__call__ 160 0.002594 0.000016 0.000010 \n", + "\n", + " Min Max \n", + "LoadImaged.__call__ 0.229194 1.545509 \n", + "ScaleIntensityRanged.__call__ 0.021944 0.097047 \n", + "CropForegroundd.__call__ 0.014580 0.045386 \n", + "RandCropByPosNegLabeld.__call__ 0.011137 0.033919 \n", + "RandAffined.__call__ 0.001104 0.001903 \n", + "Spacingd.__call__ 0.003232 0.003817 \n", + "Orientationd.__call__ 0.001737 0.002000 \n", + "EnsureChannelFirstd.__call__ 0.000261 0.000337 \n", + "Identityd.__call__ 0.000007 0.000041 " + ] + }, + "execution_count": 9, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "dataset = monai.data.Dataset(train_files, transform=xform_lazy)\n", + "data_loader = monai.data.DataLoader(dataset, batch_size=1)\n", + "\n", + "with WorkflowProfiler(TraceObjectsOnly(transform_list_lazy)) as wp:\n", + " time_start = time.time()\n", + " for _item_lazy in data_loader:\n", + " pass\n", + " time_process = time.time() - time_start\n", + "\n", + "print(f\"lazy preprocessing time: {time_process}\")\n", + "wp.get_times_summary_pd()" + ] + }, + { + "attachments": {}, + "cell_type": "markdown", + "id": "48ba0615", + "metadata": {}, + "source": [ + "## End-to-end workflow Profiling Comparison\n", + "\n", + "https://github.com/Project-MONAI/tutorials/blob/main/3d_segmentation/spleen_segmentation_3d.ipynb contains the complete workflow. Here we should modify it in two places:\n", + "- `transform_list_lazy` or `transform_list` should be used in place of the 'train_transforms' and 'val_transforms'.\n", + "- Use regular Dataset instead of CacheDataset for training and validation process." + ] + }, + { + "attachments": {}, + "cell_type": "markdown", + "id": "3173dbeb", + "metadata": {}, + "source": [ + "### Total time and every epoch time comparison\n", + "![lazy_benchmark_total_epoch_time_comparison](../figures/lazy_benchmark_total_epoch_time_comparison.png)" + ] + }, + { + "attachments": {}, + "cell_type": "markdown", + "id": "dbe52762", + "metadata": {}, + "source": [ + "### Performance comparison\n", + "The results below shows the performance comparison benchmarked on a V100 32G GPU.\n", + "- Performance curve comparison\n", + "- Reach typical metrics time comparison\n", + "- Reach typical metrics epoch comparison\n", + "\n", + "![lazy_benchmark_total_epoch_time_comparison](../figures/lazy_benchmark_metric_time_epochs.png)" + ] + } + ], + "metadata": { + "kernelspec": { + "display_name": "Python 3 (ipykernel)", + "language": "python", + "name": "python3" + }, + "language_info": { + "codemirror_mode": { + "name": "ipython", + "version": 3 + }, + "file_extension": ".py", + "mimetype": "text/x-python", + "name": "python", + "nbconvert_exporter": "python", + "pygments_lexer": "ipython3", + "version": "3.8.10" + } + }, + "nbformat": 4, + "nbformat_minor": 5 +} diff --git a/runner.sh b/runner.sh index 298dcca5de..2b13e9bf72 100755 --- a/runner.sh +++ b/runner.sh @@ -66,6 +66,7 @@ doesnt_contain_max_epochs=("${doesnt_contain_max_epochs[@]}" TCIA_PROSTATEx_Pros doesnt_contain_max_epochs=("${doesnt_contain_max_epochs[@]}" lazy_resampling_functional.ipynb) doesnt_contain_max_epochs=("${doesnt_contain_max_epochs[@]}" lazy_resampling_compose.ipynb) doesnt_contain_max_epochs=("${doesnt_contain_max_epochs[@]}" TensorRT_inference_acceleration.ipynb) +doesnt_contain_max_epochs=("${doesnt_contain_max_epochs[@]}" lazy_resampling_benchmark.ipynb) # Execution of the notebook in these folders / with the filename cannot be automated skip_run_papermill=()