diff --git a/docs/source/bundle_intro.rst b/docs/source/bundle_intro.rst new file mode 100644 index 0000000000..3d71de24b6 --- /dev/null +++ b/docs/source/bundle_intro.rst @@ -0,0 +1,10 @@ +:github_url: https://github.com/Project-MONAI/MONAI + +Bundle +====== + +.. toctree:: + :maxdepth: 1 + + mb_specification + config_syntax.md diff --git a/docs/source/config_syntax.md b/docs/source/config_syntax.md new file mode 100644 index 0000000000..bcd849cd09 --- /dev/null +++ b/docs/source/config_syntax.md @@ -0,0 +1,184 @@ +# MONAI Bundle Configuration + +The `monai.bundle` module supports building Python-based workflows via structured configurations. + +The main benefits are threefold: + +- it provides good readability and usability by separating system parameter settings from the Python code. +- it describes workflow at a relatively high level and allows for different low-level implementations. +- learning paradigms at a higher level such as federated learning and AutoML can be decoupled from the component details. + +Content: + +- [A basic example](#a-basic-example) +- [Syntax examples explained](#syntax-examples-explained) + - [`@` to interpolate with Python objects](#1--to-interpolate-with-python-objects) + - [`$` to evaluate as Python expressions](#2--to-evaluate-as-python-expressions) + - [`%` to textually replace configuration elements](#3--to-textually-replace-configuration-elements) + - [`_target_` (`_disabled_` and `_requires_`) to instantiate a Python object](#4-instantiate-a-python-object) +- [The command line interface](#the-command-line-interface) +- [Recommendations](#recommendations) + +## A basic example + +Components as part of a workflow can be specified using `JSON` or `YAML` syntax, for example, a network architecture +definition could be stored in a `demo_config.json` file with the following content: + +```json +{ + "demo_net": { + "_target_": "monai.networks.nets.BasicUNet", + "spatial_dims": 3, + "in_channels": 1, + "out_channels": 2, + "features": [16, 16, 32, 32, 64, 64] + } +} +``` + +or alternatively, in `YAML` format (`demo_config.yaml`): + +```yaml +demo_net: + _target_: monai.networks.nets.BasicUNet + spatial_dims: 3 + in_channels: 1 + out_channels: 2 + features: [16, 16, 32, 32, 64, 64] +``` + +The configuration parser can instantiate the component as a Python object: + +```py +>>> from monai.bundle import ConfigParser +>>> config = ConfigParser() +>>> config.read_config("demo_config.json") +>>> net = config.get_parsed_content("demo_net", instantiate=True) +BasicUNet features: (16, 16, 32, 32, 64, 64). +>>> print(type(net)) + +``` + +or additionally, tune the input parameters then instantiate the component: + +```py +>>> config["demo_net"]["features"] = [32, 32, 32, 64, 64, 64] +>>> net = config.get_parsed_content("demo_net", instantiate=True) +BasicUNet features: (32, 32, 32, 64, 64, 64). +``` + +For more details on the `ConfigParser` API, please see https://docs.monai.io/en/latest/bundle.html#config-parser. + +## Syntax examples explained + +A few characters and keywords are interpreted beyond the plain texts, here are examples of the syntax: + +### 1. `@` to interpolate with Python objects + +```json +"@preprocessing#transforms#keys" +``` + +_Description:_ A reference to another configuration value defined at `preprocessing#transforms#keys`. +where `#` indicates a sub-structure of this configuration file. + +```json +"@preprocessing#1" +``` + +_Description:_ `1` is interpreted as an integer, which is used to index (zero-based indexing) the `preprocessing` sub-structure. + +### 2. `$` to evaluate as Python expressions + +```json +"$print(42)" +``` + +_Description:_ `$` is a special character to indicate evaluating `print(42)` at runtime. + +```json +"$[i for i in @datalist]" +``` + +_Description:_ Create a list at runtime using the values in `datalist` as input. + +```json +"$from torchvision.models import resnet18" +``` + +_Description:_ `$` followed by an import statement is handled slightly differently from the +Python expressions. The imported module `resnet18` will be available as a global variable +to the other configuration sections. This is to simplify the use of external modules in the configuration. + +### 3. `%` to textually replace configuration elements + +```json +"%demo_config.json#demo_net#in_channels" +``` + +_Description:_ A macro to replace the current configuration element with the texts at `demo_net#in_channels` in the +`demo_config.json` file. The replacement is done before instantiating or evaluating the components. + +### 4. instantiate a Python object + +```json +{ + "demo_name":{ + "_target_": "my.python.module.Class", + "args1": "string", + "args2": 42} +} +``` + +_Description:_ This dictionary defines an object with a reference name `demo_name`, with an instantiable type +specified at `_target_` and with input arguments `args1` and `args2`. +This dictionary will be instantiated as a Pytorch object at runtime. + +`_target_` is a required key by monai bundle syntax for the Python object name. +`args1` and `args2` should be compatible with the Python object to instantiate. + +```json +{ + "component_name": { + "_target_": "my.module.Class", + "_requires_": "@cudnn_opt", + "_disabled_": "true"} +} +``` + +_Description:_ `_requires_` and `_disabled_` are optional keys. +`_requires_` specifies references (string starts with `@`) or +Python expression that will be evaluated/instantiated before `_target_` object is instantiated. +It is useful when the component does not explicitly depend on the other ConfigItems via +its arguments, but requires the dependencies to be instantiated/evaluated beforehand. +`_disabled_` specifies a flag to indicate whether to skip the instantiation. + +## The command line interface + +In addition to the Pythonic APIs, a few command line interfaces (CLI) are provided to interact with the bundle. +The primary usage is: +```bash +python -m monai.bundle COMMANDS +``` + +where `COMMANDS` is one of the following: `run`, `verify_metadata`, `ckpt_export`, ... +(please see `python -m monai.bundle --help` for a list of available options). + +To display a usage page for a command, for example `run`: +```bash +python -m monai.bundle run -- --help +``` + +The support is provided by [Python Fire](https://github.com/google/python-fire), please +make sure the optional dependency is installed, for example, +using `pip install monai[fire]` or `pip install fire`. +Details on the CLI argument parsing is provided in the +[Python Fire Guide](https://github.com/google/python-fire/blob/master/docs/guide.md#argument-parsing). + +## Recommendations +- Both `YAML` and `JSON` are supported, but the advanced features of these formats are not supported. +- Using meaningful names for the configuration elements can improve the readability. +- While it is possible to build complex configurations with the bundle syntax, + simple structures with sparse uses of expressions or references are preferred. +- For `$import ` in the configuration, please make sure there are instructions for the users to install + the `` if it is not a (optional) dependency of MONAI. diff --git a/docs/source/index.rst b/docs/source/index.rst index 1a4263db0d..6c99feac11 100644 --- a/docs/source/index.rst +++ b/docs/source/index.rst @@ -70,7 +70,7 @@ Technical documentation is available at `docs.monai.io `_ :maxdepth: 1 :caption: Specifications - mb_specification + bundle_intro Links ----- diff --git a/docs/source/mb_specification.rst b/docs/source/mb_specification.rst index d383dd7d8e..5bdfa148e2 100644 --- a/docs/source/mb_specification.rst +++ b/docs/source/mb_specification.rst @@ -13,7 +13,7 @@ This specification defines the directory structure a bundle must have and the ne Directory Structure =================== -A MONAI Bundle is defined primarily as a directory with a set of specifically named subdirectories containing the model and metadata files. The root directory should be named for the model, given as "ModelName" in this exmaple, and should contain the following structure: +A MONAI Bundle is defined primarily as a directory with a set of specifically named subdirectories containing the model and metadata files. The root directory should be named for the model, given as "ModelName" in this example, and should contain the following structure: :: @@ -55,7 +55,7 @@ This file contains the metadata information relating to the model, including wha * **optional_packages_version**: dictionary relating optional package names to their versions, these packages are not needed but are recommended to be installed with this stated minimum version. * **task**: plain-language description of what the model is meant to do. * **description**: longer form plain-language description of what the model is, what it does, etc. -* **authorship**: state author(s) of the model. +* **authors**: state author(s) of the model. * **copyright**: state model copyright. * **network_data_format**: defines the format, shape, and meaning of inputs and outputs to the model, contains keys "inputs" and "outputs" relating named inputs/outputs to their format specifiers (defined below). @@ -98,7 +98,7 @@ An example JSON metadata file: "optional_packages_version": {"nibabel": "3.2.1"}, "task": "Decathlon spleen segmentation", "description": "A pre-trained model for volumetric (3D) segmentation of the spleen from CT image", - "authorship": "MONAI team", + "authors": "MONAI team", "copyright": "Copyright (c) MONAI Consortium", "data_source": "Task09_Spleen.tar from http://medicaldecathlon.com/", "data_type": "dicom", diff --git a/monai/bundle/config_item.py b/monai/bundle/config_item.py index 0531c6f14e..3300fe91ff 100644 --- a/monai/bundle/config_item.py +++ b/monai/bundle/config_item.py @@ -21,7 +21,7 @@ from monai.bundle.utils import EXPR_KEY from monai.utils import ensure_tuple, first, instantiate, optional_import -__all__ = ["ComponentLocator", "ConfigItem", "ConfigExpression", "ConfigComponent"] +__all__ = ["ComponentLocator", "ConfigItem", "ConfigExpression", "ConfigComponent", "Instantiable"] class Instantiable(ABC): @@ -173,7 +173,7 @@ class ConfigComponent(ConfigItem, Instantiable): - ``"_requires_"`` (optional): specifies reference IDs (string starts with ``"@"``) or ``ConfigExpression`` of the dependencies for this ``ConfigComponent`` object. These dependencies will be evaluated/instantiated before this object is instantiated. It is useful when the - component doesn't explicitly depends on the other `ConfigItems` via its arguments, + component doesn't explicitly depend on the other `ConfigItems` via its arguments, but requires the dependencies to be instantiated/evaluated beforehand. - ``"_disabled_"`` (optional): a flag to indicate whether to skip the instantiation. @@ -302,7 +302,7 @@ class ConfigExpression(ConfigItem): config = "$monai.__version__" expression = ConfigExpression(config, id="test", globals={"monai": monai}) - print(expression.execute()) + print(expression.evaluate()) Args: config: content of a config item. diff --git a/monai/bundle/config_parser.py b/monai/bundle/config_parser.py index 800e18ade0..a9107a10e9 100644 --- a/monai/bundle/config_parser.py +++ b/monai/bundle/config_parser.py @@ -211,12 +211,16 @@ def get_parsed_content(self, id: str = "", **kwargs): Use digits indexing from "0" for list or other strings for dict. For example: ``"xform#5"``, ``"net#channels"``. ``""`` indicates the entire ``self.config``. kwargs: additional keyword arguments to be passed to ``_resolve_one_item``. - Currently support ``reset`` (for parse), ``instantiate`` and ``eval_expr``. All defaulting to True. + Currently support ``lazy`` (whether to retain the current config cache, default to `False`), + ``instantiate`` (whether to instantiate the `ConfigComponent`, default to `True`) and + ``eval_expr`` (whether to evaluate the `ConfigExpression`, default to `True`). """ if not self.ref_resolver.is_resolved(): # not parsed the config source yet, parse it - self.parse(kwargs.get("reset", True)) + self.parse(reset=True) + elif not kwargs.get("lazy", False): + self.parse(reset=not kwargs.get("lazy", False)) return self.ref_resolver.get_resolved_content(id=id, **kwargs) def read_meta(self, f: Union[PathLike, Sequence[PathLike], Dict], **kwargs): diff --git a/monai/bundle/scripts.py b/monai/bundle/scripts.py index 894fbd6c25..64172c4541 100644 --- a/monai/bundle/scripts.py +++ b/monai/bundle/scripts.py @@ -207,7 +207,7 @@ def verify_metadata( ): """ Verify the provided `metadata` file based on the predefined `schema`. - `metadata` content must contain the `schema` field for the URL of shcema file to download. + `metadata` content must contain the `schema` field for the URL of schema file to download. The schema standard follows: http://json-schema.org/. Args: diff --git a/tests/testing_data/data_config.json b/tests/testing_data/data_config.json index 8bcdcd244e..e337a2d2e7 100644 --- a/tests/testing_data/data_config.json +++ b/tests/testing_data/data_config.json @@ -70,9 +70,9 @@ }, "configs": { "test_meta_file": { - "url": "https://github.com/Project-MONAI/MONAI-extra-test-data/releases/download/0.8.1/meta_schema_202203171008.json", + "url": "https://github.com/Project-MONAI/MONAI-extra-test-data/releases/download/0.8.1/meta_schema_20220324.json", "hash_type": "md5", - "hash_val": "e3a7e23d1113a1f3e6c69f09b6f9ce2c" + "hash_val": "e12813de2c15672a8c8fa8466b3dfc95" } } } diff --git a/tests/testing_data/metadata.json b/tests/testing_data/metadata.json index 42a55b114c..c12ae411f1 100644 --- a/tests/testing_data/metadata.json +++ b/tests/testing_data/metadata.json @@ -1,5 +1,5 @@ { - "schema": "https://github.com/Project-MONAI/MONAI-extra-test-data/releases/download/0.8.1/meta_schema_202203171008.json", + "schema": "https://github.com/Project-MONAI/MONAI-extra-test-data/releases/download/0.8.1/meta_schema_20220324.json", "version": "0.1.0", "changelog": { "0.1.0": "complete the model package", @@ -13,7 +13,7 @@ }, "task": "Decathlon spleen segmentation", "description": "A pre-trained model for volumetric (3D) segmentation of the spleen from CT image", - "authorship": "MONAI team", + "authors": "MONAI team", "copyright": "Copyright (c) MONAI Consortium", "data_source": "Task09_Spleen.tar from http://medicaldecathlon.com/", "data_type": "dicom",