From eea17c1b3c0254aee391cf0f5057dbbdd021bb4b Mon Sep 17 00:00:00 2001 From: HarmonicReflux Date: Thu, 20 Jun 2024 14:15:45 +0100 Subject: [PATCH 01/11] I refactored developers.rst to make instructions more concise. As an "easy" fix regarding the test that makes the .ipynb notebook fail, I included `muse --model default` to be run as a necessary step to complete setting MUSE_OS up. Given the fact that after patching the setup, 304 tests pass successfully while 10 are skipped, 1 comes with an expected fail and 4 warnings , it is probably better to sort these out in separate issues to separate responsibilities and make it easier to solve them. --- docs/installation/developers.rst | 25 +++++++++++++++++++------ 1 file changed, 19 insertions(+), 6 deletions(-) diff --git a/docs/installation/developers.rst b/docs/installation/developers.rst index 905d849c4..44b49f5cd 100644 --- a/docs/installation/developers.rst +++ b/docs/installation/developers.rst @@ -3,7 +3,9 @@ Installation for developers --------------------------- -Developers of MUSE will need to have the version control tool ``git`` installed in their system and be familiar with its usage. The `Introduction to Git and GitHub for software development `_ course created by `Imperial RSE Team `_ can be a good place to start. +Developers of MUSE will need to have the version control tool ``git`` installed in their system and be familiar with its usage. +The `Introduction to Git and GitHub for software development `_ and `Further Git and GitHub for effective collaboration `_ courses by the `Imperial RSE Team `_ are excellent starting points. + Installing MUSE source code in editable mode ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ @@ -14,20 +16,29 @@ Once you have ``git`` in your system, clone the MUSE repository with: git clone https://github.com/SGIModel/MUSE_OS.git +For developing MUSE, **it is highly recommended to do so in a virtual Python environment**. This is to isolate dependencies and to avoid version conflicts on other Python libraries. +Moreover, this allows for unconstrained experimentation with the code without breaking system dependencies, which, as least as far as ``Linux`` is concerned, is an issue. -For developing MUSE, **it is highly recommended to use virtual environments**, so that you have more control on the versions of the dependencies being used, making easy to change them without affecting other tools using Python, or even to have several environments in parallel for different purposes (Eg. new features, bug fixing...). You can either use ``conda`` or ``venv`` as explained in :ref:`virtual-environments`. +Also, any operating system allows for multiple Python versions being installed in parallel for different purposes like creating new features and bug fixing. +You can either use ``conda`` or ``pyenv`` in conjunction with ``venv`` as explained in :ref:`virtual-environments`. Once you have your environment created, **activate it** and install MUSE within the environment: .. code-block:: bash cd MUSE_OS - # 1- Create virtual environment - # 2- Activate virtual environment + # 1- Create a virtual environment + # 2- Activate that virtual environment # 3- Finally, install MUSE in editable mode with: python -m pip install -e .[dev,doc] -Depending on your system, you might need to add quotation marks around ``[dev,doc]`` as in ``"[dev,doc]"``. This will install MUSE and all the dependencies required for its development. The downloaded code can be modified and the changes will be automatically reflected in the environment. +.. note:: + Depending on your system, you might need to add quotation marks around ``[dev,doc]`` as in ``"[dev,doc]"``. + + For example on ``Windows``, the command will read `python -m pip install -e .[dev,doc]`. On Ubuntu Linux, it will be `python -m pip install -e ."[dev,doc]"`. + This will install MUSE including its dependencies for development. The downloaded code can be modified and the changes will be automatically reflected in the virtual environment. + +Finally, in the activated virtual environment, run `muse --model default` to generate output data that are used to make `pytest` run without failures. .. note:: @@ -48,7 +59,9 @@ To ensure the consistency of the code with other developers, install the pre-com Running tests ~~~~~~~~~~~~~ -In the developing phase, MUSE can also be used to run test cases to check that the model would reproduce expected results from a defined set of input data. Tests can be run with the command [pytest](https://docs.pytest.org/en/latest/), from the testing framework of the same name. +If you followed **Installation for developers**, you will run and develop your own tests for MUSE. +This is to check that a model reproduces expected results for a defined set of input data. +Tests can be run with the command [pytest](https://docs.pytest.org/en/latest/), from the testing framework of the same name. Within the ``MUSE-OS`` directory, just run: From ed3e15f4e8c94800d86a4df799d1f0ce9b59c290 Mon Sep 17 00:00:00 2001 From: HarmonicReflux Date: Fri, 21 Jun 2024 16:58:42 +0100 Subject: [PATCH 02/11] This patch checks whether the file "Results/MCACapacity.csv" exists, which is true only upon running `muse --model default` or any other model that creates MCACapacity.csv and only then proceeds running the test notebooks. I hard-code this explicit file dependency which may not be the most general solution, however, due to the explicit dependency on MCACapacity.csv in one of the test notebooks, which itself is hardcoded, I think this is a straight-forward approach. --- docs/installation/developers.rst | 4 ++-- tests/test_docs_notebooks.py | 3 +++ 2 files changed, 5 insertions(+), 2 deletions(-) diff --git a/docs/installation/developers.rst b/docs/installation/developers.rst index 44b49f5cd..44a21d3aa 100644 --- a/docs/installation/developers.rst +++ b/docs/installation/developers.rst @@ -29,8 +29,8 @@ Once you have your environment created, **activate it** and install MUSE within cd MUSE_OS # 1- Create a virtual environment # 2- Activate that virtual environment - # 3- Finally, install MUSE in editable mode with: - python -m pip install -e .[dev,doc] + # 3- Install MUSE in editable mode with: python -m pip install -e .[dev,doc] + # 4 - Invoke `muse --model default` .. note:: Depending on your system, you might need to add quotation marks around ``[dev,doc]`` as in ``"[dev,doc]"``. diff --git a/tests/test_docs_notebooks.py b/tests/test_docs_notebooks.py index a06e023d6..9d70eb31a 100644 --- a/tests/test_docs_notebooks.py +++ b/tests/test_docs_notebooks.py @@ -8,6 +8,9 @@ def available_notebooks() -> List[Path]: """Locate the available notebooks in the docs.""" + if not Path("Results/MCACapacity.csv").exists(): + return [] + base_path = Path(__file__).parent.parent / "docs" return [p for p in base_path.rglob("*.ipynb") if "build" not in str(p)] From 6fb7c9ef72ed217bcc7c4ba774b20d466905a243 Mon Sep 17 00:00:00 2001 From: HarmonicReflux Date: Mon, 1 Jul 2024 12:31:54 +0100 Subject: [PATCH 03/11] Update test_docs_notebooks.py Revoked changes to exclude the test in case input file is not there as per discussion for pull requests. Updated the manual to instruct users to run the default model on Muse to complete the installation. This ensures all tests pass (some may be skipped) and avoids including model results files in the codebase. --- tests/test_docs_notebooks.py | 3 --- 1 file changed, 3 deletions(-) diff --git a/tests/test_docs_notebooks.py b/tests/test_docs_notebooks.py index f41531726..2d2d94cd2 100644 --- a/tests/test_docs_notebooks.py +++ b/tests/test_docs_notebooks.py @@ -7,9 +7,6 @@ def available_notebooks() -> list[Path]: """Locate the available notebooks in the docs.""" - if not Path("Results/MCACapacity.csv").exists(): - return [] - base_path = Path(__file__).parent.parent / "docs" return [p for p in base_path.rglob("*.ipynb") if "build" not in str(p)] From 5e7d0a5c6752fc4675a6c502854ba351b8574319 Mon Sep 17 00:00:00 2001 From: HarmonicReflux Date: Mon, 1 Jul 2024 19:14:42 +0100 Subject: [PATCH 04/11] As per Alex's suggestion, I added an explicit note to run a MUSE model before the test scripts. Running a model is now mentioned twice: once in the installation instructions and once before running the tests. This ensures readers who do not follow the guide in sequence will understand the requirement and avoid confusion. --- docs/installation/developers.rst | 22 +++++++++++++++------- 1 file changed, 15 insertions(+), 7 deletions(-) diff --git a/docs/installation/developers.rst b/docs/installation/developers.rst index cf65cd411..968970370 100644 --- a/docs/installation/developers.rst +++ b/docs/installation/developers.rst @@ -27,12 +27,13 @@ Once you have your environment created, **activate it** and install MUSE within .. code-block:: bash cd MUSE_OS - # 1- Create a virtual environment - # 2- Activate that virtual environment - # 3- Install MUSE in editable mode with: python -m pip install -e .[dev,doc] + # 1 - Create a virtual environment + # 2 - Activate that virtual environment + # 3 - Install MUSE in editable mode with: python -m pip install -e .[dev,doc] # 4 - Invoke `muse --model default` .. note:: + Depending on your system, you might need to add quotation marks around ``[dev,doc]`` as in ``"[dev,doc]"``. For example on ``Windows``, the command will read `python -m pip install -e .[dev,doc]`. On Ubuntu Linux, it will be `python -m pip install -e ."[dev,doc]"`. @@ -59,14 +60,21 @@ To ensure the consistency of the code with other developers, install the pre-com Running tests ~~~~~~~~~~~~~ -If you followed **Installation for developers**, you will run and develop your own tests for MUSE. -This is to check that a model reproduces expected results for a defined set of input data. -Tests can be run with the command [pytest](https://docs.pytest.org/en/latest/), from the testing framework of the same name. +If you followed the **Installation for developers** guide, you are likely developing your own tests for MUSE. +These tests ensure that a model produces reproducible results for a defined set of input data. +Tests can be run with the command [pytest](https://docs.pytest.org/en/latest/), using the testing framework of the same name. -Within the ``MUSE-OS`` directory, just run: +To run tests, within the ``MUSE-OS`` directory, activate the virtual environment where you installed ``MUSE`` and run a model as shown in Step 1 below. +This can be your own model or, for simplicity, the default one. +Although this was the final step when installing ``MUSE``, it is mentioned here again because running a model creates a set of files +needed for the tests themselves. In Step 2, you can then run the actual tests, which should all pass before continuing code development. .. code-block:: bash + # Step 1: Run a model + muse --model default + + # Step 2: Run the test scripts python -m pytest Building the documentation From e16cf54768cea78411fed58d5d84d9e446b3c603 Mon Sep 17 00:00:00 2001 From: HarmonicReflux Date: Mon, 8 Jul 2024 15:04:10 +0100 Subject: [PATCH 05/11] Split markdown cell while executing default model to create capacity file that is required for the notebook tests. --- docs/running-muse-example.ipynb | 23 ++++++++++++++++++----- 1 file changed, 18 insertions(+), 5 deletions(-) diff --git a/docs/running-muse-example.ipynb b/docs/running-muse-example.ipynb index 537cc66da..b03c1ba38 100644 --- a/docs/running-muse-example.ipynb +++ b/docs/running-muse-example.ipynb @@ -48,13 +48,26 @@ "\n", "### Programmatic use of MUSE\n", "\n", - "It is also possible to run one of the built-in MUSE models directly in Python using the following code:\n", - "\n", - "```python\n", + "It is also possible to run one of the built-in MUSE models directly in Python using the following code:" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "%%capture\n", "from muse import examples\n", + "\n", "model = examples.model(\"default\")\n", - "model.run()\n", - "```\n", + "model.run()" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ "\n", "The output will be produced in the current working directory, within a `Results` folder. For the case of a custom settings file:\n", "\n", From 4cc1e8e45dfabb74131ca7114cb9021e4ca76cb8 Mon Sep 17 00:00:00 2001 From: HarmonicReflux Date: Mon, 8 Jul 2024 15:11:03 +0100 Subject: [PATCH 06/11] Deleted instructions to run the default model of muse as part of the installation procedure as required file creation to run notebook tests is now fixed. --- docs/installation/developers.rst | 10 +--------- 1 file changed, 1 insertion(+), 9 deletions(-) diff --git a/docs/installation/developers.rst b/docs/installation/developers.rst index 968970370..65ea92f39 100644 --- a/docs/installation/developers.rst +++ b/docs/installation/developers.rst @@ -30,7 +30,6 @@ Once you have your environment created, **activate it** and install MUSE within # 1 - Create a virtual environment # 2 - Activate that virtual environment # 3 - Install MUSE in editable mode with: python -m pip install -e .[dev,doc] - # 4 - Invoke `muse --model default` .. note:: @@ -39,8 +38,6 @@ Once you have your environment created, **activate it** and install MUSE within For example on ``Windows``, the command will read `python -m pip install -e .[dev,doc]`. On Ubuntu Linux, it will be `python -m pip install -e ."[dev,doc]"`. This will install MUSE including its dependencies for development. The downloaded code can be modified and the changes will be automatically reflected in the virtual environment. -Finally, in the activated virtual environment, run `muse --model default` to generate output data that are used to make `pytest` run without failures. - .. note:: The source-code installation will only be accessible whilst the virtual environment is active, and can be called from the command line using the ``muse`` command with the relevant input arguments. @@ -66,15 +63,10 @@ Tests can be run with the command [pytest](https://docs.pytest.org/en/latest/), To run tests, within the ``MUSE-OS`` directory, activate the virtual environment where you installed ``MUSE`` and run a model as shown in Step 1 below. This can be your own model or, for simplicity, the default one. -Although this was the final step when installing ``MUSE``, it is mentioned here again because running a model creates a set of files -needed for the tests themselves. In Step 2, you can then run the actual tests, which should all pass before continuing code development. .. code-block:: bash - # Step 1: Run a model - muse --model default - - # Step 2: Run the test scripts + # Run the test scripts python -m pytest Building the documentation From 9b087066ad4f065031197ffdfdc44dc10017b564 Mon Sep 17 00:00:00 2001 From: HarmonicReflux Date: Mon, 8 Jul 2024 15:20:24 +0100 Subject: [PATCH 07/11] Improved user documentation. --- docs/running-muse-example.ipynb | 18 ++++++++++++------ 1 file changed, 12 insertions(+), 6 deletions(-) diff --git a/docs/running-muse-example.ipynb b/docs/running-muse-example.ipynb index b03c1ba38..cb1597bb9 100644 --- a/docs/running-muse-example.ipynb +++ b/docs/running-muse-example.ipynb @@ -20,7 +20,7 @@ "source": [ "## Running MUSE\n", "\n", - "Once MUSE have been installed, we can run an example. To start with, we will run one of the built-in MUSE examples. We will focus on the terminal version of MUSE as the GUI version should be largely self-explanatory. If you are using MUSE within a virtual environment, make sure you have it activated.\n", + "Once MUSE have been installed, we can run an example. To start with, we will run one of the built-in MUSE examples. We will focus on the terminal version of MUSE as the GUI version should be largely self-explanatory. If you are using MUSE within a virtual environment, make sure you have it activated. \n", "\n", "In all cases, if the installation instructions were followed, you should be able to run the default `muse` example running the following command in the terminal:\n", "\n", @@ -53,23 +53,29 @@ }, { "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], + "metadata": { + "ExecuteTime": { + "end_time": "2024-07-08T14:12:40.418913Z", + "start_time": "2024-07-08T14:12:14.259550Z" + } + }, "source": [ "%%capture\n", "from muse import examples\n", "\n", "model = examples.model(\"default\")\n", "model.run()" - ] + ], + "outputs": [], + "execution_count": 9 }, { "cell_type": "markdown", "metadata": {}, "source": [ + "Note: `%%capture` is a Jupyter magic method that suppresses the output of a cell. Otherwise, one would see an output [like this](example-output.ipynb).\n", "\n", - "The output will be produced in the current working directory, within a `Results` folder. For the case of a custom settings file:\n", + "The results files will be produced in the current working directory, within a `Results` folder. For the case of a custom settings file:\n", "\n", "```python\n", "from logging import getLogger\n", From eca80f19fe3f652f19f9e71281a9974822dbd859 Mon Sep 17 00:00:00 2001 From: "pre-commit-ci[bot]" <66853113+pre-commit-ci[bot]@users.noreply.github.com> Date: Mon, 8 Jul 2024 14:21:53 +0000 Subject: [PATCH 08/11] [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci --- docs/running-muse-example.ipynb | 13 ++++--------- 1 file changed, 4 insertions(+), 9 deletions(-) diff --git a/docs/running-muse-example.ipynb b/docs/running-muse-example.ipynb index cb1597bb9..1339d259a 100644 --- a/docs/running-muse-example.ipynb +++ b/docs/running-muse-example.ipynb @@ -53,21 +53,16 @@ }, { "cell_type": "code", - "metadata": { - "ExecuteTime": { - "end_time": "2024-07-08T14:12:40.418913Z", - "start_time": "2024-07-08T14:12:14.259550Z" - } - }, + "execution_count": null, + "metadata": {}, + "outputs": [], "source": [ "%%capture\n", "from muse import examples\n", "\n", "model = examples.model(\"default\")\n", "model.run()" - ], - "outputs": [], - "execution_count": 9 + ] }, { "cell_type": "markdown", From f00917d7096eb37abdf896fd229ac8eacb4c2d19 Mon Sep 17 00:00:00 2001 From: HarmonicReflux Date: Mon, 8 Jul 2024 16:31:58 +0100 Subject: [PATCH 09/11] Update docs/installation/developers.rst Co-authored-by: Tom Bland --- docs/installation/developers.rst | 1 - 1 file changed, 1 deletion(-) diff --git a/docs/installation/developers.rst b/docs/installation/developers.rst index 65ea92f39..190d953cd 100644 --- a/docs/installation/developers.rst +++ b/docs/installation/developers.rst @@ -62,7 +62,6 @@ These tests ensure that a model produces reproducible results for a defined set Tests can be run with the command [pytest](https://docs.pytest.org/en/latest/), using the testing framework of the same name. To run tests, within the ``MUSE-OS`` directory, activate the virtual environment where you installed ``MUSE`` and run a model as shown in Step 1 below. -This can be your own model or, for simplicity, the default one. .. code-block:: bash From f5ead555f1f61c4b5d36c22f1ea4533f7d4ead4c Mon Sep 17 00:00:00 2001 From: HarmonicReflux Date: Mon, 8 Jul 2024 16:32:40 +0100 Subject: [PATCH 10/11] Update docs/installation/developers.rst Co-authored-by: Tom Bland --- docs/installation/developers.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/installation/developers.rst b/docs/installation/developers.rst index 190d953cd..4ceee6613 100644 --- a/docs/installation/developers.rst +++ b/docs/installation/developers.rst @@ -61,7 +61,7 @@ If you followed the **Installation for developers** guide, you are likely develo These tests ensure that a model produces reproducible results for a defined set of input data. Tests can be run with the command [pytest](https://docs.pytest.org/en/latest/), using the testing framework of the same name. -To run tests, within the ``MUSE-OS`` directory, activate the virtual environment where you installed ``MUSE`` and run a model as shown in Step 1 below. +To run tests, within the ``MUSE-OS`` directory, activate the virtual environment where you installed ``MUSE`` and run: .. code-block:: bash From 20d3e5e6e02091bb9aa6fcd51de9b5c5ea717f7f Mon Sep 17 00:00:00 2001 From: HarmonicReflux Date: Mon, 8 Jul 2024 16:33:03 +0100 Subject: [PATCH 11/11] Update docs/installation/developers.rst Co-authored-by: Tom Bland --- docs/installation/developers.rst | 1 - 1 file changed, 1 deletion(-) diff --git a/docs/installation/developers.rst b/docs/installation/developers.rst index 4ceee6613..03f4c4014 100644 --- a/docs/installation/developers.rst +++ b/docs/installation/developers.rst @@ -65,7 +65,6 @@ To run tests, within the ``MUSE-OS`` directory, activate the virtual environment .. code-block:: bash - # Run the test scripts python -m pytest Building the documentation