diff --git a/.github/workflows/docker_quarto.yaml b/.github/workflows/docker_quarto.yaml index 72128fa6..fa80f286 100644 --- a/.github/workflows/docker_quarto.yaml +++ b/.github/workflows/docker_quarto.yaml @@ -1,4 +1,5 @@ name: Publish docker on GHCR and quarto on GitHub pages +run-name: Publish docker on GHCR and quarto on GitHub pages on: push: diff --git a/.github/workflows/lint_alt_text.yml b/.github/workflows/lint_alt_text.yml index b10f608c..f29b8db5 100644 --- a/.github/workflows/lint_alt_text.yml +++ b/.github/workflows/lint_alt_text.yml @@ -1,4 +1,5 @@ name: Check Alt Text +run-name: Check Alt Text on: push: diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index 1a93c099..3b9d93e6 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -160,6 +160,8 @@ sudo docker image rm -f desrapbookdocker ## Accessibility +This site uses [W3C's Web Accessibility Initiative (WAI) Easy Checks](https://www.w3.org/WAI/test-evaluate/easy-checks/) as a lightweight accessibility framework. Please see this [GitHub issue](https://github.com/pythonhealthdatascience/des_rap_book/issues/188) for a record of which checks are currently met and any known limitations. + Quarto's supported accessibility checker `axe` is not used because it is difficult to read, and it mostly flagged things that cannot be changed (i.e., Quarto defaults and built-ins). As such, other approaches are used. ### `lint_alt_text.py` diff --git a/LICENSE b/LICENSE index 0c1c0854..52d9d6b8 100644 --- a/LICENSE +++ b/LICENSE @@ -1,6 +1,6 @@ MIT License -Copyright (c) 2025 Reproducible Healthcare Simulations in Python and R +Copyright (c) 2026 STARS (Sharing Tools and Artefacts for Reproducible Simulations in healthcare) project team Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal diff --git a/README.md b/README.md index 81ab5ae9..bf0b92d7 100644 --- a/README.md +++ b/README.md @@ -78,6 +78,12 @@ You can also cite the archived version of this work on Zenodo: https://doi.org/1
+## Accessibility + +This site uses [W3C's Web Accessibility Initiative (WAI) Easy Checks](https://www.w3.org/WAI/test-evaluate/easy-checks/) as a lightweight accessibility framework. Please see this [GitHub issue](https://github.com/pythonhealthdatascience/des_rap_book/issues/188) for a record of which checks are currently met and any known limitations. + +
+ ## Contributors If you're interested in contributing (or just viewing this website locally), check out the `CONTRIBUTING.md` file. diff --git a/_quarto.yml b/_quarto.yml index 40df5375..7e259c86 100644 --- a/_quarto.yml +++ b/_quarto.yml @@ -124,7 +124,8 @@ website: - text: | Part of the STARS research project.
Code licence: MIT. - Text licence: CC-BY-SA 4.0. + Text licence: CC-BY-SA 4.0.
+ Changelog right: - text: | Heather, A., Monks, T., Mustafee, N., Harper, A., Alidoost, F., Challen, R., & Slater, T. (2025). DES RAP Book: Reproducible Discrete-Event Simulation in Python and R. https://github.com/pythonhealthdatascience/des_rap_book. https://doi.org/10.5281/zenodo.17094155. diff --git a/dissemination.md b/dissemination.md new file mode 100644 index 00000000..fd357d06 --- /dev/null +++ b/dissemination.md @@ -0,0 +1,24 @@ +# Dissemination / Advertising Plan + +This document outlines how this training resource is and will be shared with its target audience. + +## Publication and formal channels + +* Submit a journal publication describing the resource and its intended use, including a persistent link to the online book. +* Embed and describe the resource within the [HDR UK Futures](https://hdruklearn.org/) training platform, so it appears alongside related training materials. + +## Professional and community networks + +* Share the resource on: + * NHS-OA community Slack. + * RSE community Slack. + * LinkedIn. + * PenARC comms. + * Open Research champions network / UK Reproducibility Network. +* Write a blog post for the NHS-OA community about the resource. + +## Web presence and catalogues + +* Link the resource from the [STARS project site](https://pythonhealthdatascience.github.io/stars/). +* Register in training resource catalogues [TESS](https://tess.elixir-europe.org/) and [Glittr](https://glittr.org/). +* Include on the [Coding For Reproducible Research resource list](https://coding-for-reproducible-research.github.io/CfRR_Courses/resources/resources.html). diff --git a/environment.yaml b/environment.yaml index 8dccec38..ceeb9844 100644 --- a/environment.yaml +++ b/environment.yaml @@ -6,6 +6,7 @@ dependencies: - flake8=7.3.0 - itables=2.5.2 - jupyter=1.1.1 + - nbconvert=7.17.0 - openssl=3.3.0 - pandas=2.3.1 - plotly=6.3.0 diff --git a/pages/changelog.qmd b/pages/changelog.qmd new file mode 100644 index 00000000..f90a17b9 --- /dev/null +++ b/pages/changelog.qmd @@ -0,0 +1,18 @@ +--- +pagetitle: Changelog +--- + +{{< include /CHANGELOG.md >}} + + + diff --git a/pages/guide/environment-full.yaml b/pages/guide/environment-full.yaml index aa2b0996..e2dbca34 100644 --- a/pages/guide/environment-full.yaml +++ b/pages/guide/environment-full.yaml @@ -6,6 +6,7 @@ dependencies: - flake8=7.3.0 - itables=2.5.2 - jupyter=1.1.1 + - nbconvert-core=7.16.6 - openssl=3.3.0 - pandas=2.3.1 - plotly=6.3.0 diff --git a/pages/guide/experiments/full_run.qmd b/pages/guide/experiments/full_run.qmd index a651cc08..88794952 100644 --- a/pages/guide/experiments/full_run.qmd +++ b/pages/guide/experiments/full_run.qmd @@ -65,17 +65,37 @@ A bash script is a plain text file that contains a list of terminal commands, wr ### Where to put the script? -Save the script below as `run_notebooks.sh` in the root of your project, alongside the `notebooks/` folder. - ::: {.python-content} -This allows the script to loop over `rmarkdown/*.Rmd` without extra path adjustments. +Save the script below as `run_notebooks.sh` in the project root - for example: + +``` +├── run_notebooks.sh +├── notebooks/ +| └── ... +├── simulation/ +| └── ... +├── ... +``` + +This allows the script to loop over all `.ipynb` in the `notebooks/` folder using `notebooks/*.ipynb` without extra path adjustments. ::: ::: {.r-content} -This allows the script to loop over `notebooks/*.ipynb` without extra path adjustments. +Save the script below as `run_rmarkdown.sh` in the project root - for example: + +``` +├── run_rmarkdown.sh +├── R/ +| └── ... +├── rmarkdown/ +| └── ... +├── ... +``` + +This allows the script to loop over all `.Rmd` files in the `rmarkdown/` folder using `rmarkdown/*.Rmd` without extra path adjustments. ::: @@ -140,6 +160,17 @@ CONDA_JUPYTER=$(dirname "$(which python)")/jupyter This line finds the `jupyter` executable linked to your current python environment. This ensures it uses the correct version of Jupyter (i.e., instead of trying to run a system-wide or different environment's Jupyter). +::: {.callout-note title="Alternative path for Windows users"} + +On Windows, `jupyter.exe` is usually in the `Scripts` folder of your conda environment, not directly next to `python.exe`. For example, if `which python` returns `envs/des-example/python.exe`, then `jupyter.exe` is typically at `env/des-example/Scripts/jupyter.exe`. +You will want to change the line above to: + +```{.bash} +CONDA_JUPYTER=$(dirname "$(which python)")/Scripts/jupyter` +``` + +::: +
```{.bash} @@ -186,15 +217,17 @@ The Jupyter nbconvert command (a single command, though split across several lin * `"${CONDA_JUPYTER}" nbconvert`: Execute nbconvert from your environment. * `--to notebook --inplace --execute`: Run the notebook, overwrite in place, and execute all cells from scratch. + * By using `--inplace`, nbconvert overwrites the existing notebook file with the executed version. If you prefer to keep the original notebook and write the executed result to a new file, remove `--inplace` and use `--output`. + * You can also change `--to notebook` to another format (e.g., `--to html` generates HTML). * `--ClearMetadataPreprocessor.enabled=True`: Enable pre-processor that can clear notebook metadata. * `--ClearMetadataPreprocessor.clear_notebook_metadata=True`: Remove notebook metadata. * `"$nb"`: Specifies the notebook file to run. ::: {.callout title="Why do we have the metadata settings?" collapse="true"} -These are to **avoid changes in files that have been re-run**, when all the results and code are still the same, **just the notebook metadata has changed**. +Jupyter stores a lot of extra metadata in notebooks (kernel name, environment IDs, timing info, UI state, etc.). These fields often change even when the code and outputs are identical, which creates noisy diffs in version control. The metadata options in the command tell nbconvert to strip most of this metadata so that the file only changes when the code or outputs really change. -With these settings in our bash script, the output notebook metadata will be: +*Note: One side effect is that clearing all notebook metadata can change how the Jupyter interface behaves (e.g., some menu items like "Clear All Outputs" may disappear or look different), because the frontend relies on metadata like the kernelspec and language information.* ```{.bash} { @@ -331,13 +364,14 @@ if __name__ == "__main__": main() ``` -These can then be easily run from bash scripts like so: +These can then be easily run from the command line: ```{.bash} -#!/bin/bash python filename.py ``` +You could put your whole pipeline inside a single `main()` function and avoid bash scripts altogether. However, a separate bash script becomes useful when you want to mix different tools (e.g., `.ipynb` and `.py`) and languages in one pipeline. + ::: ::: {.r-content} @@ -406,6 +440,8 @@ You can run this R script from a Bash script or terminal with the following comm Rscript analysis/filename.R ``` +You could call your whole pipeline from inside a single R script and avoid bash scripts altogether. However, a separate bash script becomes useful when you want to mix different tools (e.g., `.Rmd` and `R` files) and languages in one pipeline. + ::: ## Literate programming @@ -430,7 +466,7 @@ Rscript analysis/filename.R :::: -This style of working can be really handy for **reproducibility**, a all the parameter choices, steps and outputs are documented alongside explanatory text. +This style of working can be really handy for **reproducibility**, as all the parameter choices, steps and outputs are documented alongside explanatory text. There are several common ways to use literate programming in research workflows: diff --git a/pages/guide/experiments/scenarios.qmd b/pages/guide/experiments/scenarios.qmd index ee785fbf..a1217c68 100644 --- a/pages/guide/experiments/scenarios.qmd +++ b/pages/guide/experiments/scenarios.qmd @@ -1,5 +1,7 @@ --- title: Scenario and sensitivity analysis +filters: + - line-highlight bibliography: scenarios_resources/references.bib date: "2025-11-06T10:44:04+00:00" --- @@ -36,17 +38,24 @@ This page continues on from: [Parallel processing](../output_analysis/parallel.q **Required packages:** -These should be available from environment setup on the [Structuring as a package](/pages/guide/setup/package.qmd) page. +The following imports are required. These should be available from environment setup on the [Structuring as a package](/pages/guide/setup/package.qmd) page. + +We have one new package: `itertools`. ::: {.python-content} ```{python} -import itertools - +# For display purposes: from IPython.display import HTML from itables import to_html_datatable + +# For the simulation modelling and analysis: +import itertools#<< +import time +from joblib import Parallel, delayed, cpu_count import numpy as np import pandas as pd +import plotly.express as px import scipy.stats as st import simpy from sim_tools.distributions import Exponential @@ -78,7 +87,7 @@ library(tidyr) ## Running and sharing scenarios -When creating a DES, you won't just run one model - you run scenarios, which are different parameter set-ups. These are used to test how outcomes change under varying conditions. +When creating a DES, you won't just run one model - you run scenarios, which can involve different parameter set‑ups and/or changes to the process flow or simulation logic (e.g., alternative configurations of resources, routing, or priority rules). These are used to test how outcomes change under varying conditions. When running scenarios, there are two important things to remember: @@ -88,7 +97,14 @@ When running scenarios, there are two important things to remember: ::: -Don't duplicate whole scripts just to test different set-ups. The idea is to build the model with functions and classes so you only need to change the parameters and re-run. That's why we guide you through building the model this way in this book. +**Don't duplicate whole scripts** just to test different set-ups. + +Using functions and classes is important here, as: + +* If your scenarios only change **parameters**, you just run the same model with different inputs. +* If your scenarios change the **configuration** of the system (e.g., routing or scheduling), you can create alternative methods, functions, or configuration objects. + +On this page, we only show examples where scenarios differ by parameters. ::: {.cream} @@ -378,7 +394,11 @@ scenario_results = run_scenarios( "number_of_doctors": [3, 4, 5]}, param_factory=Parameters ) +``` + +We'll save results to CSV - you can change the file path listed, or otherwise, you'll just need to create the `scenarios_resources` directory first. +```{python} # Save to CSV scenario_results.to_csv( "scenarios_resources/python_scenario_results.csv", index=False @@ -452,7 +472,11 @@ sensitivity_results = run_scenarios( scenarios={"interarrival_time": [4, 4.5, 5, 5.5, 6, 6.5, 7]}, param_factory=Parameters ) +``` +We'll save results to CSV - you can change the file path listed, or otherwise, you'll just need to ensure you have already made the `scenarios_resources` directory first. + +```{python} # Save to CSV sensitivity_results.to_csv( "scenarios_resources/python_sensitivity_results.csv", index=False @@ -483,6 +507,12 @@ kable(sensitivity_results) |> scroll_box(height = "400px") ::: +::: {.callout-note title="Sensitivity analysis across scenarios"} + +In practice, sensitivity analysis is often performed within each scenario - for example, you might define several scenarios (different configurations of arrivals, capacity, or routing) and then vary a key parameter (such as interarrival time) around each of them. In this book, for simplicity, we show sensitivity analysis for a single scenario at a time, and you can repeat the same approach for any other plausible scenarios you define. + +::: + ## Saving results Saving your simulation results to file is important for reproducility, as it allows others to verify your findings and generate consistent (or new) figures and analyses, even if they can't re-run your simulation. diff --git a/pages/guide/experiments/tables_figures.qmd b/pages/guide/experiments/tables_figures.qmd index 98755b47..663cf868 100644 --- a/pages/guide/experiments/tables_figures.qmd +++ b/pages/guide/experiments/tables_figures.qmd @@ -1,5 +1,7 @@ --- title: Tables and figures +filters: + - line-highlight bibliography: scenarios_resources/references.bib date: "2025-10-24T10:55:25+01:00" --- @@ -32,21 +34,30 @@ This page uses the results saved on the [Scenario and sensitivity analysis](scen **Required packages:** -These should be available from environment setup on the [Structuring as a package](/pages/guide/setup/package.qmd) page. +The following imports are required. These should be available from environment setup on the [Structuring as a package](/pages/guide/setup/package.qmd) page. ::: {.python-content} -```{python} -import os +On this page, we have three new imports: `os`, `matplotlib.colors` and `plotly.graph_objects`. +```{python} +# For display purposes: from IPython.display import HTML from itables import to_html_datatable -import matplotlib.colors as mcolors + +# For the simulation modelling and analysis: +import itertools +import os#<< +import time +from joblib import Parallel, delayed, cpu_count +import matplotlib.colors as mcolors#<< import numpy as np import pandas as pd import plotly.express as px -import plotly.graph_objects as go +import plotly.graph_objects as go#<< import scipy.stats as st +import simpy +from sim_tools.distributions import Exponential ``` ```{python} @@ -149,7 +160,6 @@ It uses the `summary_stats()` function introduced on the [Replications](../outpu ::: {.callout-note title="View `summary_stats()`" collapse="true"} ```{python} -#| echo: false #| file: ../output_analysis/replications_resources/summary_stats.py ``` @@ -206,6 +216,8 @@ def summarise_scenarios(results, groups, result_vars, path_prefix=None): ### Scenario analysis +Here and throughout page, we will save results to a folder called `tables_figures_resources`, and then a subfolder (e.g., `python_scenario/`). You can rename this path (e.g. to `os.path.join("results", "scenario"))`), or otherwise, create the listed directory structure (here, `tables_figures_resources/python_scenario/`). + ```{python} result_variables = [ "mean_wait_time", @@ -225,6 +237,8 @@ scenario_tables = summarise_scenarios( ) ``` +Click the name of each metric below to view the relevant dataframe: + ::: {.panel-tabset} ## Mean wait time @@ -270,6 +284,8 @@ sensitivity_tables = summarise_scenarios( ) ``` +Click the name of each metric below to view the relevant dataframe: + ::: {.panel-tabset} ## Mean wait time @@ -370,6 +386,8 @@ scenario_tables <- summarise_scenarios( ) ``` +Click the name of each metric below to view the relevant table: + ::: {.panel-tabset} ## Mean wait time @@ -415,6 +433,8 @@ sensitivity_tables <- summarise_scenarios( ) ``` +Click the name of each metric below to view the relevant table: + ::: {.panel-tabset} ## Mean wait time @@ -457,6 +477,8 @@ kable(sensitivity_tables[["mean_patients_in_system"]]) These functions are used to plot the results from the scenarios and sensitivity analysis. +We have used [plotly.express](https://plotly.com/python/plotly-express/) for our plots. + ```{python} def pale_colour(colour, alpha=0.2): """ @@ -620,6 +642,8 @@ scenario_plots = plot_metrics( ) ``` +Click the name of each metric below to view the relevant plot: + ::: {.panel-tabset} ## Mean wait time @@ -654,56 +678,14 @@ scenario_plots["mean_patients_in_system"] ::: -### Sensitivity analysis - -```{python} -sensitivity_plots = plot_metrics( - summary_tables=sensitivity_tables, - name_mappings=name_mappings, - path_prefix=os.path.join("tables_figures_resources", "python_sensitivity") -) -``` - -::: {.panel-tabset} - -## Mean wait time - -```{python} -sensitivity_plots["mean_wait_time"] -``` - -## Mean utilisation - -```{python} -sensitivity_plots["mean_utilisation_tw"] -``` - -## Mean queue length - -```{python} -sensitivity_plots["mean_queue_length"] -``` - -## Mean time in system - -```{python} -sensitivity_plots["mean_time_in_system"] -``` - -## Mean patients in system - -```{python} -sensitivity_plots["mean_patients_in_system"] -``` - -::: - :::: ::: {.r-content} This function is used to plot the results from the scenarios and sensitivity analysis. +We have used [ggplot2](https://ggplot2.tidyverse.org/) for our plots. + ```{r} #' Plot multiple performance measures at once #' @@ -829,6 +811,8 @@ scenario_plots <- plot_metrics( ) ``` +Click the name of each metric below to view the relevant plot: + ::: {.panel-tabset} ## Mean wait time @@ -863,7 +847,72 @@ scenario_plots[["mean_patients_in_system"]] ::: -These plots look a little funny simple due to the wide confidence intervals - with very short run times and few replications in our illustrative example, the confidence intervals are very wide! +:::: + +In the figure, each line shows the result (e.g., mean wait time for a doctor) as the patient inter-arrival time increases. The different coloured lines correspond to a different number of doctors. The shaded bands around each line represent 95% confidence intervals for the mean. + +These plots can look a little funny simple due to the wide confidence intervals - with very short run times and few replications in our illustrative example, the confidence intervals are very wide! + +::: {.callout-note title="Good practice for plots"} + +* Use consistent names for variables on your axis labels. +* Include units in axis labels (e.g., minutes, days, patients per hour). +* Make sure to plot uncertainty (e.g., confidence intervals, percentile range). +* Keep colour and line style consistent across related figures. + +::: + +:::: {.python-content} + +### Sensitivity analysis + +```{python} +sensitivity_plots = plot_metrics( + summary_tables=sensitivity_tables, + name_mappings=name_mappings, + path_prefix=os.path.join("tables_figures_resources", "python_sensitivity") +) +``` + +Click the name of each metric below to view the relevant plot: + +::: {.panel-tabset} + +## Mean wait time + +```{python} +sensitivity_plots["mean_wait_time"] +``` + +## Mean utilisation + +```{python} +sensitivity_plots["mean_utilisation_tw"] +``` + +## Mean queue length + +```{python} +sensitivity_plots["mean_queue_length"] +``` + +## Mean time in system + +```{python} +sensitivity_plots["mean_time_in_system"] +``` + +## Mean patients in system + +```{python} +sensitivity_plots["mean_patients_in_system"] +``` + +::: + +:::: + +:::: {.r-content} ### Sensitivity analysis @@ -876,6 +925,8 @@ sensitivity_plots <- plot_metrics( ) ``` +Click the name of each metric below to view the relevant plot: + ::: {.panel-tabset} ## Mean wait time @@ -912,6 +963,8 @@ sensitivity_plots[["mean_patients_in_system"]] :::: +In this figure, the blue line shows the result (e.g., mean wait time) as we increase the patient inter-arrival time. The shaded band around the line is a 95% confidence interval, reflecting uncertainty due to random variation between simulation runs. + ## Explore the example models
diff --git a/pages/guide/further_info/conclusion.qmd b/pages/guide/further_info/conclusion.qmd index ac4e7f1e..bcafe83f 100644 --- a/pages/guide/further_info/conclusion.qmd +++ b/pages/guide/further_info/conclusion.qmd @@ -105,7 +105,7 @@ Suggested citation: This book is part of the **STARS (Sharing Tools and Artefacts for Reusable and Reproducible Simulations)** project, supported by the Medical Research Council [grant number MR/Z503915/1] from 1st May 2024 to 31st October 2026. -[![](../../images/stars_banner.png)](https://pythonhealthdatascience.github.io/stars/) +[![](/images/stars_banner.png)](https://pythonhealthdatascience.github.io/stars/) STARS tackles the challenges of sharing, reusing, and reproducing discrete event simulation (DES) models in healthcare. Our goal is to create open resources using the two most popular open-source languages for DES: Python and R. As part of this project, you'll find tutorials, code examples, and tools to help researchers and practitioners develop, validate, and share DES models more effectively. diff --git a/pages/guide/further_info/feedback.qmd b/pages/guide/further_info/feedback.qmd index 3fe75ebe..e12355f7 100644 --- a/pages/guide/further_info/feedback.qmd +++ b/pages/guide/further_info/feedback.qmd @@ -13,7 +13,7 @@ We are excited to hear from people who have used this book. Your feedback and ex ## How to comment -At the bottom of every page in this book, you'll find a comment section powered by **Giscus**. After signing in with your GitHub account, you can: +At the bottom of every page in this book, you'll find a comment section. After signing in with your GitHub account, you can: * Comment on individual pages to share insights, ask questions, or flag typos/bugs. * Leave a comment here (or on the home/conclusion pages) to give feedback about the book overall. @@ -24,7 +24,7 @@ At the bottom of every page in this book, you'll find a comment section powered ## Want to contribute? -Leave a comment via Giscus. +Leave a comment via the comment section. Alternatively, you can open a GitHub issue for bugs, feature requests, or suggestions: [github.com/pythonhealthdatascience/des_rap_book/issues](https://github.com/pythonhealthdatascience/des_rap_book/issues) diff --git a/pages/guide/inputs/input_data_resources/input_files.png b/pages/guide/inputs/input_data_resources/input_files.png index 0b64c0b0..f990b0cd 100644 Binary files a/pages/guide/inputs/input_data_resources/input_files.png and b/pages/guide/inputs/input_data_resources/input_files.png differ diff --git a/pages/guide/inputs/input_data_resources/input_files.svg b/pages/guide/inputs/input_data_resources/input_files.svg index b226c723..9252a13e 100644 --- a/pages/guide/inputs/input_data_resources/input_files.svg +++ b/pages/guide/inputs/input_data_resources/input_files.svg @@ -1,90 +1,66 @@ - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + \ No newline at end of file diff --git a/pages/guide/inputs/input_data_resources/input_rap.png b/pages/guide/inputs/input_data_resources/input_rap.png index de71ce29..8cf72614 100644 Binary files a/pages/guide/inputs/input_data_resources/input_rap.png and b/pages/guide/inputs/input_data_resources/input_rap.png differ diff --git a/pages/guide/inputs/input_data_resources/input_rap.svg b/pages/guide/inputs/input_data_resources/input_rap.svg index 50552df7..81b3bfa2 100644 --- a/pages/guide/inputs/input_data_resources/input_rap.svg +++ b/pages/guide/inputs/input_data_resources/input_rap.svg @@ -4,142 +4,98 @@ - + - + - - + + - - - - - - - - - + + - - - + - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - + + - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - + + + + + - + - - - - - - - + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + \ No newline at end of file diff --git a/pages/guide/inputs/input_modelling.qmd b/pages/guide/inputs/input_modelling.qmd index 34817365..86b47ead 100644 --- a/pages/guide/inputs/input_modelling.qmd +++ b/pages/guide/inputs/input_modelling.qmd @@ -21,7 +21,7 @@ date: "2025-11-03T14:30:28+00:00" **Required packages:** -These should be available from environment setup on the [Structuring as a package](/pages/guide/setup/package.qmd) page. +The following imports are required. These should be available from environment setup on the [Structuring as a package](/pages/guide/setup/package.qmd) page. ::: {.python-content} diff --git a/pages/guide/inputs/parameters_file.qmd b/pages/guide/inputs/parameters_file.qmd index d21855f5..e6d37b15 100644 --- a/pages/guide/inputs/parameters_file.qmd +++ b/pages/guide/inputs/parameters_file.qmd @@ -28,7 +28,7 @@ date: "2025-11-06T10:43:42+00:00" **Required packages:** -These should be available from environment setup on the [Structuring as a package](/pages/guide/setup/package.qmd) page. +The following imports are required. These should be available from environment setup on the [Structuring as a package](/pages/guide/setup/package.qmd) page. ::: {.python-content} diff --git a/pages/guide/introduction.qmd b/pages/guide/introduction.qmd index 7fa67db2..03cd280c 100644 --- a/pages/guide/introduction.qmd +++ b/pages/guide/introduction.qmd @@ -151,7 +151,7 @@ Work through the guide first to understand the fundamentals, then refer to the e This is an openly available resource and we cannot guarantee one‑to‑one support for all learners. However, if you have questions or feedback about the material, you can: -* Leave a comment via the Giscus comment section at the bottom of each page. +* Leave a comment via the comment section at the bottom of each page. * Open an issue on the project's GitHub repository. * Start or contribute to a discussion in the GitHub Discussions area to ask questions or share ideas with others using the book. @@ -159,7 +159,7 @@ We do not provide a dedicated support email address for this resource. Instead, ## Reporting problems -If you spot a problem with the training material (for example, a bug in the code, a typo, or something that is unclear), please open an issue in the GitHub repository (or leave a comment via Giscus). Include as much detail as you can (page link, code snippet, error message). We label, prioritise, and track these issues in GitHub and close them once a fix has been implemented. +If you spot a problem with the training material (for example, a bug in the code, a typo, or something that is unclear), please open an issue in the GitHub repository (or leave a comment via the comment section at the bottom of each page). Include as much detail as you can (page link, code snippet, error message). We label, prioritise, and track these issues in GitHub and close them once a fix has been implemented.
diff --git a/pages/guide/model/distributions.qmd b/pages/guide/model/distributions.qmd index 930485e0..4e36e606 100644 --- a/pages/guide/model/distributions.qmd +++ b/pages/guide/model/distributions.qmd @@ -21,7 +21,7 @@ date: "2025-11-05T14:14:21+00:00" **Required packages:** -These should be available from environment setup on the [Structuring as a package](/pages/guide/setup/package.qmd) page. +The following imports are required. These should be available from environment setup on the [Structuring as a package](/pages/guide/setup/package.qmd) page. ::: {.python-content} diff --git a/pages/guide/model/distributions_resources/prng.png b/pages/guide/model/distributions_resources/prng.png deleted file mode 100644 index 9193e143..00000000 Binary files a/pages/guide/model/distributions_resources/prng.png and /dev/null differ diff --git a/pages/guide/model/distributions_resources/prng.svg b/pages/guide/model/distributions_resources/prng.svg deleted file mode 100644 index e498f3d9..00000000 --- a/pages/guide/model/distributions_resources/prng.svg +++ /dev/null @@ -1,145 +0,0 @@ - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - \ No newline at end of file diff --git a/pages/guide/model/logs.qmd b/pages/guide/model/logs.qmd index bc995fff..d17d09b2 100644 --- a/pages/guide/model/logs.qmd +++ b/pages/guide/model/logs.qmd @@ -24,7 +24,7 @@ date: "2025-11-03T10:48:30+00:00" **Pre-reading:** -This page continues on from: [Entity processing](../model/process.qmd). +This page continues on from: [Entity processing](process.qmd). > *Entity generation → Entity processing → Logging* @@ -34,18 +34,20 @@ This page continues on from: [Entity processing](../model/process.qmd). **Required packages:** -These should be available from environment setup on the [Structuring as a package](/pages/guide/setup/package.qmd) page. +The following imports are required. These should be available from environment setup on the [Structuring as a package](/pages/guide/setup/package.qmd) page. + +Since the [entity processing](process.qmd) page, we have add several new imports for logging: ::: {.python-content} ```{python} -import logging -import os -from pprint import pformat +import logging#<< +import os#<< +from pprint import pformat#<< import numpy as np -from rich.logging import RichHandler -from rich.console import Console +from rich.logging import RichHandler#<< +from rich.console import Console#<< import simpy from sim_tools.distributions import Exponential ``` diff --git a/pages/guide/model/patients.qmd b/pages/guide/model/patients.qmd index 2e3b995c..61d280e8 100644 --- a/pages/guide/model/patients.qmd +++ b/pages/guide/model/patients.qmd @@ -34,7 +34,7 @@ As this builds on concepts introduced in prior pages, we suggest first reading: **Required packages:** -These should be available from environment setup on the [Structuring as a package](/pages/guide/setup/package.qmd) page. +The following imports are required. These should be available from environment setup on the [Structuring as a package](/pages/guide/setup/package.qmd) page. ::: {.python-content} @@ -159,6 +159,21 @@ class Patient: We'll also create a class for the model. This class is explained line by line below. +::: {.callout-note title="Imports"} + +The code within this book is shown as it runs inside a Quarto document, where modules are not arranged as a Python package and relative imports are not used. + +In a package-based setup (as described on the [Structuring as a package](/pages/guide/setup/package.qmd) page), each module should include the appropriate imports (including local imports). For example: + +```{.python} +import simpy +from .patient import Patient +``` + +A complete list of external imports is shown at the top of the page; when structuring your code as a package, you will also need to add the corresponding local imports in each module. + +::: + ```{python} class Model: """ diff --git a/pages/guide/model/process.qmd b/pages/guide/model/process.qmd index 96057696..9f618d08 100644 --- a/pages/guide/model/process.qmd +++ b/pages/guide/model/process.qmd @@ -25,7 +25,7 @@ This page continues on from: [Entity generation](patients.qmd). **Required packages:** -These should be available from environment setup on the [Structuring as a package](/pages/guide/setup/package.qmd) page. +The following imports are required. These should be available from environment setup on the [Structuring as a package](/pages/guide/setup/package.qmd) page. ::: {.python-content} diff --git a/pages/guide/output_analysis/length_warmup.qmd b/pages/guide/output_analysis/length_warmup.qmd index c65cdb7b..82a864b9 100644 --- a/pages/guide/output_analysis/length_warmup.qmd +++ b/pages/guide/output_analysis/length_warmup.qmd @@ -30,19 +30,22 @@ This page continues on from: [Replications](replications.qmd). **Required packages:** -These should be available from environment setup on the [Structuring as a package](/pages/guide/setup/package.qmd) page. +The following imports are required. These should be available from environment setup on the [Structuring as a package](/pages/guide/setup/package.qmd) page. + +On this page, we add two new imports for plotting: `plotly.express` and `plotly.graph_objects`. ::: {.python-content} ```{python} -import os - +# For display purposes: from IPython.display import HTML from itables import to_html_datatable + +# For the simulation modelling and analysis: import numpy as np import pandas as pd -import plotly.express as px -import plotly.graph_objects as go +import plotly.express as px#<< +import plotly.graph_objects as go#<< import scipy.stats as st import simpy from sim_tools.distributions import Exponential @@ -972,6 +975,8 @@ def plot_time_series(audit_results, metric, warmup_time=None): return fig ``` +Click the name of each metric below to view the relevant plot: + ::: {.panel-tabset} ## Mean wait time @@ -1036,6 +1041,8 @@ plots <- time_series_inspection( ) ``` +Click the name of each metric below to view the relevant plot: + ::: {.panel-tabset} ## Mean wait time diff --git a/pages/guide/output_analysis/n_reps.qmd b/pages/guide/output_analysis/n_reps.qmd index 63cbe611..f62659b3 100644 --- a/pages/guide/output_analysis/n_reps.qmd +++ b/pages/guide/output_analysis/n_reps.qmd @@ -29,25 +29,29 @@ This page continues on from: [Replications](replications.qmd). **Required packages:** -These should be available from environment setup on the [Structuring as a package](/pages/guide/setup/package.qmd) page. +The following imports are required. These should be available from environment setup on the [Structuring as a package](/pages/guide/setup/package.qmd) page. + +Since [Replications](replications.qmd), we have several new imports: functions from `typing` and from `sim_tools.output_analysis`. ::: {.python-content} ```{python} -from typing import Protocol, runtime_checkable - +# For display purposes: from IPython.display import HTML from itables import to_html_datatable + +# For the simulation modelling and analysis: +from typing import Protocol, runtime_checkable#<< import numpy as np import pandas as pd import scipy.stats as st import simpy from sim_tools.distributions import Exponential -from sim_tools.output_analysis import ( - confidence_interval_method, - plotly_confidence_interval_method, - ReplicationsAlgorithm -) +from sim_tools.output_analysis import (#<< + confidence_interval_method,#<< + plotly_confidence_interval_method,#<< + ReplicationsAlgorithm#<< +)#<< ``` ```{python} @@ -173,7 +177,17 @@ HTML(to_html_datatable(result["run"])) We pass these results to the `confidence_interval_method` function from the [sim-tools](https://github.com/sim-tools/sim-tools) package. -The `desired_precision` parameter is set to 0.1, which means we are interested in identifying when the percentage deviation of the confidence interval from the mean falls **below 10%**. *Note: This threshold is our default but relatively arbitrary. We are just using it to help us identify the point where results are stable enough.* +The `desired_precision` parameter is set to 0.1, which means we are interested in identifying when the percentage deviation of the confidence interval from the mean falls **below 10%**. + +::: {.callout-note title="Stopping condition: `desired_precision` and confidence intervals"} + +We construct a confidence interval for each performance measure at a chosen confidence level (typically 95%). The interval has a half‑width, which is the margin of error around the estimated mean. The `desired_precision` parameter sets how small this half‑width must be relative to the mean (for example, 0.1 means “within 10% of the mean”). + +The confidence interval method keeps increasing the number of replications until this relative half‑width falls below `desired_precision` (and is above `min_rep`), indicating that additional replications are unlikely to change the estimate by more than the chosen percentage. + +**Note:** The 10% threshold used is our default, but is relatively arbitrary. We are just using it to help us identify the point where results are stable enough. + +::: ```{python} # Define metrics to inspect @@ -196,6 +210,8 @@ For each metric, the function returns: 1. The number of replications required for **deviation to fall below 10%** (and must also be **higher than `min_rep`** set in `confidence_interval_method()`). 2. A dataframe of cumulative calculations: the mean, cumulative mean, standard deviation, confidence interval bounds, and the percentage deviation of the confidence interval from the mean (displayed in the far-right column). +Click the name of each metric below to view the relevant results: + ::: {.panel-tabset} ## Mean wait time @@ -254,6 +270,8 @@ HTML(to_html_datatable(confint_results["mean_patients_in_system"][1])) To visualise when precision is achieved and check for stability, we can use the `plotly_confidence_interval_method` function from [sim-tools](https://github.com/sim-tools/sim-tools): +Click the name of each metric below to view the relevant plot: + ::: {.panel-tabset} ## Mean wait time @@ -508,6 +526,8 @@ for (m in metrics) { } ``` +Click the name of each metric below to view the relevant plot: + ::: {.panel-tabset} ## Mean wait time @@ -597,11 +617,11 @@ Here's how it works: ### Model adapter -The `ReplicationsAlgorithm` work with many different simulation models, as long as they all present the same interface for running a single replication. This common interface is defined by the defined by the `ReplicationsAlgorithmModelAdapter` protocol in `sim-tools`. +The `ReplicationsAlgorithm` is designed to work with many different simulation models. To keep it general, it doesn't know any model‑specific details; instead, it expects each model to provide the same simple interface: a `single_run(replication_number)` function that runs on replication and returns a dictionary of metric values. -A `Protocol` describes the expected methods and attributes an object must provide. In this case, the protocol states that a compatible model must implement a `single_run(replication_number)` method returning a dictionary of metric values. +The `ReplicationsAlgorithmModelAdapter` Protocol describes this expected interface. Our `SimulationAdapter` class is a small wrapper that adapts our existing `Runner` and `Parameter` classes so that they match this interface. This is an example of the adapter pattern from software design: we keep the algorithm code unchanged, and just adapt different models to plug into it. -It is tagged `@runtime_checkable` so that, in addition to helping static type checkers, the code can also perform runtime checks (for example using `isinstance`) to verify that an object really satisfies the protocol's interface. +> **Optional detail (for the curious): The Protocol is tagged with `@runtime_checkable`. You don't need to worry about this to use the algorithm but, if you're interested, it means Python can check at runtime whether an object follows this interface (for example, using `isinstance`), as well as helping static type checkers. ```{python} @runtime_checkable @@ -721,6 +741,8 @@ HTML(to_html_datatable(summary_frame)) Again, we can use `plotly_confidence_interval_method` to create plots. +Click the name of each metric below to view the relevant plot: + ::: {.panel-tabset} ## Mean wait time @@ -1440,6 +1462,8 @@ res$summary_table |> We can use the function `plot_replication_ci` from above again to create the plots. +Click the name of each metric below to view the relevant plot: + ::: {.panel-tabset} ## Mean wait time diff --git a/pages/guide/output_analysis/outputs.qmd b/pages/guide/output_analysis/outputs.qmd index 395aff60..32fafd63 100644 --- a/pages/guide/output_analysis/outputs.qmd +++ b/pages/guide/output_analysis/outputs.qmd @@ -35,13 +35,20 @@ This page continues on from: [Initialisation bias](warmup.qmd). **Required packages:** -These should be available from environment setup on the [Structuring as a package](/pages/guide/setup/package.qmd) page. +The following imports are required. These should be available from environment setup on the [Structuring as a package](/pages/guide/setup/package.qmd) page. + +On this page, we add `pandas`. We also import from `IPython` and `itables`, but that is just for displaying tables nicely within the book. ::: {.python-content} ```{python} +# For display purposes:#<< +from IPython.display import HTML#<< +from itables import to_html_datatable#<< + +# For the simulation modelling and analysis:#<< import numpy as np -import pandas as pd +import pandas as pd#<< import simpy from sim_tools.distributions import Exponential ``` @@ -357,6 +364,298 @@ At the moment, `result[["run_results"]]` is pretty empty (just has the replicati ::: +## Overview of performance measures + +On this page, we breakdown how to add several different performance measures to your model. These include: + +* Total arrivals +* Mean wait time +* Mean time in consultation +* Mean doctor utilisation +* Mean queue length +* Mean time in system +* Mean number of patients in the system +* Backlogged patient count +* Backlogged patient mean wait time + +We introduce these on-by-one below, with each measure added and the required code changes clearly highlighted so it is easy to see what is needed for each statistic. + +::: {.pale-blue} + +**If you would prefer to work from a complete example with all of the measures implemented together, you can find that in the following callout:** + +::: + +:::: {.callout title="▶  View how to record all performance measures" collapse="true"} + +Here, we just provide a brief description of the changes. For a full explanation of how these measures are implementation, please refer to the information on individual measures below. + +::: {.python-content} + +**Note:** There are two approaches to recording utilisation - here, we have used the time-weighted approach (approach 2). + +### Patient class + +There are three new attributes: + +* `wait_time` - used for mean wait time. +* `time_with_doctor` - used for mean time in consultation, backlogged patient count and backlogged patient mean wait time. +* `end_time` - used for mean time in system. + +```{python} +{{< include outputs_resources/all_measures/patient.py >}} +``` + +### Monitored resource class + +A new class `MonitoredResource` is used for mean doctor utilisation and mean queue length. + +```{python} +{{< include outputs_resources/all_measures/monitored_resource.py >}} +``` + +### Model class + +There are several new attributes and a method used for mean number of patients in the system (`area_n_in_system`, `time_last_n_in_system`, `n_in_system`, `update_n_in_system()`). + +The doctor is now a `MonitoredResource`, as required for mean doctor utilisation and mean queue length. + +During `consultation()`, the patient's `wait_time` is recorded for mean wait time. Time with the doctor is recorded for mean time in consultation, backlogged patient count and backlogged patient mean wait time. End time is recorded for mean time in system. + +```{python} +{{< include outputs_resources/all_measures/model.py >}} +``` + +### Runner class + +In the Runner class, we calculate each of the metrics... + +```{python} +{{< include outputs_resources/all_measures/runner.py >}} +``` + +### Run the model + +```{python} +runner = Runner(param=Parameters()) +result = runner.run_single(run=0) +HTML(to_html_datatable(result["patient"])) +print(result["run"]) +``` + +::: + +::: {.r-content} + +### Imports + +There are several new imports required. + +```{.r} +#' @keywords internal +"_PACKAGE" + +## usethis namespace: start +#' @importFrom dplyr arrange bind_cols bind_rows desc filter group_by lead#<< +#' @importFrom dplyr left_join mutate n n_distinct row_number slice summarise#<< +#' @importFrom dplyr transmute ungroup#<< +#' @importFrom rlang .data +#' @importFrom simmer add_generator add_resource get_attribute get_mon_arrivals#<< +#' @importFrom simmer get_mon_resources get_queue_count now release run seize#<< +#' @importFrom simmer set_attribute simmer timeout trajectory#<< +#' @importFrom tibble tibble +#' @importFrom tidyr drop_na pivot_wider#<< +#' @importFrom tidyselect any_of#<< +## usethis namespace: end + +NULL +``` + +### Parameter function + +No changes required. + +### Warm-up function + +No changes required. + +### Functions to calculate different metrics + +We introduce new functions to calculate each metric. + +```{r} +#| file: outputs_resources/all_measures/metrics.R +``` + +### Run results function + +We add a new `simulation_end_time` parameter to `get_run_results()`, and call the functions which calculate each metric. + +```{r} +#' Get average results for the provided single run. +#' +#' @param results Named list with `arrivals` from `get_mon_arrivals()` and +#' `resources` from `get_mon_resources()` +#' (`per_resource = TRUE` and `ongoing = TRUE`). +#' @param run_number Integer index of current simulation run. +#' @param simulation_end_time Time at end of simulation run.#<< +#' +#' @return Tibble with processed results from replication. +#' @export + +get_run_results <- function(results, run_number, simulation_end_time) {#<< + metrics <- list( + calc_arrivals(results[["arrivals"]]),#<< + calc_mean_wait(results[["arrivals"]]),#<< + calc_mean_serve_length(results[["arrivals"]]),#<< + calc_utilisation(results[["resources"]], simulation_end_time),#<< + calc_mean_queue_length(results[["arrivals"]], simulation_end_time),#<< + calc_mean_time_in_system(results[["arrivals"]]),#<< + calc_mean_patients_in_system(results[["patients_in_system"]],#<< + simulation_end_time)#<< + )#<< + dplyr::bind_cols(tibble(replication = run_number), metrics) +} +``` + +### Model function + +There quite a few changes to our `model()` function. These include: + +* Saving `full_run_length`, as we use it in multiple places now (and not just in `simmer::run()`). +* Within the patient trajectory, recording `doctor_queue_on_arrival`, `doctor_serve_start` and `doctor_serve_length`. +* Setting `mon = 2` for the patient generator, as we are adding our own attributes, and this is required so that these are returned. These attributes are processed and merged with the arrivals data returned by `get_mon_arrivals()`. +* Calculations for patient `time_in_system` and `wait_time`. +* Recording when each patient arrived and left (`arrivals_start`, `arrivals_end`) and using those to determine the `patients_in_system` at each timepoint. + +```{r} +#' Run simulation. +#' +#' @param param List. Model parameters. +#' @param run_number Numeric. Run number for random seed generation. +#' +#' @return Named list with tables `arrivals`, `resources` and `run_results`. +#' @export + +model <- function(param, run_number) { + + # Set random seed based on run number + set.seed(run_number) + + # Create simmer environment + env <- simmer("simulation", verbose = param[["verbose"]]) + + # Calculate full run length#<< + full_run_length <- (param[["warm_up_period"]] +#<< + param[["data_collection_period"]])#<< + + # Define the patient trajectory + patient <- trajectory("consultation") |> + # Record queue length on arrival#<< + set_attribute("doctor_queue_on_arrival",#<< + function() get_queue_count(env, "doctor")) |>#<< + seize("doctor", 1L) |> + # Record time resource is obtained#<< + set_attribute("doctor_serve_start", function() now(env)) |>#<< + # Record sampled length of consultation#<< + set_attribute("doctor_serve_length", function() {#<< + rexp(n = 1L, rate = 1L / param[["consultation_time"]])#<< + }) |>#<< + timeout(function() get_attribute(env, "doctor_serve_length")) |>#<< + release("doctor", 1L) + + env <- env |> + # Add doctor resource + add_resource("doctor", param[["number_of_doctors"]]) |> + # Add patient generator (mon=2 so can get manual attributes)#<< + add_generator("patient", patient, function() { + rexp(n = 1L, rate = 1L / param[["interarrival_time"]]) + }, mon = 2L) |>#<< + # Run the simulation + simmer::run(until = full_run_length)#<< + + # Extract information on arrivals and resources from simmer environment + result <- list( + arrivals = get_mon_arrivals(env, per_resource = TRUE, ongoing = TRUE), + resources = get_mon_resources(env) + ) + + # Get the extra arrivals attributes#<< + extra_attributes <- get_mon_attributes(env) |>#<< + dplyr::select("name", "key", "value") |>#<< + # Add column with resource name, and remove that from key#<< + mutate(resource = gsub("_.+", "", .data[["key"]]),#<< + key = gsub("^[^_]+_", "", .data[["key"]])) |>#<< + pivot_wider(names_from = "key", values_from = "value")#<< + + # Merge extra attributes with the arrival data#<< + result[["arrivals"]] <- left_join(#<< + result[["arrivals"]], extra_attributes, by = c("name", "resource")#<< + )#<< + + # Add time in system (unfinished patients will set to NaN) #<< + result[["arrivals"]][["time_in_system"]] <- (#<< + result[["arrivals"]][["end_time"]] - result[["arrivals"]][["start_time"]]#<< + )#<< + + # Filter to remove results from the warm-up period + result <- filter_warmup(result, param[["warm_up_period"]]) + + # Gather all start and end times, with a row for each, marked +1 or -1#<< + # Drop NA for end time, as those are patients who haven't left system#<< + # at the end of the simulation#<< + arrivals_start <- transmute(#<< + result[["arrivals"]], time = .data[["start_time"]], change = 1L#<< + )#<< + arrivals_end <- result[["arrivals"]] |>#<< + drop_na(all_of("end_time")) |>#<< + transmute(time = .data[["end_time"]], change = -1L)#<< + events <- bind_rows(arrivals_start, arrivals_end)#<< + + # Determine the count of patients in the service with each entry/exit#<< + result[["patients_in_system"]] <- events |>#<< + # Sort events by time#<< + arrange(.data[["time"]], desc(.data[["change"]])) |>#<< + # Use cumulative sum to find number of patients in system at each time#<< + mutate(count = cumsum(.data[["change"]])) |>#<< + dplyr::select(c("time", "count"))#<< + + # Replace "replication" value with appropriate run number + result[["arrivals"]] <- mutate(result[["arrivals"]], + replication = run_number) + result[["resources"]] <- mutate(result[["resources"]], + replication = run_number) + result[["patients_in_system"]] <- mutate(result[["patients_in_system"]],#<< + replication = run_number)#<< + + # Calculate wait time for each patient#<< + result[["arrivals"]] <- mutate(#<< + result[["arrivals"]],#<< + wait_time = .data[["serve_start"]] - .data[["start_time"]]#<< + )#<< + + # Calculate the average results for that run + result[["run_results"]] <- get_run_results( + result, run_number, simulation_end_time = full_run_length#<< + ) + + result +} +``` + +### Run the model + +```{r} +param <- create_params() +result <- model(param = param, run_number = 1L) +kable(arrange(result[["arrivals"]], start_time)) +kable(result[["run_results"]]) +``` + +::: + +:::: + ## Total arrivals :::: {.callout title="▶  View how to record the total arrivals" collapse="true"} @@ -553,7 +852,7 @@ For this measure, we need to record the **time each patient spent waiting** for A new attribute `wait_time` is added to the `Patient` class. -As it is an attribute, we have to give `wait_time` an initial value. A value like 0 would wrongly look like "no wait", so `np.nan` is used instead to clearly mean "we don't know this patient's wait time yet". Patients who still have `wait_time = np.nan` at the end of the simulation are those who were never seen, and this is used below when calculating metrics for unseen (backlogged) patients. +As it is an attribute, we have to give `wait_time` an initial value. A value like 0 would wrongly look like "no wait", so `np.nan` is used instead to clearly mean "we don't know this patient's wait time yet". Patients who still have `wait_time = np.nan` at the end of the simulation are those who were never seen, and this is used below when calculating metrics for unseen (backlogged) patients. Patients who have `wait_time = 0` at the end of the simulation did not wait for the resource (doctor) as the resource was free. ```{python} class Patient: @@ -1285,6 +1584,8 @@ kable(result[["run_results"]]) Mean utilisation measures the proportion of available doctor time that is actually used during the data collection period. We have suggested two different approaches for measuring utilisation - both return the same result. +### Approach 1 + The first approach is **easier to understand**, as it just sums consultation times for each patient. However, it relies on making **corrections** when consultations overlap the end of the simulation or span the warm-up and data collection periods. ::: {.callout title="▶  View how to record utilisation" collapse="true"} @@ -1567,7 +1868,9 @@ class Runner: ::: -While this method introduces a **new class** and may initially seem **harder to grasp**, it has the advantage of requiring **fewer changes** to the rest of the simulation code. It is also handy for [choosing the length of the warm-up period](length_warmup.qmd), as it provides you with utilisation at each timepoint (rather than a final overall utilisation). +### Approach 2 + +This second approach introduces a **new class** and may initially seem **harder to grasp**, but it has the advantage of requiring **fewer changes** to the rest of the simulation code. It is also handy for [choosing the length of the warm-up period](length_warmup.qmd), as it provides you with utilisation at each timepoint (rather than a final overall utilisation). ::: {.callout title="▶  View how to record utilisation (time-weighted approach)" collapse="true"} @@ -1610,9 +1913,9 @@ class MonitoredResource(simpy.Resource): Parameters ---------- *args : - Positional arguments to be passed to the parent class. + Positional arguments to be passed to the parent class constructor. **kwargs : - Keyword arguments to be passed to the parent class. + Keyword arguments to be passed to the parent class constructor. """ # Initialise a SimPy Resource super().__init__(*args, **kwargs) @@ -1715,9 +2018,9 @@ class MonitoredResource(simpy.Resource): Parameters ---------- *args : - Positional arguments to be passed to the parent class. + Positional arguments to be passed to the parent class constructor. **kwargs : - Keyword arguments to be passed to the parent class. + Keyword arguments to be passed to the parent class constructor. """ # Initialise a SimPy Resource super().__init__(*args, **kwargs) @@ -1725,9 +2028,11 @@ class MonitoredResource(simpy.Resource): self.init_results() ``` -This class inherits SimPy's `Resource` class (if new to inheritance, see the [Code organisation](../setup/code_structure.qmd) page for an introduction). This means it has same functionality, but we can modify, etcetc. +This class inherits SimPy's `Resource` class (if new to inheritance, see the [Code organisation](../setup/code_structure.qmd) page for an introduction). This means it has same functionality, but we can modify and add to it. + +In the `__init__` method, we initialise the object by calling the constructor of the parent `simpy.Resource` class via `super().__init__(*args, **kwargs)`, passing through all positional and keyword arguments unchanged. We then call the new `init_results` method to set up the monitoring attributes. -In the `__init__` method, the class is initialised as it would be for a standard resource class, but then also calls the new `init_results` method. +> Those with a C/C++ background might see `*args` and `**kwargs` and think of variable argument lists. In Python, `*` and `**` are used to collect (and later unpack) positional and keyword arguments, which we forward directly to the parent class constructor and methods using `super(...)(*args, **kwargs)`.
@@ -1909,6 +2214,8 @@ class Runner: ::: +Moving forward, as we develop the code in the rest of the book (for example, when we introduce [replications](replications.qmd) on the next page), we will use the second, time‑weighted utilisation approach. + :::: :::: {.r-content} @@ -4237,7 +4544,7 @@ The value is much closer to the true "average" experienced in time, because the The same logic applies for the other performance measures. In the code, we track the product of value and duration ("area under the curve") for each metric, then divide the total by the observation time. -It's called "**area under the curve**" because, when you plot the measure (e.g., queue length) over time, the total area of all the rectangles you get under the curve (height=queue x width=time) returns your average. +It's called "**area under the curve**" because, when you plot the measure (e.g., queue length) over time, the total area of all the rectangles you get under the curve, divided by the time elapsed, returns your average. ```{r} #| echo: false diff --git a/pages/guide/output_analysis/replications_resources/metrics.R b/pages/guide/output_analysis/outputs_resources/all_measures/metrics.R similarity index 100% rename from pages/guide/output_analysis/replications_resources/metrics.R rename to pages/guide/output_analysis/outputs_resources/all_measures/metrics.R diff --git a/pages/guide/output_analysis/outputs_resources/all_measures/model.py b/pages/guide/output_analysis/outputs_resources/all_measures/model.py new file mode 100644 index 00000000..655ccf0e --- /dev/null +++ b/pages/guide/output_analysis/outputs_resources/all_measures/model.py @@ -0,0 +1,196 @@ +class Model: + """ + Simulation model. + + Attributes + ---------- + param : Parameters + Simulation parameters. + run_number : int + Run number for random seed generation. + env : simpy.Environment + The SimPy environment for the simulation. + doctor : simpy.Resource + SimPy resource representing doctors. + patients : list + List of Patient objects. + results_list : list + List of dictionaries with the attributes of each patient. + area_n_in_system : list of float#<< + List containing incremental area contributions used for#<< + time-weighted statistics of the number of patients in the system.#<< + time_last_n_in_system : float#<< + Simulation time at last update of the number-in-system statistic.#<< + n_in_system: int#<< + Current number of patients present in the system, including#<< + waiting and being served.#<< + arrival_dist : Exponential + Distribution used to generate random patient inter-arrival times. + consult_dist : Exponential + Distribution used to generate length of a doctor's consultation. + """ + def __init__(self, param, run_number): + """ + Create a new Model instance. + + Parameters + ---------- + param : Parameters + Simulation parameters. + run_number : int + Run number for random seed generation. + """ + self.param = param + self.run_number = run_number + + # Create SimPy environment + self.env = simpy.Environment() + + # Create resource + self.doctor = MonitoredResource(#<< + self.env, capacity=self.param.number_of_doctors + ) + + # Create a random seed sequence based on the run number + ss = np.random.SeedSequence(self.run_number) + seeds = ss.spawn(2) + + # Set up attributes to store results + self.patients = [] + self.results_list = [] + self.area_n_in_system = [0]#<< + self.time_last_n_in_system = self.env.now#<< + self.n_in_system = 0#<< + + # Initialise distributions + self.arrival_dist = Exponential(mean=self.param.interarrival_time, + random_seed=seeds[0]) + self.consult_dist = Exponential(mean=self.param.consultation_time, + random_seed=seeds[1]) + + def update_n_in_system(self, inc):#<< + """#<< + Update the time-weighted statistics for number of patients in system.#<< + + Parameters#<< + ----------#<< + inc : int#<< + Change in the number of patients (+1, 0, -1).#<< + """#<< + # Compute time since last event and calculate area under curve for that#<< + duration = self.env.now - self.time_last_n_in_system#<< + self.area_n_in_system.append(self.n_in_system * duration)#<< + # Update time and n in system#<< + self.time_last_n_in_system = self.env.now#<< + self.n_in_system += inc#<< + + def generate_arrivals(self): + """ + Process that generates patient arrivals. + """ + while True: + # Sample and pass time to next arrival + sampled_iat = self.arrival_dist.sample() + yield self.env.timeout(sampled_iat) + + # Check whether arrived during warm-up or data collection + if self.env.now < self.param.warm_up_period: + period = "\U0001F538 WU" + else: + period = "\U0001F539 DC" + + # Create a new patient + patient = Patient(patient_id=len(self.patients)+1, + period=period, + arrival_time=self.env.now) + self.patients.append(patient) + + # Update the number in the system#<< + self.update_n_in_system(inc=1)#<< + + # Print arrival time + if self.param.verbose: + print(f"{patient.period} Patient {patient.patient_id} " + + f"arrives at time: {patient.arrival_time:.3f}") + + # Start process of consultation + self.env.process(self.consultation(patient)) + + def consultation(self, patient): + """ + Process that simulates a consultation. + + Parameters + ---------- + patient : + Instance of the Patient() class representing a single patient. + """ + start_wait = self.env.now#<< + # Patient requests access to a doctor (resource) + with self.doctor.request() as req: + yield req + + # Record how long patient waited before consultation#<< + patient.wait_time = self.env.now - start_wait#<< + + if self.param.verbose: + print(f"{patient.period} Patient {patient.patient_id} " + + f"starts consultation at: {self.env.now:.3f}") + + # Sample consultation duration and pass time spent with doctor#<< + patient.time_with_doctor = self.consult_dist.sample()#<< + yield self.env.timeout(patient.time_with_doctor)#<< + + # Update number in system#<< + self.update_n_in_system(inc=-1)#<< + + # Record end time#<< + patient.end_time = self.env.now#<< + if self.param.verbose: + print(f"{patient.period} Patient {patient.patient_id} " + + f"leaves at: {patient.end_time:.3f}") + + def reset_results(self): + """ + Reset results. + """ + self.patients = [] + self.doctor.init_results()#<< + # For number in system, we reset area and time but not the count, as#<< + # it should include any remaining warm-up patients in the count#<< + self.area_n_in_system = [0]#<< + self.time_last_n_in_system = self.env.now#<< + + def warmup(self): + """ + Reset result collection after the warm-up period. + """ + if self.param.warm_up_period > 0: + # Delay process until warm-up period has completed + yield self.env.timeout(self.param.warm_up_period) + # Reset results variables + self.reset_results() + if self.param.verbose: + print(f"Warm up period ended at time: {self.env.now}") + + def run(self): + """ + Run the simulation. + """ + # Schedule arrival generator and warm-up + self.env.process(self.generate_arrivals()) + self.env.process(self.warmup()) + + # Run the simulation + self.env.run(until=(self.param.warm_up_period + + self.param.data_collection_period)) + + # At simulation end, update time-weighted statistics by accounting#<< + # for the time from the last event up to the simulation finish.#<< + self.doctor.update_time_weighted_stats()#<< + + # Run final calculation of number in system#<< + self.update_n_in_system(inc=0)#<< + + # Create list of dictionaries containing each patient's attributes + self.results_list = [x.__dict__ for x in self.patients] diff --git a/pages/guide/output_analysis/outputs_resources/all_measures/monitored_resource.py b/pages/guide/output_analysis/outputs_resources/all_measures/monitored_resource.py new file mode 100644 index 00000000..e35855ba --- /dev/null +++ b/pages/guide/output_analysis/outputs_resources/all_measures/monitored_resource.py @@ -0,0 +1,109 @@ +class MonitoredResource(simpy.Resource): + """ + SimPy Resource subclass that tracks utilisation, queue length and mean + number of patients in the system. + + Attributes + ---------- + time_last_event : list + Time of the most recent resource request or release event. + area_resource_busy : list + Cumulative time resources were busy, for computing utilisation. + area_n_in_queue : list + Cumulative time patients spent queueing for a resource, for + computing average queue length. + + Notes + ----- + Class adapted from Monks, Harper and Heather 2025. + """ + + def __init__(self, *args, **kwargs): + """ + Initialise the MonitoredResource, resetting monitoring statistics. + + Parameters + ---------- + *args : + Positional arguments to be passed to the parent class. + **kwargs : + Keyword arguments to be passed to the parent class. + """ + # Initialise a SimPy Resource + super().__init__(*args, **kwargs) + # Run the init_results() method + self.init_results() + + def init_results(self): + """ + Reset monitoring attributes. + """ + self.time_last_event = [self._env.now] + self.area_resource_busy = [0.0] + self.area_n_in_queue = [0.0] + + def request(self, *args, **kwargs): + """ + Update time-weighted statistics before requesting a resource. + + Parameters + ---------- + *args : + Positional arguments to be passed to the parent class. + **kwargs : + Keyword arguments to be passed to the parent class. + + Returns + ------- + simpy.events.Event + Event for the resource request. + """ + # Update time-weighted statistics + self.update_time_weighted_stats() + # Request a resource + return super().request(*args, **kwargs) + + def release(self, *args, **kwargs): + """ + Update time-weighted statistics before releasing a resource. + + Parameters + ---------- + *args : + Positional arguments to be passed to the parent class. + **kwargs : + Keyword arguments to be passed to the parent class. + + Returns + ------- + simpy.events.Event + Event for the resource release. + """ + # Update time-weighted statistics + self.update_time_weighted_stats() + # Release a resource + return super().release(*args, **kwargs) + + def update_time_weighted_stats(self): + """ + Update the time-weighted statistics for resource usage. + + Notes + ----- + - These sums can be referred to as "the area under the curve". + - They are called "time-weighted" statistics as they account for how + long certain events or states persist over time. + """ + # Calculate time since last event + time_since_last_event = self._env.now - self.time_last_event[-1] + + # Add record of current time + self.time_last_event.append(self._env.now) + + # Add "area under curve" of resources in use + # self.count is the number of resources in use + self.area_resource_busy.append(self.count * time_since_last_event) + + # Add "area under curve" of people in queue + # len(self.queue) is the number of requests queued + self.area_n_in_queue.append(len(self.queue) * time_since_last_event) diff --git a/pages/guide/output_analysis/outputs_resources/all_measures/patient.py b/pages/guide/output_analysis/outputs_resources/all_measures/patient.py new file mode 100644 index 00000000..d381f7a3 --- /dev/null +++ b/pages/guide/output_analysis/outputs_resources/all_measures/patient.py @@ -0,0 +1,38 @@ +class Patient: + """ + Represents a patient. + + Attributes + ---------- + patient_id : int + Unique patient identifier. + period : str + Arrival period (warm up or data collection) with emoji. + arrival_time : float + Time patient entered the system (minutes). + wait_time : float#<< + Time spent waiting for the doctor (minutes).#<< + time_with_doctor : float#<< + Time spent in consultation with a doctor (minutes).#<< + end_time : float#<< + Time that patient leaves (minutes), or NaN if remain in system.#<< + """ + def __init__(self, patient_id, period, arrival_time): + """ + Initialises a new patient. + + Parameters + ---------- + patient_id : int + Unique patient identifier. + period : str + Arrival period (warm up or data collection) with emoji. + arrival_time : float + Time patient entered the system (minutes). + """ + self.patient_id = patient_id + self.period = period + self.arrival_time = arrival_time + self.wait_time = np.nan#<< + self.time_with_doctor = np.nan#<< + self.end_time = np.nan#<< diff --git a/pages/guide/output_analysis/outputs_resources/all_measures/runner.py b/pages/guide/output_analysis/outputs_resources/all_measures/runner.py new file mode 100644 index 00000000..1feac912 --- /dev/null +++ b/pages/guide/output_analysis/outputs_resources/all_measures/runner.py @@ -0,0 +1,81 @@ +class Runner: + """ + Run the simulation. + + Attributes + ---------- + param : Parameters + Simulation parameters. + """ + def __init__(self, param): + """ + Initialise a new instance of the Runner class. + + Parameters + ---------- + param : Parameters + Simulation parameters. + """ + self.param = param + + def run_single(self, run): + """ + Runs the simulation once and performs results calculations. + + Parameters + ---------- + run : int + Run number for the simulation. + + Returns + ------- + dict + Contains patient-level results and results from each run. + """ + model = Model(param=self.param, run_number=run) + model.run() + + # Patient results + patient_results = pd.DataFrame(model.results_list) + patient_results["run"] = run + patient_results["time_in_system"] = (#<< + patient_results["end_time"] - patient_results["arrival_time"]#<< + )#<< + # For each patient, if they haven't seen a doctor yet, calculate#<< + # their wait as current time minus arrival, else set as missing#<< + patient_results["unseen_wait_time"] = np.where(#<< + patient_results["time_with_doctor"].isna(),#<< + model.env.now - patient_results["arrival_time"], np.nan#<< + ) + + # Run results + run_results = { + "run_number": run, + "arrivals": len(patient_results),#<< + "mean_wait_time": patient_results["wait_time"].mean(),#<< + "mean_time_with_doctor": (#<< + patient_results["time_with_doctor"].mean()#<< + ),#<< + "mean_utilisation_tw": (#<< + sum(model.doctor.area_resource_busy) / (#<< + self.param.number_of_doctors *#<< + self.param.data_collection_period#<< + )#<< + ),#<< + "mean_queue_length": (#<< + sum(model.doctor.area_n_in_queue) /#<< + self.param.data_collection_period#<< + ),#<< + "mean_time_in_system": patient_results["time_in_system"].mean(),#<< + "mean_patients_in_system": (#<< + sum(model.area_n_in_system) /#<< + self.param.data_collection_period#<< + ),#<< + "unseen_count": patient_results["time_with_doctor"].isna().sum(),#<< + "unseen_wait_time": patient_results["unseen_wait_time"].mean()#<< + }#<< + + return { + "patient": patient_results, + "run": run_results + } diff --git a/pages/guide/output_analysis/parallel.qmd b/pages/guide/output_analysis/parallel.qmd index a3174f3b..5af164f7 100644 --- a/pages/guide/output_analysis/parallel.qmd +++ b/pages/guide/output_analysis/parallel.qmd @@ -33,19 +33,23 @@ This page continues on from: [Replications](replications.qmd). **Required packages:** -These should be available from environment setup on the [Structuring as a package](/pages/guide/setup/package.qmd) page. +The following imports are required. These should be available from environment setup on the [Structuring as a package](/pages/guide/setup/package.qmd) page. + +Since [Replications](replications.qmd), we have several new imports: `time`, `joblib` and `plotly.express`. ::: {.python-content} ```{python} -import time - +# For display purposes: from IPython.display import HTML from itables import to_html_datatable -from joblib import Parallel, delayed, cpu_count + +# For the simulation modelling and analysis: +import time#<< +from joblib import Parallel, delayed, cpu_count#<< import numpy as np import pandas as pd -import plotly.express as px +import plotly.express as px#<< import scipy.stats as st import simpy from sim_tools.distributions import Exponential diff --git a/pages/guide/output_analysis/replications.qmd b/pages/guide/output_analysis/replications.qmd index 9b243bd5..a7462801 100644 --- a/pages/guide/output_analysis/replications.qmd +++ b/pages/guide/output_analysis/replications.qmd @@ -30,16 +30,21 @@ This page continues on from: [Performance measures](outputs.qmd). **Required packages:** -These should be available from environment setup on the [Structuring as a package](/pages/guide/setup/package.qmd) page. +The following imports are required. These should be available from environment setup on the [Structuring as a package](/pages/guide/setup/package.qmd) page. + +For this page, we add one new import: `scipy.stats`. ::: {.python-content} ```{python} +# For display purposes: from IPython.display import HTML from itables import to_html_datatable + +# For the simulation modelling and analysis: import numpy as np import pandas as pd -import scipy.stats as st +import scipy.stats as st#<< import simpy from sim_tools.distributions import Exponential ``` @@ -157,7 +162,7 @@ No changes required. ### Summary statistics function -This function computes descriptive statistics for a dataset, returning the mean, standard deviation, and 95% confidence interval. +This is a new function which computes descriptive statistics for a dataset, returning the mean, standard deviation, and 95% confidence interval. * If the input has **no valid values**, all outputs are `NaN`. @@ -320,7 +325,7 @@ No changes required. ::: {.callout title="▶  View metrics functions" collapse="true" style="minimal"} ```{r} -#| file: replications_resources/metrics.R +#| file: outputs_resources/all_measures/metrics.R ``` ::: diff --git a/pages/guide/output_analysis/warmup.qmd b/pages/guide/output_analysis/warmup.qmd index f5ef71ad..c6d42312 100644 --- a/pages/guide/output_analysis/warmup.qmd +++ b/pages/guide/output_analysis/warmup.qmd @@ -24,7 +24,7 @@ This page continues on from: [Entity processing](../model/process.qmd). **Required packages:** -These should be available from environment setup on the [Structuring as a package](/pages/guide/setup/package.qmd) page. +The following imports are required. These should be available from environment setup on the [Structuring as a package](/pages/guide/setup/package.qmd) page. ::: {.python-content} @@ -477,7 +477,12 @@ create_params <- function( ### Model function -Add `get_mon_arrivals()` and `get_mon_resources()`. +Add two functions from `simmer`: + +* `get_mon_arrivals()` - gets dataframe of arrival statistics +* `get_mon_resources()` - gets dataframe of resource statistics + +You can see the dataframes returned by these functions in the "Run the model" section below. ```{r} #' Run simulation. diff --git a/pages/guide/setup/code_structure.qmd b/pages/guide/setup/code_structure.qmd index 6c7ca27e..7885339a 100644 --- a/pages/guide/setup/code_structure.qmd +++ b/pages/guide/setup/code_structure.qmd @@ -35,7 +35,7 @@ date: "2025-09-30T14:35:31+01:00" **Required packages:** -These should be available from environment setup on the [Structuring as a package](/pages/guide/setup/package.qmd) page. +The following imports are required. These should be available from environment setup on the [Structuring as a package](/pages/guide/setup/package.qmd) page. ```{python} import numpy as np diff --git a/pages/guide/setup/package.qmd b/pages/guide/setup/package.qmd index bdf008fc..c28151da 100644 --- a/pages/guide/setup/package.qmd +++ b/pages/guide/setup/package.qmd @@ -1039,7 +1039,7 @@ Some common issues and quick checks... | - | - | | `ModuleNotFoundError: No module named 'simulation'` | Check that you have installed the package into your environment using `pip install -e .` and that you are running your notebook from that environment. | | Changes to `simulation/model.py` are not reflect in the notebook | Make sure the first cell of your notebook loads the autoreload extension - `%load_ext autoreload` then `%autoreload 2` - and re-run that cell after opening the notebook. | -| Import works but function name is "not defined" | Confirm that your function is defined in `simulation/model.py`, that the file is saved, and that your import matches the definition (e.g., `from simulation.model import run_simulation`). +| Import works but function name is "not defined" | Confirm you have saved latest changes, that your function is defined in `simulation/model.py`, that the file is saved, and that your import matches the definition (e.g., `from simulation.model import run_simulation`). ::: @@ -1049,7 +1049,7 @@ Some common issues and quick checks... | - | - | | `devtools::check()` fails with missing `DESCRIPTION` or `NAMESPACE` | Make sure you have created `DESCRIPTION` in your project root and run `devtools::document()` at least once so that `NAMESPACE` and `man/` are generated. | | `library(simulation)` cannot find the package | Check that you have either run `devtools::load_all()` or `devtools::install()` | -| Function not found or changes to function are not reflected in Rmd | Confirm that the function is saved in a file under `R/`, that you have re-run `devtools::load_all()` and that the name in your analysis matches the function name exactly. | +| Function not found or changes to function are not reflected in Rmd | Confirm that you have saved latest changes, that function is in a file under `R/`, that you have re-run `devtools::load_all()` and that the name in your analysis matches the function name exactly. | ::: diff --git a/pages/guide/sharing/changelog.qmd b/pages/guide/sharing/changelog.qmd index 701988f4..17385b36 100644 --- a/pages/guide/sharing/changelog.qmd +++ b/pages/guide/sharing/changelog.qmd @@ -195,6 +195,8 @@ On the new release page, enter a version tag (e.g., `v0.1.0`), copy in your rele When writing a changelog, it is helpful to **look over your GitHub commits** to remember recent changes. +> This is why it's important to write clear, descriptive commit messages - they make it easier to recall what was changed and why. + If you have already created some releases, click on your latest release on GitHub and then **X commits to main since this release** to see all recent commits: ![](changelog_resources/changes_since_release.png){fig-alt="Screenshot of GitHub release with cursor hovering over 'X commits to main since this release'."} diff --git a/pages/guide/style_docs/docstrings.qmd b/pages/guide/style_docs/docstrings.qmd index 862dc747..0769b3d6 100644 --- a/pages/guide/style_docs/docstrings.qmd +++ b/pages/guide/style_docs/docstrings.qmd @@ -51,6 +51,17 @@ def add_positive_numbers(a, b): return a + b ``` +We also describe the data type of each parameter and return value in the docstring. Common types include: + +* `int` (whole number, e.g., 3). +* `float` (number with decimal pint, e.g., 3.5). +* `str` (text string, e.g., `"emergency"`). +* `bool` (True or False value). +* `list` (e.g., `list of str`, `list of int`). +* `dict` (dictionary). +* `pandas.DataFrame` or `pandas.Series`. +* Instances of your own classes e.g., `Patient`. + ::: ::: {.r-content} @@ -64,10 +75,10 @@ Example: ```{r} #' Add two positive numbers#<< #'#<< -#' @param a First number to add (must be positive).#<< -#' @param b Second number to add (must be positive).#<< +#' @param a numeric First number to add (must be positive).#<< +#' @param b numeric Second number to add (must be positive).#<< #'#<< -#' @return The sum of a and b.#<< +#' @return numeric The sum of a and b.#<< #' @export#<< add_positive_numbers <- function(a, b) { @@ -76,6 +87,23 @@ add_positive_numbers <- function(a, b) { } ``` +A simple template for R docstrings (roxygen2) is: + +* One short sentence describing the purpose of the function. +* A `@param` line for each argument: `@param name type description` +* A `@return` line with type and description of output. +* (If using package structure) An `@export` tag so the function is available to users when they load the package. + +We can choose to describe the data type of each parameter and return value in the docstring. Common types include: + +* `numeric` (numbers, e.g., `3` or `3.5`). +* `integer` (whole numbers, e.g., `3L`). +* `character` (text string, e.g., `"emergency"`). +* `logical` (TRUE/FALSE values). +* `list`. +* `data.frame`. +* `tibble` (a `data.frame` variant from tidyverse). + ::: ### How do they differ from in-line comments? @@ -105,6 +133,16 @@ a + b ::: +::: {.callout-note title="Quick guidelines for in-line comments"} + +In‑line comments should be concise and avoid clutter; more is not necessarily better. + +* Avoid repeating information that is already in the docstring. +* Don't comment obvious code - just use comments where logic or intent isn't immediately clear. +* Avoid TODO/FIXME comments - it's better to use an issue tracker like GitHub issues instead. + +::: + ### Why use docstrings? * **Clarity and consistency:** Docstrings make code easier to understand and provide a uniform way to document your work, which is especially valuable when collaborating. diff --git a/pages/guide/style_docs/linting_resources/style_guides.png b/pages/guide/style_docs/linting_resources/style_guides.png index d51a759a..e9a53261 100644 Binary files a/pages/guide/style_docs/linting_resources/style_guides.png and b/pages/guide/style_docs/linting_resources/style_guides.png differ diff --git a/pages/guide/style_docs/linting_resources/style_guides.svg b/pages/guide/style_docs/linting_resources/style_guides.svg index 2cdadf34..833f71cc 100644 --- a/pages/guide/style_docs/linting_resources/style_guides.svg +++ b/pages/guide/style_docs/linting_resources/style_guides.svg @@ -1,37 +1,30 @@ - - - - - + + + + + - - + + - - - - - - - - - - - - - - - - - - - - - + + + + + + + + + + + + + + \ No newline at end of file diff --git a/pages/guide/verification_validation/mathematical.qmd b/pages/guide/verification_validation/mathematical.qmd index dd80b50f..ba7bbbd9 100644 --- a/pages/guide/verification_validation/mathematical.qmd +++ b/pages/guide/verification_validation/mathematical.qmd @@ -1,5 +1,7 @@ --- title: Mathematical proof of correctness +filters: + - line-highlight date: "2025-10-23T16:24:23+01:00" --- @@ -28,16 +30,20 @@ The test is run on the model from the [parallel processing](../output_analysis/p **Required packages:** -These should be available from environment setup on the [Structuring as a package](/pages/guide/setup/package.qmd) page. +The following imports are required. These should be available from environment setup on the [Structuring as a package](/pages/guide/setup/package.qmd) page. + +Since [Parallel processing](/pages/guide/output_analysis/parallel.qmd), we have one new imports: `math`. ::: {.python-content} ```{python} -import math -import time - +# For display purposes: from IPython.display import HTML from itables import to_html_datatable + +# For the simulation modelling and analysis: +import math#<< +import time from joblib import Parallel, delayed, cpu_count import numpy as np import pandas as pd @@ -655,6 +661,24 @@ Our test function `test_simulation_against_theory()` uses `@pytest.mark.parametr For each of these, it compares the results from the simulation model (from `run_simulation_model()`) with the results from the mathematical model (`MMSQueue`) and checks they are reasonably similar. +::: + +::: {.r-content} + +Our test function uses `patrick` to run several different test cases. + +For each of these, it compares the results from the simulation model (from `run_simulation_model()`) with the results from the mathematical model from `queueing`, and checks they are reasonably similar. + +::: + +::: {.callout-note title="Tolerance"} + +We use a 15% threshold for the relative difference between simulation and analytical results. This allows for expected variation, while still flagging results that differ by more than would be reasonably explained by randomness alone. + +::: + +::: {.python-content} + ```{.python} @pytest.mark.parametrize( "interarrival_time,consultation_time,number_of_doctors", @@ -719,10 +743,6 @@ pytest.main(["tests_resources/test_mms.py"]) ::: {.r-content} -Our test function uses `patrick` to run several different test cases. - -For each of these, it compares the results from the simulation model (from `run_simulation_model()`) with the results from the mathematical model from `queueing`, and checks they are reasonably similar. - ```{r} #| file: "tests_resources/test_mms.R" #| eval: false @@ -741,6 +761,8 @@ testthat::test_file(file.path("tests_resources", "test_mms.R")) ::: +🎉 All tests have passed! + ## Explore the example models
diff --git a/pages/guide/verification_validation/tests.qmd b/pages/guide/verification_validation/tests.qmd index ae1af216..29c30652 100644 --- a/pages/guide/verification_validation/tests.qmd +++ b/pages/guide/verification_validation/tests.qmd @@ -29,9 +29,9 @@ This page will run tests on the model from the [parallel processing](../output_a
-Testing is the process of evaluating a model to ensure it works as expected, gives reliable results, and can handle different conditions. By **systematically checking for errors, inconsistencies, or unexpected behaviors**, testing helps improve the quality of a model, catch errors and prevent future issues. They can also be used to run test cases that check your results are consistent (i.e., that it is **reproducible**). +Testing is the process of evaluating a model to ensure it works as expected, gives reliable results, and can handle different conditions. By **systematically checking for errors, inconsistencies, or unexpected behaviors**, testing helps improve the quality of a model, identify errors and prevent future issues. They can also be used to run test cases that check your results are consistent (i.e., that it is **reproducible**). -When you create a model, you will naturally carry out tests, with simple manual checks where you observe outputs and ensure they look right. These checks can be formalised and **automated** so that you can run them after any changes, and catch any issues that arise. +When you create a model, you will naturally carry out tests, with simple manual checks where you observe outputs and ensure they look right. These checks can be formalised and **automated** so that you can run them after any changes, and identify any issues that arise. Writing tests is fundamental for [model verification](verification_validation.qmd). These tests should be run regularly and after code changes, to check that these changes haven't introduced new issues ("**regression testing**"). diff --git a/pages/guide/verification_validation/verification_validation.qmd b/pages/guide/verification_validation/verification_validation.qmd index 615a908f..131b2f4a 100644 --- a/pages/guide/verification_validation/verification_validation.qmd +++ b/pages/guide/verification_validation/verification_validation.qmd @@ -169,9 +169,9 @@ To check your routing logic, you would run boundary value tests using values jus Suppose we simulate a screening programme where patients are only eligible if their age is betwen 18 and 75 inclusive. Boundary value tests would check: -* Patients below the minimum are rejected (17). -* Patients at/near the thresholds are accepted (18, 19, 74, 75). -* Patients above the maximum are rejected (76). +* Patients below the minimum are rejected (17 or under). +* Patients at/near the thresholds are accepted (e.g., 18, 19, 74, 75). +* Patients above the maximum are rejected (76 or over). ::: {.cream} @@ -188,7 +188,7 @@ Stress testing involves running your model under extremely demanding conditions. * Do patients queue correctly? * Does it respect resource constraints? -* Do calculated performance measures remain valid (e.g., utilisation within 0-1). +* Do calculated performance measures remain valid (e.g., utilisation within 0-1 or 0-100%). ::: {.cream} @@ -260,81 +260,98 @@ This is a great baseline: start with a simple model you can compare against theo
-### Face validation +### Conceptual model validation -**Face validation** is a process where the project team, potential model users, and people with relevant experience of the healthcare system review the simulation. Based on their knowledge and experiences, they compare the model's behaviour and results with what would be expected in the real system and judge whether the model appears realistic and appropriate (@Balci1998). +Conceptual model validation determines whether the theories, assumptions, and underlying structure of your conceptual model are correct and reasonable for the intended purpose (@Sargent2013, @Robinson2007). This focuses on whether the model includes the necessary details and relationships to meet the study objectives. -Evaluating and reporting face validity is a requirement of the DES reporting checklist from @Zhang2025 (which is based on reports from the International Society for Pharmacoeconomics and Outcomes Research (ISPOR)-Society for Medical Decision Making (SMDM) Modeling Good Research Practices Task Force). +Key questions include: + +* Are the model's assumptions realistic and justified? +* Does the model structure capture the essential elements of the real system? +* Are simplifications appropriate given the study purpose? +* Do the causal relationships make sense? ::: {.cream} **What does this mean in practice?** -✔️ Present key simulation outputs and model behaviour to people such as: - -* Project team members. -* Intended users of the model (e.g., healthcare analysts, managers). -* People familiar with the real system (e.g., clinicians, frontline staff, patient representatives). +✔️ Document and justify all modeling assumptions. -Ask for their subjective feedback on whether the model and results "look right". Discuss specific areas, such as whether performance measures (e.g., patient flow, wait times) match expectations under similar conditions. +✔️ Review the conceptual model with people familiar with the real system to assess completeness and accuracy. ::: -### Turing test +### Input data validation -**Turing test** validation is a technique where experts familiar with the healthcare system are shown real-world output data and simulation output data, without knowing which is which. This provides a "blind test": they must decide which results they think are from the actual system and which are from the model. If experts cannot reliably tell the difference, it increases confidence in the model validity (@Balci1998). +Data validation involves confirming that the data used in your model are appropriate and accurately reflect the real system (@Sargent2013). -This can be supplementary to face validation and other comparisons - rather than asking reviewers whether outputs seem plausible, they test whether they are indistinguishable. +This includes checking that statistical distributions are appropriate representations and that any data transformations preserve the essential characteristics of the original data (@Banks2005). ::: {.cream} **What does this mean in practice?** -✔️ Collect matching sets of model output and real system, remove identifying labels, and present them to a panel of experts. Record whether experts can distinguish simulation outputs from real data. Use their feedback on distinguishing features to further improve the simulation. +✔️ Check the datasets used - screen for outliers, determine if they are correct, and if the reason for them occurring should be incorporated into the simulation (@Sargent2013). -::: +✔️ Ensure you have performed appropriate input modelling steps when choosing your distributions - see [input modelling](../inputs/input_modelling.qmd) page. -### Predictive validation +::: -**Predictive validation** means running the simulation model based on data from a particular time period of your system (e.g., a specific year, season or event). The model results should then be compared with the actual observed outcomes from that time period (@Balci1998). +### Graphical comparison -Predictive validation is a requirement in the DES reporting checklist from @Zhang2025. +**Graphical comparison** involves comparing graphs of model results over time with graphs of actual observed outcomes. You should look for similarities in patterns, trends, periodicities (i.e., recurring events), and other characteristics (@Balci1998). ::: {.cream} **What does this mean in practice?** -✔️ Use historical arrival data, staffing schedules, treatment times, or other inputs from a specific time period to drive your simulation. Compare the simulation's predictions for that period (e.g., waiting times, bed occupancy) against the real outcomes for the same period. +✔️ Create time-series plots and distributions of key results (e.g., daily patient arrivals, resource utilisation, waiting times) for both the model and the actual system, and compare the graphs to assess whether patterns and trends are similar. -✔️ Consider varying the periods you validate on—year-by-year, season-by-season, or even for particular policy changes or events—to detect strengths or weaknesses in the model across different scenarios. +::: -✔️ Use graphical comparisons (e.g., time series plots) or statistical measures (e.g., goodness-of-fit, mean errors, confidence intervals) to assess how closely the model matches reality - see below. +### Statistical comparison + +Statistical comparison involves using formal **statistical techniques** to compare model outputs with real system data. @Balci1998 recommend a validation procedure using simultaneous confidence intervals. Other statistical techniques including t-tests and goodness-of-fits tests have also been proposed. + +This is a type of **external validation** (comparing model results to empirical data of actual event), which is a requirement in the DES reporting checklist from @Zhang2025. + +::: {.cream} + +**What does this mean in practice?** + +✔️ Collect real system data on key performance measures (e.g., wait times, lengths of stay, throughput) and compare with model outputs statistically using appropriate tests. ::: -### Graphical comparison +### Turing test -**Graphical comparison** involves comparing graphs of model variables over time with graphs of real system variables. You should look for similarities in patterns, trends, periodicities (i.e., recurring events), and other characteristics (@Balci1998). +**Turing test** validation is a technique where experts familiar with the healthcare system are shown real-world output data and simulation output data, without knowing which is which. This provides a "blind test": they must decide which results they think are from the actual system and which are from the model. If experts cannot reliably tell the difference, it increases confidence in the model validity (@Balci1998). + +This can be supplementary to face validation and other comparisons - rather than asking reviewers whether outputs seem plausible, they test whether they are indistinguishable. ::: {.cream} **What does this mean in practice?** -✔️ Create time-series plots of key metrics (e.g., daily patient arrivals, resource utilisation) for both the model and system. Create distributions of waiting times, service times, and other performance measures. Compare the model and system graphs. +✔️ Collect matching sets of model output and real system, remove identifying labels, and present them to a panel of experts. Record whether experts can distinguish simulation outputs from real data. Use their feedback on distinguishing features to further improve the simulation. ::: -### Statistical comparison +### Predictive validation -Statistical comparison involves using formal **statistical techniques** to compare model outputs with real system data. @Balci1998 recommend a validation procedure using simultaneous confidence intervals. Other statistical techniques including t-tests and goodness-of-fits tests have also been proposed. +**Predictive validation** means running the simulation model based on data from a particular time period of your system (e.g., a specific year, season or event). The model results should then be compared with the actual observed outcomes from that time period (@Balci1998). -This is a type of **external validation** (comparing model results to empirical data of actual event), which is a requirement in the DES reporting checklist from @Zhang2025. +Predictive validation is a requirement in the DES reporting checklist from @Zhang2025. ::: {.cream} **What does this mean in practice?** -✔️ Collect real system data on key performance measures (e.g., wait times, lengths of stay, throughput) and compare with model outputs statistically using appropriate tests. +✔️ Use historical arrival data, staffing schedules, treatment times, or other inputs from a specific time period to drive your simulation. Compare the simulation's predictions for that period (e.g., waiting times, bed occupancy) against the real outcomes for the same period. + +✔️ Consider varying the periods you validate on—year-by-year, season-by-season, or even for particular policy changes or events—to detect strengths or weaknesses in the model across different scenarios. + +✔️ Use graphical comparisons (e.g., time series plots) or statistical measures (e.g., goodness-of-fit, mean errors, confidence intervals) to assess how closely the model matches reality - see below. ::: @@ -346,11 +363,7 @@ If you build an animation that shows your simulation model as it runs, it can he **What does this mean in practice?** -Creating an animation takes time, but it has several benefits beyond just validation. - -* ✔️ **Validation:** It can help you spot mistakes -* ✔️ **Communication:** It is really helpful for sharing the models with others (such as with stakeholders and decision makers). -* ✔️ **Reuse:** It supports reuse of the model (as in the [STARS framework](https://pythonhealthdatascience.github.io/stars/pages/research/stars_framework_reuse.html)). +There are free and open-source tools for creating animations. ::: {.python-content} @@ -358,56 +371,53 @@ In Python, check out [vidigi](https://github.com/Bergam0t/vidigi/tree/main) - a ::: -:::: - -### Comparison testing - -If two or more simulation models exist for the same system (perhaps built using different methods, languages, or by different groups), **comparison testing** (or "back-to-back testing") can be used. Running both models with identical input data and configurations and comparing their outputs can reveal problems or inconsistencies in at least one model's logic, assumptions or implementation (@Balci1998). - -::: {.cream} +::: {.r-content} -**What does this mean in practice?** +In R, check out [bupaR's `animate_process` function](https://bupaverse.github.io/docs/animate_maps.html), which can be used to create animated visualisations of simmer models. -✔️ If you have multiple models of the same system, compare them! +::: -You probably won't have two models often, but when you do, comparing them is very useful. For example, if you create an open-source version of a commercial model or translate a model between R and Python (as in this book), comparing results can quickly uncover issues and improve quality. +Though it may take time, creating an animation has several benefits beyond just validation. -::: +* ✔️ **Validation:** It can help you spot mistakes +* ✔️ **Communication:** It is really helpful for sharing the models with others (such as with stakeholders and decision makers) and supporting discussion and communication between different stakeholder groups. +* ✔️ **Reuse:** It supports reuse of the model (as in the [STARS framework](https://pythonhealthdatascience.github.io/stars/pages/research/stars_framework_reuse.html)). -### Input data validation +:::: -Data validation involves confirming that the data used in your model are appropriate and accurately reflect the real system (@Sargent2013). +### Comparison testing -This includes checking that statistical distributions are appropriate representations and that any data transformations preserve the essential characteristics of the original data (@Banks2005). +If two or more simulation models exist for the same system (perhaps built using different methods, languages, or by different groups), **comparison testing** (or "back-to-back testing") can be used. Running both models with identical input data and configurations and comparing their outputs can reveal problems or inconsistencies in at least one model's logic, assumptions or implementation (@Balci1998). ::: {.cream} **What does this mean in practice?** -✔️ Check the datasets used - screen for outliers, determine if they are correct, and if the reason for them occurring should be incorporated into the simulation (@Sargent2013). +✔️ If you have multiple models of the same system, compare them! -✔️ Ensure you have performed appropriate input modelling steps when choosing your distributions - see [input modelling](../inputs/input_modelling.qmd) page. +You probably won't have two models often, but when you do, comparing them is very useful. For example, if you create an open-source version of a commercial model or translate a model between R and Python (as in this book), comparing results can quickly uncover issues and improve quality. ::: -### Conceptual model validation +*Comparison testing cannot guarantee that either model is "right" - both could be wrong in the same way. However, it does identify any disagreements between the models, and will prompt you to re-check assumptions, logic and data handling, and so will help improve the quality of all models involved.* -Conceptual model validation determines whether the theories, assumptions, and underlying structure of your conceptual model are correct and reasonable for the intended purpose (@Sargent2013, @Robinson2007). This focuses on whether the model includes the necessary details and relationships to meet the study objectives. +### Face validation -Key questions include: +**Face validation** is a process where the project team, potential model users, and people with relevant experience of the healthcare system review the simulation. Based on their knowledge and experiences, they compare the model's behaviour and results with what would be expected in the real system and judge whether the model appears realistic and appropriate (@Balci1998). -* Are the model's assumptions realistic and justified? -* Does the model structure capture the essential elements of the real system? -* Are simplifications appropriate given the study purpose? -* Do the causal relationships make sense? +Evaluating and reporting face validity is a requirement of the DES reporting checklist from @Zhang2025 (which is based on reports from the International Society for Pharmacoeconomics and Outcomes Research (ISPOR)-Society for Medical Decision Making (SMDM) Modeling Good Research Practices Task Force). ::: {.cream} **What does this mean in practice?** -✔️ Document and justify all modeling assumptions. +✔️ Present key simulation outputs and model behaviour to people such as: -✔️ Review the conceptual model with people familiar with the real system to assess completeness and accuracy. +* People familiar with the real system (e.g., clinicians, frontline staff, patient representatives). +* Intended users of the model (e.g., healthcare analysts, managers). +* Project team members. + +Ask for their subjective feedback on whether the model and results "look right". Discuss specific areas, such as whether performance measures (e.g., patient flow, wait times) match expectations under similar conditions. ::: @@ -442,7 +452,7 @@ Cross validation is a requirement in the DES reporting checklist from @Zhang2025 **What does this mean in practice?** -✔️ Search for similar simulation studies and compare the key assumptions, methods and results. Discuss discrepancies and explain reasons for different findings or approaches. Use insights from other studies to improve or validate your own model. +✔️ Search for similar simulation studies and compare the key assumptions, model structures, and results with your own. Discuss important differences and explain possible reasons. ::: @@ -464,6 +474,18 @@ We have completed steps to **verify** the model. **Validation** has not been per ::: {.python-content} +Verification can be viewed in [GitHub Issue #84](https://github.com/pythonhealthdatascience/pydesrap_mms/issues/84) + +::: + +::: {.r-content} + +Verification can be viewed in [GitHub Issue #97](https://github.com/pythonhealthdatascience/rdesrap_mms/issues/97) + +::: + +::: {.python-content} + {{< include /html/pydesrapstroke.html >}} ::: @@ -476,13 +498,30 @@ We have completed steps to **verify** the model. **Validation** has not been per We have completed steps to **verify** the model. **Validation** has not been performed as this example just replicates an existing published model, and is not being re-applied to a real system. +::: {.python-content} + +Verification can be viewed in [GitHub Issue #27](https://github.com/pythonhealthdatascience/pydesrap_stroke/issues/27) + +::: + +::: {.r-content} + +Verification can be viewed in [GitHub Issue #18](https://github.com/pythonhealthdatascience/rdesrap_stroke/issues/18) + +::: + ## Test yourself We encourage you to apply as many verification and validation methods as you can to your simulation models. Each approach gives a fresh angle on model quality, and together they build a robust foundation of trust in your work. However, trying to do everything at once can feel overwhelming! Remember, verification and validation are iterative processes - it's normal and expected to work on them bit by bit, returning to add, review, and improve methods as your project develops. -To help you get started, we've put together a checklist you can copy into a [GitHub issue](../sharing/peer_review.qmd#github-issues) on your project. Use it to track which methods you've tried, what you want to tackle next, and which seem most relevant or feasible for your current context. This makes it easier to plan, prioritise, and document your verification and validation efforts over time. +To help you get started, we've put together a simple Markdown checklist that you can use to plan or track your work. You can either: + +* Open a new [GitHub issue](../sharing/peer_review.qmd#github-issues) in your repository, copy and paste the checklist, and then use the tickboxes to record progress. +* Download the `.md` file (`verification_validation_checklist.md`) into your repository and update it as you complete activities. + +Use the checklist to track which methods you've tried, what you want to tackle next, and which seem most relevant or feasible for your current context. {{< downloadthis verification_validation_resources/verification_validation_checklist.md dname="verification_validation_checklist" label="Download the verification and validation checklist" type="primary" >}} diff --git a/pages/guide/verification_validation/verification_validation_resources/references.bib b/pages/guide/verification_validation/verification_validation_resources/references.bib index 4ccc9ed0..3ed64672 100644 --- a/pages/guide/verification_validation/verification_validation_resources/references.bib +++ b/pages/guide/verification_validation/verification_validation_resources/references.bib @@ -46,12 +46,11 @@ @inproceedings{Roungas2017 } @online{KLE2025, - title = {Boundary Value Testing (Chapter 2, Unit 1)}, - author = {{KLE'S Nijalingappa College}}, - year = {2025}, - month = jul, - url = {https://bca.klesnc.edu.in/wp-content/uploads/2025/07/Chapter-2Unit-1-Boundary-Value-Testing.pdf}, - note = {Accessed: 2025-10-09} + title = {Boundary Value Testing (Chapter 2, Unit 1), Bachelor of Computer Application (BCA) programme study material, Accessed 2025-10-09}, + author = {{KLE'S Nijalingappa College}}, + year = {2025}, + month = jul, + url = {https://bca.klesnc.edu.in/wp-content/uploads/2025/07/Chapter-2Unit-1-Boundary-Value-Testing.pdf} } @article{Zhang2025, diff --git a/pages/guide/verification_validation/verification_validation_resources/verification_validation_checklist.md b/pages/guide/verification_validation/verification_validation_resources/verification_validation_checklist.md index 7256c958..6f04b83c 100644 --- a/pages/guide/verification_validation/verification_validation_resources/verification_validation_checklist.md +++ b/pages/guide/verification_validation/verification_validation_resources/verification_validation_checklist.md @@ -45,9 +45,23 @@ Mathematical proof of correctness ## Validation -Face validation +Conceptual model validation -* [ ] Present key simulation outputs and model behaviour to people such as: project team members; intended users of the model (e.g., healthcare analysts, managers); people familiar with the real system (e.g., clinicians, frontline staff, patient representatives). Ask for their subjective feedback on whether the model and results "look right". Discuss specific areas, such as whether performance measures (e.g., patient flow, wait times) match expectations under similar conditions. +* [ ] Document and justify all modeling assumptions. +* [ ] Review the conceptual model with people familiar with the real system to assess completeness and accuracy. + +Input data validation + +* [ ] Check the datasets used - screen for outliers, determine if they are correct, and if the reason for them occurring should be incorporated into the simulation. +* [ ] Ensure you have performed appropriate input modelling steps when choosing your distributions. + +Graphical comparison + +* [ ] Create time-series plots and distributions of key results (e.g., daily patient arrivals, resource utilisation, waiting times) for both the model and the actual system, and compare the graphs to assess whether patterns and trends are similar. + +Statistical comparison + +* [ ] Collect real system data on key performance measures (e.g., wait times, lengths of stay, throughput) and compare with model outputs statistically using appropriate tests. Turing test @@ -59,14 +73,6 @@ Predictive validation * [ ] Consider varying the periods you validate on—year-by-year, season-by-season, or even for particular policy changes or events—to detect strengths or weaknesses in the model across different scenarios. * [ ] Use graphical comparisons (e.g., time series plots) or statistical measures (e.g., goodness-of-fit, mean errors, confidence intervals) to assess how closely the model matches reality - see below. -Graphical comparison - -* [ ] Create time-series plots of key metrics (e.g., daily patient arrivals, resource utilisation) for both the model and system. Create distributions of waiting times, service times, and other performance measures. Compare the model and system graphs. - -Statistical comparison - -* [ ] Collect real system data on key performance measures (e.g., wait times, lengths of stay, throughput) and compare with model outputs statistically using appropriate tests. - Animation visualisation * [ ] Create an animation to help with validation (as well as communicaton and reuse). @@ -75,15 +81,9 @@ Comparison testing * [ ] If you have multiple models of the same system, compare them! -Input data validation - -* [ ] Check the datasets used - screen for outliers, determine if they are correct, and if the reason for them occurring should be incorporated into the simulation. -* [ ] Ensure you have performed appropriate input modelling steps when choosing your distributions. - -Conceptual model validation +Face validation -* [ ] Document and justify all modeling assumptions. -* [ ] Review the conceptual model with people familiar with the real system to assess completeness and accuracy. +* [ ] Present key simulation outputs and model behaviour to people such as: project team members; intended users of the model (e.g., healthcare analysts, managers); people familiar with the real system (e.g., clinicians, frontline staff, patient representatives). Ask for their subjective feedback on whether the model and results "look right". Discuss specific areas, such as whether performance measures (e.g., patient flow, wait times) match expectations under similar conditions. Experimentation validation diff --git a/pages/impact/impact.qmd b/pages/impact/impact.qmd index 8d7a1a18..959b8601 100644 --- a/pages/impact/impact.qmd +++ b/pages/impact/impact.qmd @@ -19,3 +19,12 @@ The DES RAP Book is already being used and adapted in several teaching and train * **HSMA "Little Book of DES"** DES RAP Book is referenced in several places in the [Little Book of DES](https://des.hsma.co.uk/) from the [Health Service Modelling Associates (HSMA) Programme](https://hsma.co.uk/) at the University of Exeter. + +## Research + +* **NHS model reuse project (planned)** + Collaboration between King’s, The Strategy Unit and the University of Exeter, where DES models are provided to NHS analysts. It explores whether they are able to reproduce and reuse them in their own context, and the model sample includes our [python stroke example model](https://github.com/pythonhealthdatascience/pydesrap_stroke). + +* **Stroke model of Same Day Emergency Care (SDEC) and CT Perfusion (CTP) scanning** + Hyperacute and acute stroke pathway model created by John Williams at Maidstone and Tunbridge Wells NHS Trust. Now being updated using DES RAP principles to add documentation, reproducible workflows, testing and other RAP infrastructure. + \ No newline at end of file diff --git a/pages/intros/foss_resources/proprietary_foss.png b/pages/intros/foss_resources/proprietary_foss.png index 5ed6b15f..ff771184 100644 Binary files a/pages/intros/foss_resources/proprietary_foss.png and b/pages/intros/foss_resources/proprietary_foss.png differ diff --git a/pages/intros/foss_resources/proprietary_foss.svg b/pages/intros/foss_resources/proprietary_foss.svg index 2ccf1e73..b221cd44 100644 --- a/pages/intros/foss_resources/proprietary_foss.svg +++ b/pages/intros/foss_resources/proprietary_foss.svg @@ -1,44 +1,38 @@ - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + \ No newline at end of file