From 08e6bdd3de4145e8e48f013e437abd5775bcd63a Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Fabian=20Fr=C3=B6hlich?= Date: Tue, 22 Apr 2025 16:52:52 +0100 Subject: [PATCH 01/15] update spec --- doc/v2/_static/petab_schema_v2.yaml | 138 ++++++++++++--------------- doc/v2/documentation_data_format.rst | 74 +++++++++----- 2 files changed, 110 insertions(+), 102 deletions(-) diff --git a/doc/v2/_static/petab_schema_v2.yaml b/doc/v2/_static/petab_schema_v2.yaml index b09421b2..8f246b2d 100644 --- a/doc/v2/_static/petab_schema_v2.yaml +++ b/doc/v2/_static/petab_schema_v2.yaml @@ -22,96 +22,77 @@ properties: File name (absolute or relative) or URL to PEtab parameter table containing parameters of all models listed in `problems`. A single table may be split into multiple files and described as an array here. - problems: - type: array - description: | - One or multiple PEtab problems (sets of model, condition, observable - and measurement files). If different model and data files are - independent, they can be specified as separate PEtab problems, which - may allow more efficient handling. Files in one problem cannot refer - to models entities or data specified inside another problem. - items: - type: object - description: | - A set of PEtab model, observable and measurement - files and optional condition, experiment, and visualization files. - - properties: - - model_files: - type: object - description: One or multiple models - - # the model ID - patternProperties: - "^[a-zA-Z_]\\w*$": - type: object - properties: - location: - type: string - description: Model file name or URL - language: - type: string - description: | - Model language, e.g., 'sbml', 'cellml', 'bngl', 'pysb' - required: - - location - - language - additionalProperties: false - - measurement_files: - type: array - description: List of PEtab measurement files. - - items: + model_files: + type: object + description: One or multiple models + + # the model ID + patternProperties: + "^[a-zA-Z_]\\w*$": + type: object + properties: + location: type: string - description: PEtab measurement file name or URL. + description: Model file name or URL + language: + type: string + description: | + Model language, e.g., 'sbml', 'cellml', 'bngl', 'pysb' + required: + - location + - language + additionalProperties: false - condition_files: - type: array - description: List of PEtab condition files. + measurement_files: + type: array + description: List of PEtab measurement files. - items: - type: string - description: PEtab condition file name or URL. + items: + type: string + description: PEtab measurement file name or URL. + + condition_files: + type: array + description: List of PEtab condition files. - experiment_files: - type: array - description: List of PEtab experiment files + items: + type: string + description: PEtab condition file name or URL. - items: - type: string - description: PEtab experiment file name or URL. + experiment_files: + type: array + description: List of PEtab experiment files - observable_files: - type: array - description: List of PEtab observable files. + items: + type: string + description: PEtab experiment file name or URL. - items: - type: string - description: PEtab observable file name or URL. + observable_files: + type: array + description: List of PEtab observable files. + + items: + type: string + description: PEtab observable file name or URL. - visualization_files: - type: array - description: List of PEtab visualization files. + visualization_files: + type: array + description: List of PEtab visualization files. - items: - type: string - description: PEtab visualization file name or URL. + items: + type: string + description: PEtab visualization file name or URL. - mapping_files: - type: array - description: List of PEtab mapping files. + mapping_files: + type: array + description: List of PEtab mapping files. - items: - type: string - description: PEtab mapping file name or URL. + items: + type: string + description: PEtab mapping file name or URL. required: - - model_files - - observable_files - - measurement_files extensions: type: object @@ -142,4 +123,7 @@ properties: required: - format_version - parameter_file - - problems + - model_files + - observable_files + - measurement_files + diff --git a/doc/v2/documentation_data_format.rst b/doc/v2/documentation_data_format.rst index 22c0d4bd..722987ac 100644 --- a/doc/v2/documentation_data_format.rst +++ b/doc/v2/documentation_data_format.rst @@ -53,7 +53,7 @@ text-based files in `Tab-Separated Values (TSV) `_ format (Figure 2), including: -- A :ref:`model ` file specifying the base model +- One or multiple (optional) :ref:`model ` file(s) specifying the base model(s) [SBML, CELLML, BNGL, PYSB, ...]. - A :ref:`measurement file ` containing experimental @@ -153,6 +153,7 @@ PEtab 2.0.0 is a major update of the PEtab format. The main changes are: the PEtab format. * The admissible values for ``estimate`` in the :ref:`v2_parameters_table` are now ``true`` and ``false`` instead of ``1`` and ``0``. +* .. _v2_model: .. _v2_model_entities: @@ -185,10 +186,10 @@ PEtab distinguishes between three types of entities: Conditions table ---------------- -The optional conditions table defines discrete changes to the model. These (sets of) -changes typically represent interventions, perturbations, or changes in the -environment of the system of interest. These modifications are referred to as -(experimental) *conditions*. +The optional conditions table defines discrete changes to the simulated model. +These (sets of) changes typically represent interventions, perturbations, or +changes in the environment of the system of interest. These modifications are +referred to as experimental) *conditions*. Conditions are applied at specific time points, which are defined in the :ref:`v2_experiments_table`. This allows for the specification of time @@ -278,6 +279,9 @@ phases, following the logic of the event assignments in a single SBML event. * For subsequent time periods, the *current* values are taken from the simulation results at the end of the preceding time period. + * For problems involving multiple models, any variable not present in + the currently simulated model will default to a value of zero. + 2. **Assignment of the evaluated ``targetValues`` to their targets** All evaluated ``targetValues`` are simultaneously assigned to their @@ -295,6 +299,9 @@ phases, following the logic of the event assignments in a single SBML event. For further details, refer to SBML semantic test suite case `01779 `_. + * For problems involving multiple models, any assignment to targets not + present in the currently simulated model will be ignored. + 3. **Update of derived variables** After target values have been assigned, all derived variables (i.e., @@ -444,13 +451,17 @@ order: Additional (non-standard) columns may be added. If the additional plotting functionality of PEtab should be used, such columns could be -+-----+-------------+---------------+ -| ... | [datasetId] | [replicateId] | -+=====+=============+===============+ -| ... | [datasetId] | [replicateId] | -+-----+-------------+---------------+ -| ... | ... | ... | -+-----+-------------+---------------+ ++-----+-------------+---------------+-----------+ +| ... | [datasetId] | [replicateId] | [modelId] | ++=====+=============+===============+===========+ +| ... | [datasetId] | [replicateId] | [modelId] | ++-----+-------------+---------------+-----------+ +| ... | | | | ++-----+-------------+---------------+-----------+ +| ... | | | | ++-----+-------------+---------------+-----------+ +| ... | ... | ... | | ++-----+-------------+---------------+-----------+ where ``datasetId`` is a necessary column to use particular plotting functionality, and ``replicateId`` is optional, which can be used to group @@ -533,6 +544,14 @@ Detailed field description The replicateId can be used to discern replicates with the same ``datasetId``, which is helpful for plotting e.g. error bars. +- ``modelId`` [PETAB_ID, OPTIONAL, REFERENCES(yaml.models.model_id)] + + The modelId specifies which model to simulate for each data point. modelIds + are defined by the keys of the models dict in the PEtab problem YAML file. + This column is required when multiple models are defined in the PEtab problem. + For problems with a single model, this column is ignored, and the same model + is used for all simulations. + .. _v2_observables_table: Observables table @@ -593,12 +612,15 @@ Detailed field description * ``observableFormula`` [STRING] Observation function as plain text formula expression. - May contain any symbol defined in the SBML model (including model time ``time``) - or parameter table. In the simplest case just an SBML species ID - or an ``AssignmentRule`` target. Additionally, any observable ID + May contain any symbol defined any of the employed models (including model + time ``time``) or parameter table. In the simplest case, for SBML models, + just a species ID or an ``AssignmentRule`` target. Additionally, any observable ID introduced in the observable table may be referenced, but circular definitions must be avoided. + In problems with multiple models, symbols not defined in the currently simulated + model default to a value of zero. + May introduce new parameters of the form ``observableParameter${n}_${observableId}``, which are overridden by ``observableParameters`` in the measurement table (see description there). @@ -719,7 +741,7 @@ and *must not* include: - "Parameters" that are not *constant* entities (e.g., in an SBML model, the targets of *AssignmentRules* or *EventAssignments*) - Any parameters that do not have valid PEtab IDs - (e.g., SBML *local* parameters) + (e.g., SBML *local* parameters that are not mapped in the mapping table) it *may* include: @@ -758,7 +780,7 @@ Detailed field description - ``parameterId`` [PETAB_ID, REQUIRED] The ``parameterId`` of the parameter described in this row. This has to match - the ID of a parameter specified in the SBML model, a parameter introduced + the ID of a parameter specified any model, a parameter introduced as override in the condition table, or a parameter occurring in the ``observableParameters`` or ``noiseParameters`` column of the measurement table (see above). @@ -990,8 +1012,8 @@ Detailed field description - ``modelEntityId`` [STRING or empty, REQUIRED] - A globally unique identifier defined in the model, or empty if the entity is - not present in the model. This does not have to be a valid PEtab identifier. + A globally unique identifier defined in any model, or empty if the entity is + not present any model. This does not have to be a valid PEtab identifier. Rows with empty ``modelEntityId`` serve as annotations only. For example, in SBML, local parameters may be referenced as @@ -1033,12 +1055,14 @@ easy validation: Parameter estimation problems combining multiple models ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ -Parameter estimation problems can comprise multiple models. For now, PEtab -allows one to specify multiple models with corresponding condition and -measurement tables, and one joint parameter table. This means that the parameter -namespace is global. Therefore, parameters with the same ID in different models -will be considered identical. - +Parameter estimation problems can involve multiple models, with a specific model +assigned to each individual data point. However, all tables in a PEtab problem apply +uniformly to all models. As a result: +- Multiple models may need to be simulated for a single experiment. +- Each model may be simulated for a distinct subset of experiments. +- The number of conditions that must be simulated for an experiment may vary depending + on the model used. +- Each parameter in the parameter table hase the same value across all models. .. _v2_objective_function: From 975d5ae5a45b48741fcb380d1416cdada5e17e0f Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Fabian=20Fr=C3=B6hlich?= Date: Wed, 23 Apr 2025 10:35:01 +0100 Subject: [PATCH 02/15] Apply suggestions from code review Co-authored-by: Dilan Pathirana <59329744+dilpath@users.noreply.github.com> Co-authored-by: Sebastian Persson <46872750+sebapersson@users.noreply.github.com> --- doc/v2/documentation_data_format.rst | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/doc/v2/documentation_data_format.rst b/doc/v2/documentation_data_format.rst index 722987ac..10ffa07b 100644 --- a/doc/v2/documentation_data_format.rst +++ b/doc/v2/documentation_data_format.rst @@ -189,7 +189,7 @@ Conditions table The optional conditions table defines discrete changes to the simulated model. These (sets of) changes typically represent interventions, perturbations, or changes in the environment of the system of interest. These modifications are -referred to as experimental) *conditions*. +referred to as (experimental) *conditions*. Conditions are applied at specific time points, which are defined in the :ref:`v2_experiments_table`. This allows for the specification of time @@ -780,7 +780,7 @@ Detailed field description - ``parameterId`` [PETAB_ID, REQUIRED] The ``parameterId`` of the parameter described in this row. This has to match - the ID of a parameter specified any model, a parameter introduced + the ID of a parameter specified at least one model, a parameter introduced as override in the condition table, or a parameter occurring in the ``observableParameters`` or ``noiseParameters`` column of the measurement table (see above). @@ -1013,7 +1013,7 @@ Detailed field description - ``modelEntityId`` [STRING or empty, REQUIRED] A globally unique identifier defined in any model, or empty if the entity is - not present any model. This does not have to be a valid PEtab identifier. + not present in any model. This does not have to be a valid PEtab identifier. Rows with empty ``modelEntityId`` serve as annotations only. For example, in SBML, local parameters may be referenced as @@ -1062,7 +1062,7 @@ uniformly to all models. As a result: - Each model may be simulated for a distinct subset of experiments. - The number of conditions that must be simulated for an experiment may vary depending on the model used. -- Each parameter in the parameter table hase the same value across all models. +- Each parameter in the parameter table has the same value across all models. .. _v2_objective_function: From 73d6b28e99759dce2c2767e66142deb1d2fe73d1 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Fabian=20Fr=C3=B6hlich?= Date: Thu, 22 May 2025 10:29:28 +0100 Subject: [PATCH 03/15] update invalid conditions/observables --- doc/v2/documentation_data_format.rst | 15 ++++++++------- 1 file changed, 8 insertions(+), 7 deletions(-) diff --git a/doc/v2/documentation_data_format.rst b/doc/v2/documentation_data_format.rst index 0198a5e8..2f500bad 100644 --- a/doc/v2/documentation_data_format.rst +++ b/doc/v2/documentation_data_format.rst @@ -282,9 +282,6 @@ phases, following the logic of the event assignments in a single SBML event. * For subsequent time periods, the *current* values are taken from the simulation results at the end of the preceding time period. - * For problems involving multiple models, any variable not present in - the currently simulated model will default to a value of zero. - 2. **Assignment of the evaluated ``targetValues`` to their targets** All evaluated ``targetValues`` are simultaneously assigned to their @@ -558,6 +555,13 @@ Detailed field description For problems with a single model, this column is ignored, and the same model is used for all simulations. + For model definitions that include multiple models, only those experiments and + bservables that appear in the same rows of the measurement table need to be + valid for a given model. This means that all symbols used in the corresponding + observable formulas, as well as all symbols assigned in the associated + conditions, must be present in that specific model. Conditions and observables + do not need to be valid for models to which they are not applied. + .. _v2_observables_table: Observables table @@ -618,15 +622,12 @@ Detailed field description * ``observableFormula`` [STRING] Observation function as plain text formula expression. - May contain any symbol defined any of the employed models (including model + May contain any symbol defined in a model (including model time ``time``) or parameter table. In the simplest case, for SBML models, just a species ID or an ``AssignmentRule`` target. Additionally, any observable ID introduced in the observable table may be referenced, but circular definitions must be avoided. - In problems with multiple models, symbols not defined in the currently simulated - model default to a value of zero. - May introduce new parameters of the form ``observableParameter${n}_${observableId}``, which are overridden by ``observableParameters`` in the measurement table (see description there). From 0c23354a313c93e4f575d25b619667a679b8281f Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Fabian=20Fr=C3=B6hlich?= Date: Thu, 22 May 2025 12:51:44 +0100 Subject: [PATCH 04/15] update multi-model description --- doc/v2/documentation_data_format.rst | 60 +++++++++++++++++++++------- 1 file changed, 45 insertions(+), 15 deletions(-) diff --git a/doc/v2/documentation_data_format.rst b/doc/v2/documentation_data_format.rst index 2f500bad..ea4865ed 100644 --- a/doc/v2/documentation_data_format.rst +++ b/doc/v2/documentation_data_format.rst @@ -555,13 +555,6 @@ Detailed field description For problems with a single model, this column is ignored, and the same model is used for all simulations. - For model definitions that include multiple models, only those experiments and - bservables that appear in the same rows of the measurement table need to be - valid for a given model. This means that all symbols used in the corresponding - observable formulas, as well as all symbols assigned in the associated - conditions, must be present in that specific model. Conditions and observables - do not need to be valid for models to which they are not applied. - .. _v2_observables_table: Observables table @@ -1147,14 +1140,51 @@ easy validation: Parameter estimation problems combining multiple models ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ -Parameter estimation problems can involve multiple models, with a specific model -assigned to each individual data point. However, all tables in a PEtab problem apply -uniformly to all models. As a result: -- Multiple models may need to be simulated for a single experiment. -- Each model may be simulated for a distinct subset of experiments. -- The number of conditions that must be simulated for an experiment may vary depending - on the model used. -- Each parameter in the parameter table has the same value across all models. +Purpose +^^^^^^^ +PEtab supports defining multiple models within a single problem specification. This +feature is designed to enable users to define experiment-specific model variants or +submodels. Rather than implementing a single global, parameterized model, users can +define multiple smaller, self-contained models that differ structurally as needed. + +This approach offers several benefits: + +- Simplified model definition for users, as each variant can be independently +specified. +- Improved simulation performance for tool developers, as smaller models can be +simulated more efficiently. + +Scope and Application +^^^^^^^^^^^^^^^^^^^^^ +While multiple models are intended to be applied to different experiments, model +selection is specified at the level of individual data points in the +:ref:`v2_measurements_table`. This design enables: + +- Reuse of experiments across models. +- Fine-grained model-to-data assignment. + +With the exception of the :ref:`v2_measurements_table`, all other PEtab tables apply +uniformly to all models. + +This design has several implications: + +- A single experiment may require simulations using multiple models. +- Each model may be associated with a distinct subset of experiments. +- The number of conditions to be simulated for a model-specific instance +of an experiment may vary across models. +- Each parameter defined in the :ref:`v2_parameters_table` has a shared value across all +models. + +Validation Rules +^^^^^^^^^^^^^^^^ +For any given model Only those experiments and observables that appear in the +same rows of the :ref:`v2_measurements_table` need to be valid. This means that all +symbols used in the corresponding ``observableFormula`` and all symbols assigned +in the associated condition definitions must be defined in the model. + +Conditions and observables that are not applied to a model do not need to be +valid for that model. + .. _v2_objective_function: From 1a1a93fd6c570c94e18396687babb13a97092d7a Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Fabian=20Fr=C3=B6hlich?= Date: Fri, 23 May 2025 08:28:24 +0100 Subject: [PATCH 05/15] Apply suggestions from code review Co-authored-by: Dilan Pathirana <59329744+dilpath@users.noreply.github.com> --- doc/v2/documentation_data_format.rst | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/doc/v2/documentation_data_format.rst b/doc/v2/documentation_data_format.rst index f9f10ace..71ec3c83 100644 --- a/doc/v2/documentation_data_format.rst +++ b/doc/v2/documentation_data_format.rst @@ -786,7 +786,7 @@ Detailed field description - ``parameterId`` [PETAB_ID, REQUIRED] The ``parameterId`` of the parameter described in this row. This has to match - the ID of a parameter specified at least one model, a parameter introduced + the ID of a parameter specified in at least one model, a parameter introduced as override in the condition table, or a parameter occurring in the ``observableParameters`` or ``noiseParameters`` column of the measurement table (see above). @@ -1183,7 +1183,7 @@ models. Validation Rules ^^^^^^^^^^^^^^^^ -For any given model Only those experiments and observables that appear in the +For any given model, only those experiments and observables that appear in the same rows of the :ref:`v2_measurements_table` need to be valid. This means that all symbols used in the corresponding ``observableFormula`` and all symbols assigned in the associated condition definitions must be defined in the model. From 628b0888ed60f9e6335a9dcacc6a22afc413326e Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Fabian=20Fr=C3=B6hlich?= Date: Fri, 23 May 2025 08:28:58 +0100 Subject: [PATCH 06/15] remove missing assignments --- doc/v2/documentation_data_format.rst | 3 --- 1 file changed, 3 deletions(-) diff --git a/doc/v2/documentation_data_format.rst b/doc/v2/documentation_data_format.rst index 71ec3c83..3523fc2f 100644 --- a/doc/v2/documentation_data_format.rst +++ b/doc/v2/documentation_data_format.rst @@ -304,9 +304,6 @@ phases, following the logic of the event assignments in a single SBML event. For further details, refer to SBML semantic test suite case `01779 `_. - * For problems involving multiple models, any assignment to targets not - present in the currently simulated model will be ignored. - 3. **Update of derived variables** After target values have been assigned, all derived variables (i.e., From d9ad66425e16b2158423d98b997fe2f56bd0f6a8 Mon Sep 17 00:00:00 2001 From: Daniel Weindl Date: Wed, 4 Jun 2025 08:49:51 +0200 Subject: [PATCH 07/15] rst: fix lists --- doc/v2/documentation_data_format.rst | 36 ++++++++++++++-------------- 1 file changed, 18 insertions(+), 18 deletions(-) diff --git a/doc/v2/documentation_data_format.rst b/doc/v2/documentation_data_format.rst index b5af2a93..2427df17 100644 --- a/doc/v2/documentation_data_format.rst +++ b/doc/v2/documentation_data_format.rst @@ -195,9 +195,9 @@ PEtab distinguishes between three types of entities: Conditions table ---------------- -The optional conditions table defines discrete changes to the simulated model. -These (sets of) changes typically represent interventions, perturbations, or -changes in the environment of the system of interest. These modifications are +The optional conditions table defines discrete changes to the simulated model. +These (sets of) changes typically represent interventions, perturbations, or +changes in the environment of the system of interest. These modifications are referred to as (experimental) *conditions*. Conditions are applied at specific time points, which are defined in the @@ -526,7 +526,7 @@ Detailed field description The modelId specifies which model to simulate for each data point. modelIds are defined by the keys of the models dict in the PEtab problem YAML file. - This column is required when multiple models are defined in the PEtab problem. + This column is required when multiple models are defined in the PEtab problem. For problems with a single model, this column is ignored, and the same model is used for all simulations. @@ -591,8 +591,8 @@ Detailed field description * ``observableFormula`` [STRING] Observation function as plain text formula expression. - May contain any symbol defined in a model (including model - time ``time``) or parameter table. In the simplest case, for SBML models, + May contain any symbol defined in a model (including model + time ``time``) or parameter table. In the simplest case, for SBML models, just a species ID or an ``AssignmentRule`` target. Additionally, any observable ID introduced in the observable table may be referenced, but circular definitions must be avoided. @@ -993,15 +993,15 @@ define multiple smaller, self-contained models that differ structurally as neede This approach offers several benefits: -- Simplified model definition for users, as each variant can be independently -specified. -- Improved simulation performance for tool developers, as smaller models can be -simulated more efficiently. +- Simplified model definition for users, as each variant can be independently + specified. +- Improved simulation performance for tool developers, as smaller models can be + simulated more efficiently. Scope and Application ^^^^^^^^^^^^^^^^^^^^^ While multiple models are intended to be applied to different experiments, model -selection is specified at the level of individual data points in the +selection is specified at the level of individual data points in the :ref:`v2_measurements_table`. This design enables: - Reuse of experiments across models. @@ -1015,18 +1015,18 @@ This design has several implications: - A single experiment may require simulations using multiple models. - Each model may be associated with a distinct subset of experiments. - The number of conditions to be simulated for a model-specific instance -of an experiment may vary across models. -- Each parameter defined in the :ref:`v2_parameters_table` has a shared value across all -models. + of an experiment may vary across models. +- Each parameter defined in the :ref:`v2_parameters_table` has a shared value + across all models. Validation Rules ^^^^^^^^^^^^^^^^ -For any given model, only those experiments and observables that appear in the -same rows of the :ref:`v2_measurements_table` need to be valid. This means that all -symbols used in the corresponding ``observableFormula`` and all symbols assigned +For any given model, only those experiments and observables that appear in the +same rows of the :ref:`v2_measurements_table` need to be valid. This means that all +symbols used in the corresponding ``observableFormula`` and all symbols assigned in the associated condition definitions must be defined in the model. -Conditions and observables that are not applied to a model do not need to be +Conditions and observables that are not applied to a model do not need to be valid for that model. From 637c570063f271cb55f6ceff8d9cfbdb1ac4573a Mon Sep 17 00:00:00 2001 From: Daniel Weindl Date: Wed, 4 Jun 2025 08:56:39 +0200 Subject: [PATCH 08/15] rst: title levels --- doc/v2/documentation_data_format.rst | 9 ++++++--- 1 file changed, 6 insertions(+), 3 deletions(-) diff --git a/doc/v2/documentation_data_format.rst b/doc/v2/documentation_data_format.rst index 2427df17..087a8342 100644 --- a/doc/v2/documentation_data_format.rst +++ b/doc/v2/documentation_data_format.rst @@ -985,7 +985,8 @@ Parameter estimation problems combining multiple models ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Purpose -^^^^^^^ ++++++++ + PEtab supports defining multiple models within a single problem specification. This feature is designed to enable users to define experiment-specific model variants or submodels. Rather than implementing a single global, parameterized model, users can @@ -999,7 +1000,8 @@ This approach offers several benefits: simulated more efficiently. Scope and Application -^^^^^^^^^^^^^^^^^^^^^ ++++++++++++++++++++++ + While multiple models are intended to be applied to different experiments, model selection is specified at the level of individual data points in the :ref:`v2_measurements_table`. This design enables: @@ -1020,7 +1022,8 @@ This design has several implications: across all models. Validation Rules -^^^^^^^^^^^^^^^^ +++++++++++++++++ + For any given model, only those experiments and observables that appear in the same rows of the :ref:`v2_measurements_table` need to be valid. This means that all symbols used in the corresponding ``observableFormula`` and all symbols assigned From 9194c63a2873944000e34668e51c90b959111016 Mon Sep 17 00:00:00 2001 From: Daniel Weindl Date: Thu, 5 Jun 2025 09:42:10 +0200 Subject: [PATCH 09/15] list of changes --- doc/v2/documentation_data_format.rst | 9 +++++++++ 1 file changed, 9 insertions(+) diff --git a/doc/v2/documentation_data_format.rst b/doc/v2/documentation_data_format.rst index 0a576337..2a9482c1 100644 --- a/doc/v2/documentation_data_format.rst +++ b/doc/v2/documentation_data_format.rst @@ -111,12 +111,20 @@ should not alter the definition of the estimation problem itself. parameters, but excluding parameters listed in the :ref:`v2_parameters_table` (independent of their ``estimate`` value). +.. _v2_changes: Changes from PEtab 1.0.0 ------------------------ PEtab 2.0.0 is a major update of the PEtab format. The main changes are: +* The :ref:`PEtab YAML problem description ` is now + mandatory and its structure changed. In particular, the `problems` list + has been flattened. * Support for models in other formats than SBML (:ref:`v2_model`). +* The use of different models for different measurements is now + supported via the optional ``modelId`` column in the + :ref:`v2_measurements_table`, see also :ref:`v2_multiple_models`. + This was poorly defined in PEtab 1.0.0 and probably not used in practice. * The (now optional) conditions table format changed from wide to long (:ref:`v2_conditions_table`). * ``simulationConditionId`` and ``preequilibrationConditionId`` in the @@ -1020,6 +1028,7 @@ easy validation: .. literalinclude:: _static/petab_schema_v2.yaml :language: yaml +.. _v2_multiple_models: Parameter estimation problems combining multiple models ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ From 468d6799ef34cf1b9353d0f9bad62f3ff1929524 Mon Sep 17 00:00:00 2001 From: Daniel Weindl Date: Thu, 5 Jun 2025 09:46:31 +0200 Subject: [PATCH 10/15] Drop vis table from yaml (missed in https://github.com/PEtab-dev/PEtab/pull/631); fix dtype --- doc/v2/_static/petab_schema_v2.yaml | 13 +------------ 1 file changed, 1 insertion(+), 12 deletions(-) diff --git a/doc/v2/_static/petab_schema_v2.yaml b/doc/v2/_static/petab_schema_v2.yaml index 8f246b2d..77f29ab6 100644 --- a/doc/v2/_static/petab_schema_v2.yaml +++ b/doc/v2/_static/petab_schema_v2.yaml @@ -76,14 +76,6 @@ properties: type: string description: PEtab observable file name or URL. - visualization_files: - type: array - description: List of PEtab visualization files. - - items: - type: string - description: PEtab visualization file name or URL. - mapping_files: type: array description: List of PEtab mapping files. @@ -92,8 +84,6 @@ properties: type: string description: PEtab mapping file name or URL. - required: - extensions: type: object description: | @@ -109,7 +99,7 @@ properties: type: string pattern: ^([1-9][0-9]*!)?(0|[1-9][0-9]*)(\.(0|[1-9][0-9]*))*((a|b|rc)(0|[1-9][0-9]*))?(\.post(0|[1-9][0-9]*))?(\.dev(0|[1-9][0-9]*))?$ required: - type: bool + type: boolean description: | Indicates whether the extension is required for the mathematical interpretation of the problem. @@ -126,4 +116,3 @@ required: - model_files - observable_files - measurement_files - From fc2b34cf239dfb8737d6a5cfefb630453ba45e09 Mon Sep 17 00:00:00 2001 From: Daniel Weindl Date: Thu, 5 Jun 2025 09:47:40 +0200 Subject: [PATCH 11/15] clarify --- doc/v2/documentation_data_format.rst | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-) diff --git a/doc/v2/documentation_data_format.rst b/doc/v2/documentation_data_format.rst index 2a9482c1..4eedf7a4 100644 --- a/doc/v2/documentation_data_format.rst +++ b/doc/v2/documentation_data_format.rst @@ -1063,7 +1063,10 @@ uniformly to all models. This design has several implications: -- A single experiment may require simulations using multiple models. +- A single experiment may need to be simulated with different models for + different measurements. + However, a single simulation of a given experiment is always performed + using one single model - Each model may be associated with a distinct subset of experiments. - The number of conditions to be simulated for a model-specific instance of an experiment may vary across models. From 449668858c6f636310308bee172c270d8d7b31ca Mon Sep 17 00:00:00 2001 From: Daniel Weindl Date: Fri, 6 Jun 2025 10:45:48 +0200 Subject: [PATCH 12/15] Update doc/v2/documentation_data_format.rst Co-authored-by: Sebastian Persson <46872750+sebapersson@users.noreply.github.com> --- doc/v2/documentation_data_format.rst | 9 +++------ 1 file changed, 3 insertions(+), 6 deletions(-) diff --git a/doc/v2/documentation_data_format.rst b/doc/v2/documentation_data_format.rst index 4eedf7a4..6028a782 100644 --- a/doc/v2/documentation_data_format.rst +++ b/doc/v2/documentation_data_format.rst @@ -541,12 +541,9 @@ Detailed field description - ``modelId`` [PETAB_ID, OPTIONAL, REFERENCES(yaml.models.model_id)] - The modelId specifies which model to simulate for each data point. modelIds - are defined by the keys of the models dict in the PEtab problem YAML file. - This column is required when multiple models are defined in the PEtab problem. - For problems with a single model, this column is ignored, and the same model - is used for all simulations. - + Which model to simulate for each data point. modelIds are defined by the keys of + the models dict in the PEtab problem YAML file. This column is required when + multiple models are defined in the PEtab problem, otherwise it is ignored. .. _v2_simulation_table: Simulation table From 9dc1289c017be4e572486d92e1bf1f8a0196a06c Mon Sep 17 00:00:00 2001 From: Daniel Weindl Date: Fri, 6 Jun 2025 10:57:46 +0200 Subject: [PATCH 13/15] fixup --- doc/v2/documentation_data_format.rst | 10 +++++++--- 1 file changed, 7 insertions(+), 3 deletions(-) diff --git a/doc/v2/documentation_data_format.rst b/doc/v2/documentation_data_format.rst index 6028a782..b1e04ce9 100644 --- a/doc/v2/documentation_data_format.rst +++ b/doc/v2/documentation_data_format.rst @@ -541,9 +541,13 @@ Detailed field description - ``modelId`` [PETAB_ID, OPTIONAL, REFERENCES(yaml.models.model_id)] - Which model to simulate for each data point. modelIds are defined by the keys of - the models dict in the PEtab problem YAML file. This column is required when - multiple models are defined in the PEtab problem, otherwise it is ignored. + Which model to simulate for each data point. Model IDs are defined by the + keys of the `models` object in the PEtab problem YAML file. + This column is required when multiple models are defined in the PEtab + problem. + For problems with a single model, this column is optional, + and its values default to the ID of the only model present. + .. _v2_simulation_table: Simulation table From db906a6f6e76f7f9cf3613aef9f8ce3d44a374c8 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Fabian=20Fr=C3=B6hlich?= Date: Wed, 2 Jul 2025 09:14:54 +0100 Subject: [PATCH 14/15] update multi-model spec --- doc/v2/documentation_data_format.rst | 17 +++++++++++------ 1 file changed, 11 insertions(+), 6 deletions(-) diff --git a/doc/v2/documentation_data_format.rst b/doc/v2/documentation_data_format.rst index b1e04ce9..bbfe439e 100644 --- a/doc/v2/documentation_data_format.rst +++ b/doc/v2/documentation_data_format.rst @@ -1059,20 +1059,25 @@ selection is specified at the level of individual data points in the - Reuse of experiments across models. - Fine-grained model-to-data assignment. -With the exception of the :ref:`v2_measurements_table`, all other PEtab tables apply -uniformly to all models. +With the exception of the :ref:`v2_measurements_table`, all other PEtab tables apply +to all models. Parameters listed in the parameter table are defined globally and +shared across all models. In contrast, entries in all other tables implicitly define +model-specific instances of observables, conditions, experiments, etc., with their +respective PEtab IDs existing in local, model-specific namespaces. Each PEtab +subproblem defined in this way must constitute a valid PEtab problem on its own. This design has several implications: - A single experiment may need to be simulated with different models for - different measurements. - However, a single simulation of a given experiment is always performed - using one single model + different measurements. However, a single simulation of a given experiment + is always performed using one single model. - Each model may be associated with a distinct subset of experiments. - The number of conditions to be simulated for a model-specific instance of an experiment may vary across models. - Each parameter defined in the :ref:`v2_parameters_table` has a shared value - across all models. + across all models. Parameters not listed in the parameter table do not share + values, which can result in model-specific instantiations of model observables + referencing these parameters. Validation Rules ++++++++++++++++ From 69dae687145ca794f0a96774eee93e2bb897c46b Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Fabian=20Fr=C3=B6hlich?= Date: Wed, 2 Jul 2025 13:56:59 +0100 Subject: [PATCH 15/15] review comment --- doc/v2/documentation_data_format.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/doc/v2/documentation_data_format.rst b/doc/v2/documentation_data_format.rst index b1e04ce9..b1fd1e4f 100644 --- a/doc/v2/documentation_data_format.rst +++ b/doc/v2/documentation_data_format.rst @@ -213,7 +213,7 @@ PEtab distinguishes between three types of entities: Conditions table ---------------- -The optional conditions table defines discrete changes to the simulated model. +The optional conditions table defines discrete changes to the simulated model(s). These (sets of) changes typically represent interventions, perturbations, or changes in the environment of the system of interest. These modifications are referred to as (experimental) *conditions*.