Skip to content

Unify config files to expedition.yaml#217

Merged
j-atkins merged 15 commits intov1-devfrom
consolidate-configs
Oct 21, 2025
Merged

Unify config files to expedition.yaml#217
j-atkins merged 15 commits intov1-devfrom
consolidate-configs

Conversation

@j-atkins
Copy link
Collaborator

@j-atkins j-atkins commented Oct 17, 2025 •
edited
Loading

This PR consolidates the old dual ship_config.yaml and schedule.yaml config files into one expedition.yaml file, as outlined as an objective in the VirtualShip design doc for v1-dev.

  • Tests added

erikvansebille
erikvansebille reviewed Oct 20, 2025
View reviewed changes
@j-atkins
Copy link
Collaborator Author

j-atkins commented Oct 20, 2025

Thanks for the comments @erikvansebille, changes now implemented.

In summary:

  • The ShipConfig model is now removed, given that - apart from the ship speed - it mostly contained configs related to the instruments. This has been replaced by an InstrumentsConfig model instead for the instrument configurations.
  • Accordingly, ship_config is replaced by instruments_config top-level key in expedition.yaml.
  • ship_speed_knots is moved to its own top-level YAML key.

The expedition.yaml structure now looks something like:

schedule:
  space_time_region:
      ...
  waypoints:
    ...
instruments_config:
  adcp_config:
    ...
ship_speed_knots: ...

@erikvansebille
Copy link
Member

erikvansebille commented Oct 20, 2025

Very nice improvement! One small suggestion left: you could move ship_speed to under the (revived) ship_config, which could then (later) also house other ship information like number of berths (if/when relevant for e.g. planning shifts), icestrength class, max days at sea, etc?

@j-atkins
Copy link
Collaborator Author

j-atkins commented Oct 20, 2025

Ah yes I thought about to leave room for more easily expanding the configuration options down the line. I didn't in the end for the sake of reducing the number of layers in the YAML, but I will now do that restructuring given there's agreement it could be useful!

@j-atkins
Copy link
Collaborator Author

j-atkins commented Oct 20, 2025

Very nice improvement! One small suggestion left: you could move ship_speed to under the (revived) ship_config, which could then (later) also house other ship information like number of berths (if/when relevant for e.g. planning shifts), icestrength class, max days at sea, etc?

ship_config now revived 👍

@j-atkins
Copy link
Collaborator Author

j-atkins commented Oct 21, 2025

Are you happy for me to merge into v1-dev @erikvansebille?

@erikvansebille
Copy link
Member

erikvansebille commented Oct 21, 2025

Yep, go for it!

@j-atkins j-atkins merged commit a4f9deb into v1-dev Oct 21, 2025
9 of 11 checks passed
@j-atkins j-atkins deleted the consolidate-configs branch October 21, 2025 11:39
j-atkins added a commit that referenced this pull request Nov 21, 2025
@j-atkins @VeckoTheGecko
@pre-commit-ci @erikvansebille
v0.3 dev (#226)
3bc4324 
Introduces substantial refactoring, new features, and documentation updates to VirtualShip. Overall, the changes aim to unify configuration, centralise instrument logic and overhaul data ingestion.

-----

* Unify config files to expedition.yaml (#217)

Consolidates/unifies the old dual ship_config.yaml and schedule.yaml config files into one expedition.yaml file, in line with  v1 dev objectives.

* Update link to website (#215)

* first refactoring step, parent instrument classes

* ignore refactoring notes in gitignore

* add note to remove upon completing v1 dev

* scratch inputdataset objects integration to _fetch

* add call to download_data()

* Add new instrument classes and update InputDataset to include depth parameters

* Refactor instrument handling in _fetch and update imports for consistency

* Refactor instrument classes and re-add (temporary) simulation functions across multiple files

* improve/clarify comments and notes

* Refactor ArgoFloat and XBT classes to include depth parameters and remove outdated comments

* avoid circular import issues

* make tests for InputDataset base class

* refactor instrument handling in _fetch.py and update expedition model to include get_instruments method

* refactor instrument error handling in Expedition model and remove Schedule and ShipConfig classes

* add is_underway property to InstrumentType and filter instruments in plan UI

* enhance CLI output for fetching

* general fixes and new error class

* refactor test cases to use Expedition object

* move instruments base classes out of models/ dir

* update base class imports

* make get_instruments_registry more robust with testing

* update mock reanalysis period and refactor tests to use expedition fixture

* refactor: reorganize instrument classes and update imports for clarity

* implement instrument registration and input dataset retrieval

* refactor: reorganize imports in instrument test files for consistency

* further refactoring: instrument classes to use a unified InputDataset and Instrument structure

* evaporate simulate_measurements.py; centralise run logic

* draft up check land using bathymetry

* small bug fixes

* patch copernicus product id search logic to new instrument classes, plus more debugging; verbose INFO is outstanding

* adding U and V to instruments where missing

* enhanced error messaging for XBT in too shallow regions

* bug fixes

* version U and V downloaded

* dummy U and V

* Neaten up logging output

* small bug fixes

* tidy up

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Refactor type hints and improve test coverage for instruments and utils

* Remove TODO comments and tidy up imports in test files

* Refactor bathymetry error handling, update verification methods and remove unused function

* move product id selection logic to utils

* update

* first draft direct ingestion via copernicusmarine (CTD only)

* refactor bathymetry data handling and add (temporary) timing for performance evaluation

* update instrument constructors for Copernicus Marine ingestion

* move expedition/do_expedition.py to cli/_run.py, rename Instrument.run() to Instrument.execute()

* move checkpoint class to models, move expedition_cost() to utils.py

* update imports for expedition_cost

* working with drifters (bodge), CTD_BGC not yet working

* remove fetch and all associated logic

* update docstrings/--help info

* add buffers to drifters and argos, plus depth limits for drifters

* remove _creds.py

* CTD_BGC fieldset bug fix

* fixing bugs associated with BGC data access

* update dependencies

* logic for handling copernicus credentials

* add support for taking local pre-downloaded data with --from-data optional flag in virtualship run

* update drifter to be at -1m depth, to avoid out of bounds at surface

* tidy up

* bug fixes for unnecessary copernicusmarine call when using pre-downloaded data

* remove redundant tests

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* update tests (not yet instrument subclasses)

* update instrument tests

* test is on land tests in schedule.verify()

* Run pre-commit

* update pre-download ingestion methods to take files split by time

* fix bug

* fix bug in ingesting bgc data from disk

* tidy up

* add test for data directory structure compliance

* update docs

* edits to docs

* add more checks to docs compliance testing

* TODO in readme

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Update docs/user-guide/documentation/pre_download_data.md

Co-authored-by: Erik van Sebille <erikvansebille@gmail.com>

* Apply suggestions from code review

Co-authored-by: Erik van Sebille <erikvansebille@gmail.com>

* Set t_min to first day of month for monthly resolution

Adjust t_min to the first day of the month based on schedule start date.

* remove redundant parameters from instrument classes

* change variable name

* make _find_files_in_timerange standalone from Instrument base class

* Update error docstring

* update plan UI logic to update space-time region dynamically

* fix xbt bug

* add warnings to ADCP max depth config if exceeds authentic limits

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* error messaging for case where measurements cause schedule to be missed

* revert to using ScheduleProblem class

* add more informative messaging on ScheduleProblem

* change test to mock using data from disk to avoid copernicus calls

* remove accidental breakpoint

---------

Co-authored-by: Nick Hodgskin <36369090+VeckoTheGecko@users.noreply.github.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Erik van Sebille <erikvansebille@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@erikvansebille erikvansebille erikvansebille left review comments

@VeckoTheGecko VeckoTheGecko Awaiting requested review from VeckoTheGecko

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@j-atkins @erikvansebille @VeckoTheGecko