Skip to content

Conversation

@hategan
Copy link
Collaborator

@hategan hategan commented Feb 24, 2024

This PR adds file staging to some executors (local and the batch family).

  • It does so mostly through templates.
  • It addresses the status updates issues mentioned in Added file staging job-api-spec#164 using a UDP service that simple netcat commands (and we have to carefully test there because there might be multiple versions of that tool) can send from the batch script. As a backup (if not netcat can be found), a file is used.
  • Solves the issue of the state order inversion vs. when-is-the-native-id-available in the local executor by making the local executor template based (like the batch scheduler ones) and implementing staging as it is done in the batch scheduler executors. This has the disadvantage that it only delays finding a better solution for the native_id/order issue that will inevitably happen with synthetic-staging executors, but the plus is that the local executor is closer, for testing purposes, to the batch scheduler ones.
  • Adds some testing tools for managing temporary files/directories
  • Ensures that polling threads for batch scheduler executors are shut down when the executor is GC-ed

@hategan hategan requested a review from andre-merzky February 24, 2024 18:01
@codecov
Copy link

codecov bot commented Feb 24, 2024

Codecov Report

❌ Patch coverage is 88.80597% with 75 lines in your changes missing coverage. Please review.
✅ Project coverage is 75.00%. Comparing base (0c14103) to head (01a0663).
⚠️ Report is 43 commits behind head on main.

Files with missing lines Patch % Lines
src/psij/staging.py 82.60% 20 Missing ⚠️
src/psij/utils.py 89.09% 18 Missing ⚠️
tests/_test_tools.py 76.00% 12 Missing ⚠️
tests/conftest.py 31.25% 11 Missing ⚠️
...c/psij/executors/batch/batch_scheduler_executor.py 84.90% 8 Missing ⚠️
src/psij/executors/local.py 86.66% 6 Missing ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##             main     #450      +/-   ##
==========================================
+ Coverage   74.92%   75.00%   +0.07%     
==========================================
  Files          94       91       -3     
  Lines        3972     4568     +596     
==========================================
+ Hits         2976     3426     +450     
- Misses        996     1142     +146     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

…eled

jobs and spam the pytest suite, since pytest doesn't capture the log after
the test is done.
special devices which bash will happily emulate if not found.
Since the addition of staging, errors can reasonably occur outside of
the launcher as well as inside the launcher and pre- and post- launch
scripts. So we treat the output of the whole script (including launcher)
as potential indicator of an error.
…s to

CWD (the CI runner runs things in /tmp).
…to avoid

clashes with other things that might be defined in the environment.
- use `StageOutFlags` to express when cleanup happens rather than a boolean
flag
- fixed equality operator in `JobSpec`
- added `__str__`, `__eq__` and `__hash__` methods for staging objects
various interfaces when the loopback will suffice
…lean

old submit scripts to avoid gazillions of small files in ~/.psij/work.
@hategan hategan merged commit 0be2b94 into main Jan 21, 2026
12 checks passed
@hategan hategan deleted the file_staging branch January 21, 2026 21:54
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants