Configurability using config file by Victor-Schwan · Pull Request #3 · key4hep/TrackerPerformanceScripts

Victor-Schwan · 2024-05-31T09:14:10Z

BEGINRELEASENOTES

Paths to personal and detector-specific files/folders outsourced into the config file
Choose config file with argparse
Path operations using pathlib
so far, only tested with condorJobs_sim.py

ENDRELEASENOTES

…xecution at the end

…nt moved here

…be perf_counter_ns() is better without wait

introduced functions; cld_config added for cld specific variables

Zehvogel

Thx!

Some comments below. I did not look much at the other files yet.

…. condorJobs_reco

clone in `bash_script` in `condorJobs_sim` removed

Zehvogel · 2024-06-05T07:34:51Z

@Victor-Schwan is this ready now?

Victor-Schwan · 2024-06-05T08:08:01Z

Not ready yet. When I ran it on lxbatch there was a permission problem in sim_jobs which needs to be debugged. And I have an almost completed commit waiting removing double definitions of vars and some other refacs

Zehvogel · 2024-06-05T08:12:27Z

Ok

now only definied in config file and not anymore again in scripts; integer variables defined as `ints` and not `strings`: unnecessary type casting removed

…unding when dividing Nevts to jobs)

Zehvogel

Some more comments.

I guess we will need a future PR on simJobs_reco to make it a bit more flexible but for now I am happy.

I will merge after @gaswk approves. :)

…ing compatible with general changes

Zehvogel · 2024-06-19T09:17:40Z

@gaswk is everything fine from your side now?

nitasad · 2024-06-19T09:25:24Z

@gaswk is everything fine from your side now?

I am waiting for your response to my previous comment regarding the naming conventions for the output files. To maintain consistency, should we use _ instead of . for all output names? Currently, the outputs from the reco are named _edm4hep.root and _aida.root by default.

Zehvogel · 2024-06-19T09:33:44Z

I am waiting for your response to my previous comment regarding the naming conventions for the output files. To maintain consistency, should we use _ instead of . for all output names? Currently, the outputs from the reco are named _edm4hep.root and _aida.root by default.

Oops, sorry for missing that. I would prefer . instead of _ but if CLDConfig still produces _ we would need to change it there first of course :/ I think a have a change for that ready somewhere but that also changes some more parts.

@Victor-Schwan opposite to previous instructions I would then suggest that you change back to _ for this PR and we make the change to . later...

…in condorJobs_sim and _reco adapted

Victor-Schwan · 2024-06-19T12:12:59Z

I added a bool var in the config file which switches between _edm4hep.root and .edm4hep.root paths. What do you think? @Zehvogel @gaswk

Zehvogel

I think you meant this?

nitasad · 2024-06-19T13:43:24Z

Everything is fine for me now!

Victor-Schwan added 11 commits May 16, 2024 19:32

variables outsourced to config.py; condorJobs_sim runs until actual e…

26ad226

…xecution at the end

switch from str paths to Path objects

9880448

minor fix

06033d0

reco: switched to pathlib, more config vars added; conffig: environme…

ecb75d1

…nt moved here

time stuff removed; wait slowed down the execution significantly, may…

389170d

…be perf_counter_ns() is better without wait

job number calc in verbose mode fixed

fee1771

verbose mode for debugging of slow loop removed

ddc294b

minor refactoring Condor/condorJobs_sim.py

5d8fa22

minor

c293833

added afs paths to config

51dbf54

mostly chhanges to paths

7548679

Zehvogel self-assigned this May 31, 2024

Victor-Schwan added 2 commits May 31, 2024 13:40

condorJobs_sim: argparse to choose config file,

e41ee39

introduced functions; cld_config added for cld specific variables

condorJobs_sim: improved argparsing: .py suffix can be omitted

18bb9cd

Zehvogel reviewed May 31, 2024

View reviewed changes

Comment thread TrackingPerformance/Condor/condorJobs_sim.py Outdated

Zehvogel reviewed May 31, 2024

View reviewed changes

Victor-Schwan added 2 commits May 31, 2024 14:54

parse_arg and load_config outsourced to utils; easily reusable in e.g…

46bfe63

…. condorJobs_reco

JOB_FLAVOR outsourced to config

acebf1c

tmadlener reviewed Jun 3, 2024

View reviewed changes

Comment thread TrackingPerformance/Condor/condorJobs_reco.py Outdated

This was referenced Jun 3, 2024

Bug: Check and rerun event simulation if jobs crashed #4

Open

Cloning in the batch jobs? #5

Open

Victor-Schwan added 2 commits June 3, 2024 15:44

config gun args actually incorporated in ddsim command;

0600e0d

clone in `bash_script` in `condorJobs_sim` removed

reco: adde cfile from

7975c8b

Victor-Schwan mentioned this pull request Jun 3, 2024

Job submission where? #6

Open

Victor-Schwan added 2 commits June 3, 2024 16:57

renamed config template

7c91e95

minor addendum config template

6a8a238

double definitions of vars removed:

915b9b6

now only definied in config file and not anymore again in scripts; integer variables defined as `ints` and not `strings`: unnecessary type casting removed

Victor-Schwan added 5 commits June 5, 2024 17:17

number of events specified in config is ceiled instead of floored (ro…

dab3fa7

…unding when dividing Nevts to jobs)

Alignment of names in and

2d9db28

refac, alignment of naming and structure in sim and reco

7ece7d1

fix path for xrootd

138a7b5

minor fix; refacs: mainly renaming vars

298dcfa

Victor-Schwan marked this pull request as ready for review June 11, 2024 12:40

Zehvogel reviewed Jun 11, 2024

View reviewed changes

Comment thread TrackingPerformance/Condor/condorJobs_reco.py

Comment thread TrackingPerformance/Condor/condorJobs_reco.py Outdated

Comment thread TrackingPerformance/Condor/condorJobs_sim.py Outdated

Victor-Schwan mentioned this pull request Jun 11, 2024

Move ResVDX_UV_ from condorJobs_reco to config file? #8

Open

sim: output file rm on afs; minor fixes; make Plotting/analysis_track…

5e3cef0

…ing compatible with general changes

nitasad reviewed Jun 11, 2024

View reviewed changes

Comment thread TrackingPerformance/Condor/condorJobs_reco.py Outdated

nitasad reviewed Jun 11, 2024

View reviewed changes

Comment thread TrackingPerformance/Condor/condorJobs_reco.py

nitasad reviewed Jun 11, 2024

View reviewed changes

Comment thread TrackingPerformance/Condor/condorJobs_reco.py Outdated

Victor-Schwan added 4 commits June 11, 2024 17:46

changes to reco requested by Gaelle

8c2a8e4

ResVDX_UV_ removed from reco as requested by Gaelle

bcd52b3

Revert Plotting/analysis_tracking.py to state before Victor's PR

b5a93fd

reco: change aida file suffix to .aida.root

6fa0479

Victor-Schwan added 2 commits June 19, 2024 13:53

minor: typehint main functions

237bd06

config option EDM4HEP_SUFFIX_WITH_UNDERSCORE added and path building …

4163c0b

…in condorJobs_sim and _reco adapted

Zehvogel requested changes Jun 19, 2024

View reviewed changes

Comment thread TrackingPerformance/config_template.py Outdated

Comment thread TrackingPerformance/config_v.py Outdated

minor fixup

bdd3deb

Zehvogel approved these changes Jun 19, 2024

View reviewed changes

Zehvogel merged commit fd4d078 into key4hep:main Jun 19, 2024

Conversation

Victor-Schwan commented May 31, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Zehvogel left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Zehvogel commented Jun 5, 2024

Uh oh!

Victor-Schwan commented Jun 5, 2024

Uh oh!

Zehvogel commented Jun 5, 2024

Uh oh!

Zehvogel left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Zehvogel commented Jun 19, 2024

Uh oh!

nitasad commented Jun 19, 2024

Uh oh!

Zehvogel commented Jun 19, 2024

Uh oh!

Victor-Schwan commented Jun 19, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Zehvogel left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

nitasad commented Jun 19, 2024

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Victor-Schwan commented May 31, 2024 •

edited

Loading

Victor-Schwan commented Jun 19, 2024 •

edited

Loading