Make the sqlite theorydb into a folder of yaml files

[scarlehoff]

This is my proposal for what we discussed in Amsterdam.

At the moment (for validphys) this doesn't change anything. All fields are mandatory as they were before. I think we should first change it to the layout we want (yaml file or other) and afterwards decide what fields are important and what are no.

All interactions with the theory database for validphys happened through the validphys2/src/validphys/theorydbutils.py module. All other changes to validphys are such that the database points to theory_cards instead of theory.db.

I've also modified the docs (I've searched for theory.db and changed stuff whenever they were mentioned).

I have to admit I am not fully convinced RE having a file per theory. Having them in a single file would allow us to do this:

theory_41: &theory_41
  Comments: theory 41
  <other parameters>

theory_42:
  <<: *theory_41
  Comments: Theory inheriting from 41

40 000 000.yml --- baseline
40 000 001.yml

40 001 000.yml --- baseline
40 001 001.yml

40 001 000: 40 000 000.yml

which might be very nice.

Another option is that we leave the ones we currently have as "one per file" (but with the syntax id: <data>) and going forward organize the theories as 41_012.yaml and inside all the theories 41_012_000: <data>, 41_012_000: <data> etc.

(edit: the regression test failed, maybe by casting some of the quantities as int or float changed slightly the result, but I'll deal with it at the end)

[felixhekhorn]

I have to admit I am not fully convinced RE having a file per theory. Having them in a single file would allow us to do this:

• Implementing (simple) inheritance should be easy enough for us to use
• having a huge theory file (~600*30 lines) is not much better then a sql database; it wouldn't be much readable either and we would couple (more or less) again several changes to the db together

so I would still prefer one unique theory per file

[scarlehoff]

Implementing (simple) inheritance should be easy enough for us to use

Actually, inheritance might also work across files with the default yaml loading.

I guess you are also against the mixture?

Another option is that we leave the ones we currently have as "one per file" (but with the syntax id: ) and going forward organize the theories as 41_012.yaml and inside all the theories 41_012_000: , 41_012_000: etc.

[giacomomagni]

LGTM.

• Can you fix also:

nnpdf/pyproject.toml

Lines 30 to 34 in 41471d8

	# The default profile is included together with the validphys package
	"validphys2/src/validphys/nnprofile_default.yaml",
	# Same for commondata and theory.db
	"validphys2/src/validphys/datafiles/commondata/*",
	"validphys2/src/validphys/datafiles/theory.db",

*And:

https://github.com/NNPDF/nnpdf/blob/theory_to_yaml_files/doc/sphinx/source/tutorials/reproduce.rst

[RoyStegeman]

https://github.com/NNPDF/nnpdf/blob/theory_to_yaml_files/doc/sphinx/source/tutorials/reproduce.rst

Why would this need to be fixed? Note that both the dockerfile and yaml environment contain a specific (now outdated) version of the nnpdf code

[giacomomagni]

Why would this need to be fixed? Note that both the dockerfile and yaml environment contain a specific (now outdated) version of the nnpdf code

Ah okay, then no need to fix it.

[RoyStegeman]

If we wish to search the files using grep as you suggested it would also help to have a single file per theory such. Also in that case we'll want to be very strict about formatting e.g. ktThr: 1.0 should always be that and not ktThr: 1.0. Perhaps a formatting check in the tests can help with that?

[andreab1997]

I would start removing parameters that we already know are useless. For example this MaxNfPdf but also MaxNfAs and EScaleVar (and most likely others...)

[scarlehoff]

I will do that in subsequent PRs. The idea is that this leaves validphys essentially unchanged.

[scarlehoff]

should always be that and not ktThr: 1.0. Perhaps a formatting check in the tests can help with that?

This is important, yes. Maybe it is time to add a pre-commit hook to this repo...

[RoyStegeman]

Maybe it is time to add a pre-commit hook to this repo...

Something like that, but at least initially I'd be happy to have it check only this theory database and not the entire repository

[felixhekhorn]

I guess you are also against the mixture?

Another option is that we leave the ones we currently have as "one per file" (but with the syntax id: ) and going forward organize the theories as 41_012.yaml and inside all the theories 41_012_000: , 41_012_000: etc.

I was a bit hesitant, but then this

If we wish to search the files using grep as you suggested it would also help to have a single file per theory such. Also in that case we'll want to be very strict about formatting e.g. ktThr: 1.0 should always be that and not ktThr: 1.0. Perhaps a formatting check in the tests can help with that?

is a good argument against and have really one file per theory

[andreab1997]

As promised, I had a second look to the code part and it seems fine to me