Should validphys tests use new API

[wilsonmr]

recently I have been changing some of the underlying functionality of certain functions, see for example #427

Each time all of the tests have to be changed because they're using providers explicitly, until this is done the build breaks, see #447

One of the advantages with the API would be that we can rewrite these tests to use runcard level inputs, so that the backend can easily be completely changed, provided the end result is the same.

I'm not sure exactly which tests would want to rely solely on the API, because I guess if each test made a new call to the API the intermediate results wouldn't be cached in the same way that they currently are with the tests, which would lead to a slow down.

[wilsonmr]

Well the loading of resources is cached

>>> from validphys.app import API
>>> 
>>> a = API.plot_pdfs(pdfs=["NNPDF31_nlo_as_0118"], Q=2)
PDF: NNPDF31_nlo_as_0118  ErrorType: Monte Carlo booked
LHAPDF 6.2.1 loading all 101 PDFs in set NNPDF31_nlo_as_0118
NNPDF31_nlo_as_0118, version 1; 101 PDF members
NNPDF31_nlo_as_0118 Initialised with 100 members and errorType replicas
>>> b = API.plot_pdfs(pdfs=["NNPDF31_nlo_as_0118"], Q=2)

[Zaharid]

Tests are typically divided among others in "unit tests" and "integration tests". The former are to validate that a specific piece of logic actually works (does the sort function actually sort or does sort([3,1,2]) return something like [1,2]) while integration test check whole program level things work from the user perspective.

In general in validphys we don't do a large amount of "low level logic" (e.g. we don't implement things like sorting or computing the median), although we do do some of that (e.g. the Stats classes or some functions in plotutils, which should have better tests than they do). We are thus rather more interested in the integration tests, for which the API is a good match, for the reasons you mentioned.

Regarding caching, I don't want to rely on caching of loaded objects too much because that's really there only because of the combined limitations of reportengine and the wrappers. The wrappers don't know how to "pickle" objects and send them over the wire for say the multiprocessing usage and reportengine is at the moment completely stupid regarding deallocating memory (it doesn't do it at all). But those are all things to be improved. Maybe we should have a caching version of the API. It could work like:

with api.context as ctx:
    ctx.provider(---)

[wilsonmr]

Ok so with regards to the integration tests, do you want to add some kind of context for the api first, or could I have a stab at updating the relevant tests using the api?

[Zaharid]

I would say we can do the tests: With a judicious use of fixtures and the like, it is unlike that they will get much worse.