ADF11 repository: rates[element].raw_data is actually 10**raw_data

[Mateasek]

The raw_data is actually 10**adf11_rates. I am bringing this up as a question. I am not a numerical expert, so how big difference can be between:

1. 10**rates
2. interpolate

and

1. interpolate
2. 10**rates

What should be the preferred solution? Should the raw_data store the original log10 form of the data? I'm sorry if I missed somewhere issue discussing this.

[Mateasek]

The conversion happens directly in the parse_adf11 function

openadas/cherab/openadas/parse/adf11.py

Lines 97 to 98 in 0f5cc25

	rates_table = 10**rates_table
	rates_table = Cm3ToM3.to(rates_table)

. To be consistent with what @CnlPepper wrote here #20 (comment), I think the conversion should be moved at least to the install.py functions which will now be doing also the conversion in the charge notation. What do you think?

[Mateasek]

And the same would actually apply to temperature and density values too:

openadas/cherab/openadas/parse/adf11.py

Lines 68 to 69 in 0f5cc25

	densities = 10**densities # convert from log10(cm^-3) to cm^-3
	densities = PerCm3ToPerM3.to(densities) # convert units from cm^-3 to m^-3

and

openadas/cherab/openadas/parse/adf11.py

Lines 70 to 71 in 0f5cc25

	temperatures = tmp[n_densities:] # read temperature values
	temperatures = 10**temperatures # convert from log10(eV) to eV

[mattngc]

@CnlPepper is the numerical expert so I'll let him comment on the reasoning.
But from my understanding the correct ordering is

1. 10**rates
2. interpolate

But you are right @Mateasek, I've probably put these conversions in the wrong place. We should follow the principles set out by @CnlPepper in #20 (comment). The local ADAS cache should mirror the ADAS format, and the conversions should be done when installing into the CHERAB format.

[CnlPepper]

I've been in two minds about how best to solve this. I've had a look through the code again and I've settled on taking a stricter line on the layers. We should strip out all conversions from the ADF file parsers, create some new utility functions to perform conversions in a new install module (the install.py is getting long). And add the conversion routine calls between the parse_adf* and repository.update_* calls.

Something simple like:

def install_adf11plt(element, file_path, download=False, repository_path=None, adas_path=None):

    ...

    # decode file and write out rates
    rate = parse_adf11(element, path)
    rate = convert_adf11(rate)
    repository.update_line_power_rates(rate, repository_path)

We can then keep each step in the work-flow cleaner.

[mattngc]

Ok, but I think the scope of the discussions in this and the related issues (#16, #20, #21) are growing a bit too much. If the standard has already been defined and we are just enforcing it, then that is easy. The issue here is that the data standard hasn't been adequately defined in the documentation. I think we are encountering this now because this is really the first time external collaborators have tried to implement their own rates. There should be an action on @CnlPepper and I to put together a developer guide in the documentation for OpenADAS. Basically a how to document that describes the principles of how the module is suppose to work and what goes where. Many of these standards have been implemented already by @CnlPepper when he refactored the code last year. But we need a documentation page that lays out these principles and how new contributions should be code reviewed.

[Mateasek]

this is also connected #20 (comment)

[Mateasek]

I think this can be closed. I understand now cherab repository is built to unify data from multiple sources and thus has to work on a unified number format.