Integrate ASPEN CLI

[Geet-George]

Here you can download ASPEN: https://www.eol.ucar.edu/software/aspen
Here are the relevant docs: https://ncar.github.io/aspendocs/man_aspenqc.html

[Geet-George]

@d70-t could you give me a summary of the issue we are facing here, so that I can know exactly what to ask from NCAR?

[d70-t]

I managed to get an instance of Aspen-QC running using docker. Here are the steps to reproduce:

Create a docker image using this Dockerfile

FROM --platform=linux/amd64 centos:7
RUN yum install -y epel-release  # extra packages (netcdf etc...)
RUN yum install -y boost-regex boost-filesystem xerces-c boost-program-options netcdf netcdf-cxx  # dependencies
RUN mkdir /src
WORKDIR /src
RUN curl "https://archive.eol.ucar.edu/docs/software/aspen/AspenV3.4.9_CentOS7.9.2009_x86_64.tar.gz" | tar -xzf -
WORKDIR /src/AspenV3.4.9_CentOS7.9.2009_x86_64/bin/
ENTRYPOINT ["./Aspen-QC"]
CMD ["-i", "-e"]

Build the docker image using:

docker build -t aspenqc .

pipe some level 0 data through aspenqc, e.g. this way:

curl "https://observations.ipsl.fr/aeris/eurec4a-data/PRODUCTS/MERGED-MEASUREMENTS/JOANNE/v1.0.0/Level%200/HALO/20200128/D20200128_225221.7" | docker run --rm -i aspenqc

EDIT: explicitly specify platform as linux/amd64. (The aspen software is only available on x86_64 for linux distributions, so even if we a running on different host hardware architectures like arm, we have to have an x86 based operating system).

[d70-t]

Using the method above, it should be relatively straightforward to run aspen (via docker) on any machine available. We might eventually even put the image on e.g. dockerhub or another registry, such that installing and running aspen will be a one-liner (approximately the last line in the post above).

What's missing: In order to integrate Level0 -> Level1 Conversion into this package, we have to agree which data format we'd consider as Level1 data. My current understanding is "whatever comes out of Aspen" is considered Level1. But Aspen can create a variety of formats (eol, frd, csv, class, wmo, netcdf, skewt svg, BUFR), however:

Note that netcdf and BUFR files cannot be written to stdout

Being able to output through stdout has the advantage that we wouldn't have to touch a filesystem when running the processing, but we can't do netCDF. However people like netCDF and if really needed, we could also go for this options.

[tmieslinger]

I tested the above and it works well when specifying the platform.
In addition, we might want to output netcdf as the current setup of the following processing builds on Level_1 data in netcdf format, right? To write netcdcf output one has to connect the docker container to the local system through a bind mount, e.g. the example file from above could be processed as:

docker run --rm --mount type=bind,source=local_path_to_data_directory,target=/data aspenqc -i /data/20200128/Level_0/D20200128_225221.7 -n /data/20200128/Level_1/D20200128_225221.7.nc

source= directory which the docker image can see and has access to
target= name of the source dir within the docker image.

One open question is how such a docker image shall eb made available, either

1. within the halodrops repo
2. or in a separate repo

Either way there should be a CI connected that builds the image and pushes it to the GitHub container registry.

[Geet-George]

Yes, from just a glimpse, bind mount seems like it will do the trick.

I understand that the files at source can be read from target. But just to better understand from my naive PoV, will the output written to the target (i.e. arg value after -n) be "synced" with source? i.e. with the command you wrote above, will the files be available at "source"? Or is there a different command to write to source?

As for how to make the docker image available, I think if we can have everything in one, that would be great. Less scrambling and worries for users... But if there is an argument for a separate repo, which might make things better in some other way, I'm happy to learn.

I agree with the CI definitely (we also need more for other parts of the package :P), but if possible would you or someone else be able to help out with this issue here? I won't be the fastest or the best at doing this.

[tmieslinger]

TL;DR: two repos might be less messy in terms of the CI, but might be good to be coordinated within a "halodrops" organisation

Yes, if set up as stated above, the output netcdf file will be stored on the local machine at local_path_to_data_directory/20200128/Level_1/D20200128_225221.7.nc, but we are free to choose the location. THis was a first try. We probably want to stick to the data directory structure as stated here.

After talking to @d70-t it seems that a separate repo for the ASPEN processing might be of advantage. Imagining that we have a docker image for one specific ASPEN version, we might connect it with one release of the respective repo and a release could trigger pushing the image to the registry. For another/newer APSEN version, we'd simply do the respective changes, test it and create a new release which generates (via the CI) a new registry entry. Within halodrops we could call the ASPEN docker image with the software version that we want to have, so we are flexible on the versions at any time.

Alternatively, having the ASPEN processing within the halodrops repo, we would have multiple CIs (which is possible), but it might be a bit complicated to specify the exact triggering of the individual CI such that building the ASPEN docker image and pushing it to the registry only happens if one of the files related to that software part is touched.

If we decide on having two repos though, it might become convenient to put them both into a "halodrops" organisation. @Geet-George have you ever thought of opening an organisation for the halodrops software?

[Geet-George]

I guess that makes sense. Allowing two repos to do the job could also make it easier to have the same halodrops processing run for different ASPEN versions.

That said I'm now in two minds about this, because if it is not super user-friendly (i.e. working with two packages), it might end up not being used as much. People might fall back to the same way of batch processing using their own ASPEN versions locally and then stick only with L1 files, e.g. how the files are used by HAMP or WALES.

Unless I completely misunderstood you, and you were thinking of having the separate aspen repo as a dependency for the halodrops package... In which case, I think we should definitely do that.

P. S. Never thought halodrops would need to scale up to an organization, but we can of course do that.

[tmieslinger]

We started a new repo for the ASPEN processing based on this issue and the Docker image described above. Currently, there is ASPEN v4.0.2 available via the image ghcr.io/atmdrops/aspenqc

Feel free to test it as described in the README and provide feedback in the respective pull request.