Provide configuration for haggis

[led02]

Currently, we have some basic command line interface.

We need advanced configuration possibilities, beyond command line.

• Which format to use (TOML?).
• Where to put configuration (hermes.toml?)
• How to access configuration from haggis?

[cdboer]

Many language ecosystems offer a way to store packaging and configuration metadata in a declarative fashion. These files could be a good place to store the configuration for hermes. Some that come to mind:

• pyproject.toml for Python
• Cargo.toml for Rust

Both offer ways to store configuration for third party tools:

• The tool table is explicitly mentioned in PEP 518 as a place to store 3-rd party configuration in a pyproject.toml file (for example: If you look at the pyproject.toml contained within this repository, you'll see the tool table used by your build backend)
• Cargo.toml supports a metadata table that allows 3-rd party tools to store their configuration in it

Of course, a seperate configuration file would still make sense for all cases that either don't have a centralized configuration option such as a pyproject.toml or choose not to use it.

In any way, adding a seperate configuration file to a project adds to the clutter of metadata files already in use by different tools (relevant XKCD). Supporting the "native" configuration approach of a laguage could lower the barrier to entry for the respective community. I'm aware that adding support for all different configuration files available across different ecosystems is a daunting task (if not outright impossible) but wanted to add this to the discussion.

[poikilotherm]

The config must provide options to

• alter weight of source
• alter source order

Instead of creating complex mapping classes and wrapping data in files falling out of harvesting, maybe encoding weights and order in the filename helps?

[led02]

#40 introduces some basic configration using pyproject.toml (mainly for logging configuration). However, this can be easily extended.

[poikilotherm]

I meant to write this earlier, but FWIW...

Should we think about making weights/priority of a source configurable down to the field level? There might be occasions where the same source might be more trustworthy on one thing and at the same time unreliable for another.

Maybe it would be good to do this by giving a reference to a target field as JSON-LD identifier and it's priority, defaulting to the sources overall priority?

[led02]

Should we think about making weights/priority of a source configurable down to the field level? There might be occasions where the same source might be more trustworthy on one thing and at the same time unreliable for another.

Yes, this should be possible.

Maybe it would be good to do this by giving a reference to a target field as JSON-LD identifier and it's priority, defaulting to the sources overall priority?

The current architecture / data model is heavily based on ContextPath objects (which are basically like JSON Paths but a little less powerful). They can be used just fine to select fields by the JSON-LD identifier or the path within a mapping-based dataset. In addition, the merge stragtegies for (sub-)datasets is rather flexible and should be configurable. In conjunction, this allows exactly what you are aiming for I guess.

[sdruskat]

During a recent discussion with @led02, I think we ended up favouring a central config file (e.g., hermes.toml).

I agree with @cdboer's comment that this will add clutter, but it also makes it very clear, where the workflow is configured. I think I prefer clarity over bootstrapping, and the latter would add an anormous amount of work to implement HERMES config for the different formats (and we will always miss those that don't work with such configs, or where configs don't allow for 3rd party config).

[sdruskat]

Having said that, @led02 and me also discussed that it's generally a good idea to let people do what they want.

In the case of email address mapping, for example, people may have a .mailmap for their git config anyway, and so we could support writing such configs as well (also to pyproject.toml, Cargo.toml, pom.xml, etc.) in post-processing. (This is a sideline though which should be implemented as an afterthought almost, perhaps as Future work even, could be part of the "increase usability" argument).

There's a meta weight attached to this as well, where people may want to config that they prefer native over hermes.toml or vice versa.

[sdruskat]

Closing as tracked in #46, which this overlaps.