Refactoring of data model [PARENT ISSUE]

[led02]

Note (@sdruskat): For an updated plan, see #153 (comment).

I'm planning some ... kind of bigger refactoring to the data model.

The goal is, that the high-level API keeps more or less the same.
I.e., code that uses some of the .*Context models should in most cases continue to work unmodified.
The only exception is the renaming (see below).

My questions to my fellow developers:

• When to do this?
• How much of this should be integrated into Provide processed version number #67?

[jkelling]

• Regarding when: definitely before or after Provide processed version number #67 ,these this cannot happen at the same time.
• Regarding Provide processed version number #67 : maybe we should define what the API to access the different versions should look like. Then Provide processed version number #67 could add it to the current *Context ideally in such a way, that the refactoring will not touch it much, apart from the renaming. Do you think this is possible?

[zyzzyxdonta]

It will collide with #151 as well. But no changes on the relevant parts are pushed yet, so I can easily rebase onto your changes.

[led02]

• Regarding when: definitely before or after Provide processed version number #67

Agreed. Especially as it is also somewhat relevant for #151 this needs to happen BEFORE #67 (as it is required there) and might be done AFTER #151 (if @zyzzyxdonta does not rely on this feature).

However, there is already a 'low-key' change in #154 that adds the desired functionality. Maybe we can split this up:

• Use the alreday implemented new behaviour which does not break anything yet.
• Implement against it, as the API should barely change.
• Do the refactoring after Provide processed version number #67, Use the correct software license when depositing files to Invenio instance #151

• Regarding Provide processed version number #67 : maybe we should define what the API to access the different versions should look like. Then Provide processed version number #67 could add it to the current *Context ideally in such a way, that the refactoring will not touch it much, apart from the renaming. Do you think this is possible?

I think this is actually how the change was intended to happen. :)

[zyzzyxdonta]

No, I don't rely on it. Alright, then I'll do #151 first. 😄

[jkelling]

I think, the discussion at #154 has shown, that this issue should be resolved before #67, because the refactoring of the MergeStrategys should automatically address #67 in my mind.

[led02]

Okay, I've been thinking... (Yes, please don't laugh at me!)

The current split of data and tags is not very practical and leads to confusion (e.g., it makes the generation of a nice and tidy Markdown curation file pretty cumbersome).

I've also realized that there is a clear difference between linked data (assigning unique identifiers and maybe some semantics to terms) and a JSON schema (defining how terms need to be arranged to be valid, probably adding constraints but not adding semantics). For merging different datasets, the semantics are important, not the schema. Hence, we need to operate on "well-defined" terms (e.g., checking for @type attributes).

We are also already on the road to extending the semantics we borrow from CodeMeta for our internal data model by own, Hermes-specific terms. This would allow us to simply extend the context we are using by a hermes:tags and a hermes:alt term that then could carry the Meta-Metadata and alternatives we have found and could not merge into a common model.

The second thought which enlightened me like the streak of a thunderbolt was, that it might be a far better idea to move the merger-logic away from the actual data model into an own Merger class. I.e., instead of using the current approach of

for harvest_ctx in all_harvest_ctxs:
    ctx.merge_from(harvest_ctx)

we would do

ctx, *tail = all_harvest_ctxs
for other_ctx in tail:
    ctx = Merger.merge(ctx, other_ctx)

or even

ctx = Merger.merge(*all_harvest_ctxs)

Let's discuss this in one of the next meetings, before I continue to implement weird stuff again that nobody (including me) understands later.

[led02]

Nobody really objected so far (even in the HERMES Jour Fixe), so I'll go ahead and at least start to implement the tags / alternative changes (which will result in an extended Hermes LD context).

Refactoring of the "merger" will be done based on these changes. This will most probably break the workflow for some time (in favour of avoiding another monster PR by me 😅 )

[led02]

Well, in fact if we use the IRI as key into the dictionary, we only need some fancy way to map vocabulary to IRIs. On top of this: JSON-LD does provide us with standardized terms bound to a context (i.e., that's what one of the primary functions are). So to sum up: We probably won't even need to have a special dict type for our data...

Hence, I'll start with the merger (working on dictionaries) to get rid of lots of ugly stuff.

Meanwhile, I'm wondering how important it is for us to actually access data["foo"]["bar"]["baz"] as data["foo.bar.baz"] (or to put it more clearly: will we continue to use the ContextPath class...)?

[zyzzyxdonta]

data["foo.bar.baz"] alone doesn't seem all that important to me. But JSONPath/JmesPath functionality might become interesting once we try to implement some merge strategies (although merge strategies will probably quite simple as long as we don't have a huge amount of harvesters). So I guess the fancy stuff can wait until hermes 2.0 🤷🏻‍♂️

[led02]

@SKernchen made me stumble upon https://pypi.org/project/pydantic-schemaorg/

Now, I'm again ... considering 😅 But with good progress, doing source code experiments and trying not to diverge too much :D

[led02]

For now, I'll continue to use the plain dict but use the schema-org pydantic classes for creation of the CodeMeta dataset...