Is there ever a reason to "clone a bare mets.xml"

[kba]

Currently ocrd workspace clone will by default only download/copy the mets.xml and none of the files linked in it.

The reasoning behind this was to reduce bandwidth requirements to allow users to download only the files they needed.

However, with the strong focus on relative file paths and general FS-based operations, I propose to invert the logic: By default download/copy all the files and turn that behavior off with a flag.

The main problem I have with the current approach is that it will break resolving relative files. While the process is running, the src_baseurl of the workspace is still known, but it is gone after process ends. Hence you end up with a mets.xml file with relative files and no idea where it came from or how to resolve the files.

[bertsky]

The reasoning behind this was to reduce bandwidth requirements to allow users to download only the files they needed.

I find this reasoning is still valid, regardless of how workspaces are implemented.

While the process is running, the src_baseurl of the workspace is still known, but it is gone after process ends. Hence you end up with a mets.xml file with relative files and no idea where it came from or how to resolve the files.

Please elaborate: why would you want to convert URLs to relative unless the user chose download=True, and why would a single src_baseurl help in any way with the individual file URLs (that can all be from different sources, completely independent of the METS)?

[bertsky]

Please elaborate: why would you want to convert URLs to relative unless the user chose download=True, and why would a single src_baseurl help in any way with the individual file URLs (that can all be from different sources, completely independent of the METS)?

Oh, you were of course referring to relative paths in FLocat, not absolute paths or remote URLs. I think I do understand now. But IMO the above use case (deferred download) only exists for remote URLs, not for filesystem paths!

Moreover, seeing #319 happen, it just became clear to me: If you want to keep the deferred download logic in place even for FS paths, you must put the absolute path of the source METS in baseurl when cloning (or more precisely, when doing workspace_from_url), not the relative path. That way, you can still correctly convert to relative when download=True is requested.

[kba]

you must put the absolute path of the source METS in baseurl when cloning (or more precisely, when doing workspace_from_url),

That is what I meant with src_baseurl, it's the URL to prepend to any relative mets:file/mets:FLocat/@xlink:href to resolve them (could be an http URL or an absolute or relative file path). But after cloning the workspace, it is gone. We could but should not save it in the METS file (though it might be helpful to have in METS for http URL, it's really not for file paths). We could use some hack to save it to the workspace, like Eclipse and other IDE (annoyingly) do.

Since this has the Final Workshop issue, we need a solution ASAP. Any ideas for a good mechanism for this?

[wrznr]

For me: No change required.

1. It is not useful to download all file groups per default. Usually, we only need one.
2. When I run a processor and specify an input file group the files of which are not yet physically present, they are downloaded automatically. This is very convenient.
3. If I have to have a specific file group physically present (e.g. for using workflow-configuration), I can always resort to ocrd workspace find --download.

[kba]

OK, I retract my proposal to always download and agree there is no need for change. Probably best to recommend users to use ocrd workspace find --download for their bulk downloading needs since it's much more flexible.

If data providers actually have relative FLocat their published METS data it is likely a mistake. Beyond OCR-D I haven't seen relative paths in METS, so I think it's better to focus on more pressing issues like filegroup removal and bulk adding files rather than accomodating hypothetical scenarios.

[cneud]

As a last resort, the source mets.xml can be put in a separate fileGrp too.

[kba]

As a last resort, the source mets.xml can be put in a separate fileGrp too.

That's an excellent idea. No idea why this did not cross my mind... ❓ Should we ever have the need to have a mechanism like that in the future that's how it should be done.