Dataset query

[jl-wynen]

We need a way to query for datasets so that users don't have to switch between Scitacean and Pyscicat or a frontend. There are some open questions that need to be answered first:

Which query language?

• Use the loopback-specific query language. It uses a suboptimal language and may get replaced in the future. But it is available now.
• Use a MongoDB-based language. This would give users the most power and a better syntax. But it is not available yet and it is unclear when, and if, it will be implemented.

If we start with the loopback language, we may have to keep supporting both languages for a long time while phasing out the loopback language.

What objects to return?

With the high-level Client, we should probably return list[Dataset]. But the construction of Dataset is rather costly (even with embedded datablocks, see below). So this would only be useful for queries that yield few results. We can add an upper bound to the query function along the lines of

def query_datasets(self, query, *, max_results=100) -> list[Dataset]:
    results = self.do_query(query)
    if len(results) > max_results:
        raise RuntimeError(...)
    return list(map(make_dataset, results))

The low-level ScicatClient could return list[DownloadDataset]. But for minimal overhead, it could also return the raw JSON. But that may not be the right interface for Scitacean.

Datablocks

In order to get the full metadata, we would have to get the PIDs from the returned DownloadDatasets and make a separate request to download datablocks for each. It would be better if we could use an endpoint for the query that returns both the dataset and datablock. Then we can assemble the Dataset objects directly.

Incidentally, this could also improve performance of get_dataset.

[jl-wynen]

@nitrosx Can you check if I missed anything and add it?

[nitrosx]

It looks good.
I would suggest to start with the loopback syntax and name the function query_lb, so we can have an MVP soon and start testing.
We can later move to query_mongo, or simply query, which uses the mongodb syntax. This would require a new endpoint, that it is not yet implemented, and requires approval from the community .