Build CIFF(s) for MS MARCO for tutorial

[elshize]

• Select collection(s) -- probably one with quantized BM25, one LSR
• Build CIFF
• Upload somewhere, preferably long-term, but at least until the tutorial
• Post links
• Link query sets that should be used

[JMMackenzie]

I'm working on this now - Should I create a CIFF, or a PISA canonical? It might be easier to go right from canonical?

[elshize]

That's a good question. It might actually be easier to have CIFF and create all of the metadata files with the new script. Otherwise we'll need to either use low-level commands (but then we'll have to explain them) or create the metadata ourselves.

When you say PISA canonical, would that be done via CIFF or bypassing CIFF?

[JMMackenzie]

So basically we will do:

• Download CIFF
• CIFF2PISA (? right?) to get the canonical format files
• Index the canonical files

I am suggesting instead we:

• Download tarball containing the canonical files directly
• Index the canonical files

The only benefit is that we need not install/run the ciff2pisa tool, but perhaps it's good to have that step anyway? Not too sure what the pros/cons are otherwise.

EDIT: So to answer your question - done via CIFF, not in lieu of it.

[elshize]

The docker image I'm creating for the tutorial has ciff2pisa, that's not a problem. I've tested running it, which looks something like pisa index ciff file.ciff ..., which then does all the steps up to compression, wand data, etc.

I would say let's create a CIFF for this ticket, the rest can be done with a single command now, so if we decide we want to avoid that ciff2pisa step, then we can easily do it then. Then we can pre-package that. But the most important is to build CIFF.

[JMMackenzie]

Neat, that's great. In that case, I'll send a link soon to a ciff. I'm hosting it on UQ's cloud service, I just need to wait for it to propagate. We might want a permanent solution later once we confirm it's all working. The zip is only ~350M anyway.

[JMMackenzie]

URL: https://cloud.rdm.uq.edu.au/index.php/s/kdq5Lts5RafBzqF
Password: sigirpadua2025

I need to figure out how we can directly download this via wget or etc - I had a quick tinker but couldn't get it to work.

[JMMackenzie]

Here is the LSR (100 query sample): https://www.dropbox.com/scl/fi/fkqso0vv8mwq9yaajf4pj/lsr-small.zip?rlkey=ff212z3i0d0y0n45kd5wiekzc&st=1i9clk1h&dl=0

[JMMackenzie]

Leaving this issue open - although these are now complete, we need a more permanent home for them.