Change client's index reading behavior (breaking)

[consideRatio]

With the local env structure now diverging from the index structure, we need to change how the client reads against an index with a clear breaking change. This issue is extracted from point 6 at #210.

Guidelines from Tilo

Look at core/src/resolve/standard.rs and its definition of RemoteIndexResolver, and replace its reliance on HTTPEnvironmentAsync (implemented in core/src/resolve/reqwest_http.rs), with something new that looks mostly like it, but with some changes on how it implements the ReadEnvironment trait.

Current index structure

The current index structure is like the old env structure you can inspect after having used sysand env to install things, or by looking at https://beta.sysand.org/entries.txt or https://beta.sysand.org/a0aacee34dd4cd5e2d07ab43d5e30772ec86dbf3c8fafb033bad338dd7a0f02e/versions.txt for example.

The .kpar references can be files or folders, and both are supported.

(old index root folder)
 ├──entries.txt
 ├──iri_hash_1
 │  ├──versions.txt
 │  ├──1.2.2.kpar
 │  └──1.2.3.kpar
 └──iri_hash_2
    ├──versions.txt
    ├──2.3.3.kpar
    └──2.3.4.kpar

Root entries.txt looks like this

iri_1
iri_2

Project scoped versions.txt looks like this

1.2.2
1.2.3

We are currently working towards a structure like below, adjusted following discussions with Andrius from an internal GitLab MR:

(index root folder)
 ├──index.json
 ├──_iri
 │  ├──iri_hash_1
 │  │  ├──versions.json
 │  │  ├──1.1.0.
 │  │  |  ├──project.kpar
 │  │  |  ├──.project.json
 │  │  |  └──.meta.json
 │  │  └──1.0.0.
 │  │     ├──project.kpar
 │  │     ├──.project.json
 │  │     └──.meta.json
 │  └──iri_hash_2 (contains same structure as iri_hash_1)
 ├──publisher1
 │  ├──name1 (contains same structure as iri_hash_1)
 │  └──name2 (contains same structure as iri_hash_1)
 └──publisher2
    └──name1 (contains same structure as iri_hash_1)

Root index.json has the following structure:

{
  "projects": [
    {"iri": "pkg:sysand/abc/def"},
    {"iri": "https://example.org/project.kpar"}
  ]
}

Each versions.json has the structure examplified below. version and usage fields are direct equivalents of whats found in .project.json. The

{
  "versions": [
    {
      "version": "2.3.4",
      "usage": [
        { "resource": "pkg:sysand/abc/dep", "versionConstraint": "<2" }
      ],
      "project_digest": "sha256:<64-hex>",
      "kpar_size": 12345,
      "kpar_digest": "sha256:<64-hex>"
    },
    {
      "version": "0.0.1",
      "usage": [],
      "project_digest": "sha256:<64-hex>",
      "kpar_size": 6789,
      "kpar_digest": "sha256:<64-hex>"
    }
  ]
}

The entries in the versions list should be semver sorted based on the version, and that means that 10.0.0 is larger than 10.0.0-beta.1 as well as 2.0.0, which isn't what you get if you do a direct string comparison, so for this sorting we should rely on a semver library provided comparison.

All five per-entry fields are required; the client rejects a versions.json
that omits any of them.

• project_digest — sha256 over the canonicalized info + meta pair of
  the kpar's .project.json / .meta.json. The client writes this into the
  lockfile at lock time without fetching the archive.
• kpar_size — byte length of the kpar archive, recorded on the lockfile
  source entry so the client never has to issue a HEAD.
• kpar_digest — sha256 over the kpar archive bytes. The client verifies
  the streamed body against this value at download time and rejects a
  mismatched archive with a DigestMismatch error.

The index add command should reject non-semver versions or dependencies with version constraints that aren't parsable by semver::VersionReq.

---

This is already implemented in our index in development, under the /index path. There however we only accept nice packages with constraints on the name field, publisher field, and version field.