manifest json fields order

[runcom]

Almost silly question, is json fields order in manifest important? My guess while developing a feature for docker was that order is important otherwise the manifest digest changes.

I may be missing something but is there some requirement about json fields order (especially the ones in descriptor) in the manifest json file? Should there be one to somehow assure digests are computed correctly? (and digests for manifests in image layout are correct for the same image)

Should we add a IMPLEMENTATION NOTE around this?

[runcom]

@vbatts @philips @stevvooe ^^

[runcom]

/cc @mtrmac

[mtrmac]

Looking at https://github.com/opencontainers/image-spec/blob/master/descriptor.md#digests-and-verification , it talks about verifying “bytes”. So I was thinking that the process goes:

• Taking input data, create a JSON object, serialize using JSON into a blob
• Compute a digest of this blob; publish the blob, use the digest in a content descriptor or so.

and

• Given a content descriptor or so, download the blob, verify the blob against the digest
• Then deserialize the blob using JSON into a JSON object, and extract the needed data out of this.

This is completely independent on the order of fields in JSON; the only “requirement” is that the producer and consumer of the JSON cooperate enough that the serialization and deserialization results in a semantically equivalent object.

[runcom]

FWIW I believe the same applies to config objects which are json content as well. (I'll do some testing)

[wking]

On Mon, Sep 05, 2016 at 10:44:00AM -0700, Miloslav Trmač wrote:

So I was thinking that the process goes:

• Taking input data, create a JSON object, serialize using JSON into
  a blob
• Compute a digest of this blob; publish the blob, use the digest in
  a content descriptor or so.

and

• Given a content descriptor or so, download the blob, verify the
  blob against the digest
• Then deserialize the blob using JSON into a JSON object, and
  extract the needed data out of this.

This is my understanding as well. Howerver, there is still some
benefit to suggesting a JSON-normaliation procedure because it
increases the chances that a newly-created JSON blob will match the
hash of an existing JSON blob (fewer CAS blobs for the same semantic
content). I feel like some normalization schemes were name-dropped on
dev@ last year, but haven't been able to dig up links. And this
applies to all JSON (and really, any format for which there are
normalization schemes that don't affect the semantics); it's not
specific to the OCI-specified formats.

[vbatts]

@runcom order only matters once there is a digest of the objects being used.

As a rule, no JSON parser/marshaller particularly guarantees order. Because to that point, even the same order, same content, "same document", but variations in indentation mean it is a "new" object.

[runcom]

@runcom order only matters once there is a digest of the objects being used.

As a rule, no JSON parser/marshaller particularly guarantees order. Because to that point, even the same order, same content, "same document", but variations in indentation mean it is a "new" object.

@vbatts yeah, I figured, so it's fine to have different manifest digests for the same image (this is where I was confused)
thanks

[stevvooe]

@vbatts @runcom Docker uses canonical json to try to provide some stability on round trip. IMHO, OCI should be less strict about round tripping, as it is a source of complexity and instability in practice, versus keeping stable artifacts.

The v2 manifest schema object follows https://github.com/docker/distribution/blob/master/docs/spec/json.md. This allows the specification to define an order then fallback to lexical sorting in the absence of an ordering. It is also written to be compatible with the implications of Go's json package, which follows struct ordering for structs and sorted ordering for maps.

[runcom]

thanks @stevvooe that's actually useful

[wking]

tl;dr: In the absence of a canonicalized JSON spec, I think we want to
stay out of this business. But if we find an external canonicalized
JSON spec, I think we want something like:

Implementations SHOULD serialize JSON blobs following {some
canonicalization spec}.

But I can't find a suitable spec, and without such a spec I think we
should avoid talking about JSON canonicalization completely.

Longer answer:

On Tue, Sep 06, 2016 at 02:01:18PM -0700, Stephen Day wrote:

@vbatts @runcom Docker uses canonical json to try to provide some
stability on round trip. IMHO, OCI should be less strict about round
tripping, as it is a source of complexity and instability in
practice, versus keeping stable artifacts.

The OCI should not require round-trip stability. But where it can
SHOULD a spec to encourage round-trip stability, I think that's a good
thing. I don't see how the SHOULD I propose above would introduce
either complexity or instability; an implementation can skip
canonicalization if it seems like too much trouble.

I'm not sure what there is to reference for ‘{some canonicalization
spec}’. 1 is all I can find in a quick search, and citing a wiki
seems unwise ;). The JOSE folks discussed JSON canonicalization in
2013 2, and the conclusion there seemed mixed between (a) “we
probably don't care”, (b) “canonicalizing XML was terrible, JSON is
probably bad too”, and (c) “canonicalalizing JSON isn't that bad”.
But I don't see anything serious about canonicalizing JSON in RFC 7515
3, so either (a) or (b) must have won out.

[stevvooe]

@wking http://wiki.laptop.org/go/Canonical_JSON is the "canonical JSON" spec to which we are referring. Docker has a go implementation at https://github.com/docker/go/tree/master/canonical/json.

[wking]

On Wed, Sep 07, 2016 at 11:26:29AM -0700, Stephen Day wrote:

@wking http://wiki.laptop.org/go/Canonical_JSON is the "canonical
JSON" spec to which we are referring. Docker has a go implementation
at https://github.com/docker/go/tree/master/canonical/json.

Sounds good. So the spec can SHOULD canonical JSON for serializing
JSON blobs, and we can point at that spec (with an implementation note
pointing at the Docker library and other known implementations).
Would you still have concerns about complexity and instability in that
case? If so, can you spell them out in a bit more detail?

[stevvooe]

@wking No. That is not at all what I've said. Go back, read my comments again.

[wking]

On Wed, Sep 07, 2016 at 11:52:14AM -0700, Stephen Day wrote:

@wking No. That is not at all what I've said. Go back, read my
comments again.

You said 1:

Docker uses canonical json to try to provide some stability on round
trip. IMHO, OCI should be less strict about round tripping, as it is
a source of complexity and instability in practice, versus keeping
stable artifacts.

I read that as:

• We don't want a MUST for canonicalization (from “OCI should be less
  strict about round tripping”), which I agree with.
• Something about complexity/instability, which I don't understand yet
  [2,3]. What is your source of complexity and instability? If I
  parse your sentence literally, you mean “round tripping”, which can
  be related to canonicalization or not depending on how you implement
  it. Given the context of this issue, I expected you are talking
  about canonicalization being related to complexity/instability. But
  I can't see a connection between SHOULD-level canonicalization and
  complexity/instability [2,3].