ostree, src: support copy of compressed layers #392

giuseppe · 2017-12-14T17:30:53Z

add implementation for LayerInfosForCopy so we can provide a modified manifest to be used for copy

runcom · 2017-12-15T14:06:15Z

Looks ok, @mtrmac wdyt?

giuseppe · 2018-01-11T09:45:34Z

ping

giuseppe · 2018-01-15T09:26:26Z

cc @nalind

mtrmac · 2018-01-16T14:03:05Z

ostree/ostree_dest.go

+	digester := digest.Canonical.Digester()
+	tee := io.TeeReader(its, digester.Hash())
+
+	written, err := io.Copy(ioutil.Discard, tee)


Isn’t this basically io.Copy(digester, its)?

correct, thanks for finding it. Fixed it in a newer version

mtrmac · 2018-01-16T14:14:51Z

ostree/ostree_src.go

+	}
+
+	var schema manifestSchema
+	if err := json.Unmarshal(manifestBlob, &schema); err != nil {


Please use the containers/image/manifest package (manifest.FromBlob), both for manifest types and for manifest parsing.

This doesn't work as expected because LayersDescriptors and FSLayers list layers in the opposite directions; besides, the manifestSchema franken-type containing a subset of both schema1 and schema2, and undocumented, is pretty surprising.

missed this comment in the last revision. Going to address it now.

mtrmac · 2018-01-16T14:16:46Z

ostree/ostree_src.go

+	ref        ostreeReference
+	tmpDir     string
+	repo       *C.struct_OstreeRepo
+	compressed map[digest.Digest]digest.Digest


Please at least add a comment here saying what is the key and what is the value of the map. (Ideally the field name would be self-explanatory, but that seems difficult.)

mtrmac · 2018-01-16T14:17:25Z

ostree/ostree_src.go

 func (s *ostreeImageSource) GetBlob(info types.BlobInfo) (io.ReadCloser, int64, error) {
+
 	blob := info.Digest.Hex()
+	compressedBlob, found := s.compressed[info.Digest]


It’s not in general guaranteed that LayerInfosForCopy will be called before GetBlob; if GetBlob depends on this map, it may need to build it on demand.

thanks, I've modified it to check if compressed is initialized, and use LayerInfosForCopy when it is nil

mtrmac

LGTM otherwise.

mtrmac · 2018-01-19T21:56:27Z

ostree/ostree_src.go

+			MediaType: layerBlob.MediaType,
+		}
+		s.compressed[uncompressedDigest] = layerBlob.Digest
+		updatedBlobInfos = append([]types.BlobInfo{blobInfo}, updatedBlobInfos...)


LayerInfosForCopy should return layers in the order of man.LayerInfos, not reversed.

mtrmac · 2018-01-19T21:58:58Z

ostree/ostree_src.go

+	updatedBlobInfos := []types.BlobInfo{}
+	manifestBlob, manifestType, err := s.GetManifest(nil)
+	if err != nil {
+		return nil


Not sure about some other cases, but at least in this one it seems clear that err shouldn’t be thrown away; can you add an error return value to LayerInfosForCopy?

this is the signature from the ImageSource interface. The only user of this function in ostree_src is GetBlob and I handle there the nil return, which is considered an error.

Right; please change the interface. c/i/copy/imageCopier.copyLayers should be able to tell the difference between “no changes needed” and “reading manifest failed” / ”manifest invalid”.

I've pushed a new change to change the interface

mtrmac · 2018-01-23T20:15:11Z

ostree/ostree_src.go

+		uncompressedSize, err := strconv.ParseInt(uncompressedSizeStr, 10, 64)
+		if err != nil {
+			return nil, nil
+		}


Is it reasonable to ignore the above errors? At least the last one seems potentially worth failing on.

Up to you.

thanks, I've changed that to return an error. Also, I have added a patch that will trigger the layer to be pulled if this metadata is missing.

mtrmac · 2018-01-23T20:15:39Z

storage/storage_image.go

 // LayerInfosForCopy() returns the list of layer blobs that make up the root filesystem of
 // the image, after they've been decompressed.
-func (s *storageImageSource) LayerInfosForCopy() []types.BlobInfo {
+func (s *storageImageSource) LayerInfosForCopy() ([]types.BlobInfo, error) {


At least some of the errors in here should probably cause copy.Image to fail.

Cc: @nalind

I preferred to not touch this part as I am not really familiar with it. I can change it if it is required for this PR

Generally, if storageImageSource.LayerInfosForCopy() is returning nil, the copy attempt is going to fail later on when the digests don't match, since I think the majority of images use compressed layers. So yeah, pretty much anywhere we return nil from there, we should start returning the error value that caused us to do so.

I've added a patch that changes LayerInfosForCopy in storage/storage_image.go to propagate the errors.

mtrmac · 2018-01-24T14:03:07Z

Thanks!

LGTM, will wait a bit to give @nalind time to comment on storageImageSource.LayerInfosForCopy

mtrmac · 2018-01-26T10:14:48Z

👍 for the record; let’s at least track storageImageSource.LayerInfosForCopy in an issue.

rhatdan · 2018-01-29T12:58:58Z

@runcom PTAL

runcom · 2018-01-30T11:30:46Z

@runcom PTAL

I'm waiting on #392 (comment)

giuseppe · 2018-02-01T15:59:49Z

I took care of the last comment I got on this PR

rhatdan · 2018-02-02T18:13:13Z

@runcom PTAL

giuseppe · 2018-02-06T10:24:36Z

rebased ⬆️

we will use it later for LayerInfosForCopy Signed-off-by: Giuseppe Scrivano <gscrivan@redhat.com>

Signed-off-by: Giuseppe Scrivano <gscrivan@redhat.com>

it enables the copy of compressed images. Signed-off-by: Giuseppe Scrivano <gscrivan@redhat.com>

Signed-off-by: Giuseppe Scrivano <gscrivan@redhat.com>

giuseppe · 2018-02-21T12:04:07Z

rebased again. Could we please move this forward?

mtrmac · 2018-02-21T15:35:05Z

@runcom ping

runcom · 2018-02-22T11:22:27Z

LGTM

giuseppe force-pushed the ostree-support-copy-layer-infos branch from 03fa006 to 15961b8 Compare December 14, 2017 17:31

giuseppe mentioned this pull request Dec 14, 2017

Rework how we use containers/storage #305

Merged

giuseppe force-pushed the ostree-support-copy-layer-infos branch from 15961b8 to b706d9a Compare December 15, 2017 13:40

giuseppe force-pushed the ostree-support-copy-layer-infos branch 2 times, most recently from 5987770 to 5f2b693 Compare January 15, 2018 09:51

mtrmac reviewed Jan 16, 2018

View reviewed changes

giuseppe force-pushed the ostree-support-copy-layer-infos branch 7 times, most recently from 5790ef7 to 961bfca Compare January 19, 2018 12:54

mtrmac reviewed Jan 19, 2018

View reviewed changes

giuseppe force-pushed the ostree-support-copy-layer-infos branch 8 times, most recently from 975896a to d80ea35 Compare January 23, 2018 14:38

mtrmac reviewed Jan 23, 2018

View reviewed changes

giuseppe force-pushed the ostree-support-copy-layer-infos branch from d80ea35 to 0d3a4d0 Compare January 24, 2018 08:53

giuseppe force-pushed the ostree-support-copy-layer-infos branch from 0f47cc9 to 1562f1c Compare February 6, 2018 10:24

giuseppe force-pushed the ostree-support-copy-layer-infos branch from 1562f1c to a4df631 Compare February 6, 2018 10:44

giuseppe mentioned this pull request Feb 6, 2018

Introduce new type PAYLOAD_LINK ostreedev/ostree#1443

Closed

giuseppe added 5 commits February 21, 2018 13:03

ostree: store uncompressed size and digest

8435227

we will use it later for LayerInfosForCopy Signed-off-by: Giuseppe Scrivano <gscrivan@redhat.com>

types: change LayerInfosForCopy to return an error

3b84e21

Signed-off-by: Giuseppe Scrivano <gscrivan@redhat.com>

ostree: provide LayerInfosForCopy

c15f789

it enables the copy of compressed images. Signed-off-by: Giuseppe Scrivano <gscrivan@redhat.com>

ostree: re-pull layer if it misses uncompressed_digest|uncompressed_size

d994578

Signed-off-by: Giuseppe Scrivano <gscrivan@redhat.com>

storage: LayerInfosForCopy propagates errors

755cd3f

Signed-off-by: Giuseppe Scrivano <gscrivan@redhat.com>

giuseppe force-pushed the ostree-support-copy-layer-infos branch from a4df631 to 755cd3f Compare February 21, 2018 12:03

runcom merged commit b44c527 into containers:master Feb 22, 2018

ostree, src: support copy of compressed layers #392

ostree, src: support copy of compressed layers #392

Uh oh!

Conversation

giuseppe commented Dec 14, 2017

Uh oh!

runcom commented Dec 15, 2017

Uh oh!

giuseppe commented Jan 11, 2018

Uh oh!

giuseppe commented Jan 15, 2018

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

mtrmac Jan 16, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

mtrmac left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

giuseppe Jan 20, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

mtrmac commented Jan 24, 2018 • edited by runcom Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

mtrmac commented Jan 26, 2018

Uh oh!

rhatdan commented Jan 29, 2018

Uh oh!

runcom commented Jan 30, 2018

Uh oh!

giuseppe commented Feb 1, 2018

Uh oh!

rhatdan commented Feb 2, 2018

Uh oh!

giuseppe commented Feb 6, 2018

Uh oh!

giuseppe commented Feb 21, 2018

Uh oh!

mtrmac commented Feb 21, 2018

Uh oh!

runcom commented Feb 22, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Reviewers

mtrmac Jan 16, 2018 •

edited

Loading

giuseppe Jan 20, 2018 •

edited

Loading

mtrmac commented Jan 24, 2018 •

edited by runcom

Loading

runcom commented Feb 22, 2018 •

edited

Loading