Skip to content

Support xz-encoded images#3160

Merged
openshift-merge-robot merged 2 commits intoopenshift:masterfrom
LorbusChris:xz-support
Mar 1, 2020
Merged

Support xz-encoded images#3160
openshift-merge-robot merged 2 commits intoopenshift:masterfrom
LorbusChris:xz-support

Conversation

@LorbusChris
Copy link
Copy Markdown
Contributor

@LorbusChris LorbusChris commented Feb 22, 2020 •
edited
Loading

Uses github.com/h2non/filetype for MIME type checking
and github.com/ulikunitz/xz to uncompress artifacts,
identified by application/x-xz

@openshift-ci-robot openshift-ci-robot added the size/XS Denotes a PR that changes 0-9 lines, ignoring generated files. label Feb 22, 2020
@abhinavdahiya
Copy link
Copy Markdown
Contributor

abhinavdahiya commented Feb 22, 2020

how is octect stream request to xz??

afaik vSphere ova is also octect stream and that is not xz compressed. I think we need to probably use the correct application type?

Also did this not have a closer call?

@LorbusChris
Copy link
Copy Markdown
Contributor Author

LorbusChris commented Feb 22, 2020

I think there was a catch wrt the octet-stream format, deferring to @vrutkovs

@LorbusChris LorbusChris requested a review from vrutkovs February 24, 2020 05:53
@vrutkovs
Copy link
Copy Markdown
Contributor

vrutkovs commented Feb 24, 2020

This should be fixed by FCOS server really, it serves all files as application/octet-stream

@LorbusChris
Copy link
Copy Markdown
Contributor Author

LorbusChris commented Feb 24, 2020

@vrutkovs do you know where that code lives offhand?

Also, do we need the closer call Abhinav mentioned?

@LorbusChris
Copy link
Copy Markdown
Contributor Author

LorbusChris commented Feb 24, 2020

The xz.Reader does not have a Close method.

@LorbusChris
Copy link
Copy Markdown
Contributor Author

LorbusChris commented Feb 24, 2020

http.DetectContentType only knows about MIME types in this spec: https://mimesniff.spec.whatwg.org/

That list is very much browser focused and a type only gets added there with proper browser support and testing, which I don't expect to happen for xz any time soon.

As any type not included in their list will default to octet-stream we still have to come up with a better mechanism for distinguishing them.

I'm thinking we should look at the file extension if we get the octet-stream MIME type default? I'll come up with something.

@openshift-ci-robot openshift-ci-robot added size/S Denotes a PR that changes 10-29 lines, ignoring generated files. size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files. and removed size/XS Denotes a PR that changes 0-9 lines, ignoring generated files. size/S Denotes a PR that changes 10-29 lines, ignoring generated files. labels Feb 24, 2020
@LorbusChris
Copy link
Copy Markdown
Contributor Author

LorbusChris commented Feb 24, 2020

Updated to use github.com/h2non/filetype for file type checking, which knows xz.

@LorbusChris LorbusChris mentioned this pull request Feb 24, 2020
Closed
@abhinavdahiya
Copy link
Copy Markdown
Contributor

abhinavdahiya commented Feb 24, 2020

i'm fine with the new vendor dep, but currently the commits are in mix state.. can you make sure you squash all non-vendor in one or make sure switch and add happen in separate ones ?

@LorbusChris LorbusChris force-pushed the xz-support branch 2 times, most recently from d4018de to dfe0b41 Compare February 25, 2020 12:35
@LorbusChris
Copy link
Copy Markdown
Contributor Author

LorbusChris commented Feb 25, 2020

updated :)

@LorbusChris
Copy link
Copy Markdown
Contributor Author

LorbusChris commented Feb 25, 2020

/retest

@LorbusChris
Copy link
Copy Markdown
Contributor Author

LorbusChris commented Feb 25, 2020 

Pulling image docker-registry.default.svc:5000/ci-op-rlh54...d40f249e0cca700891e64dfa733f9bef28a6c4ba03b97c8124258c5 ...
error: Unable to update build status: resource name may not be empty

looks like a CI hickup.
/retest

@LorbusChris
Copy link
Copy Markdown
Contributor Author

LorbusChris commented Feb 25, 2020

/retest

1 similar comment
@LorbusChris
Copy link
Copy Markdown
Contributor Author

LorbusChris commented Feb 25, 2020

/retest

@LorbusChris
Copy link
Copy Markdown
Contributor Author

LorbusChris commented Feb 25, 2020

/refresh

@LorbusChris
Copy link
Copy Markdown
Contributor Author

LorbusChris commented Feb 25, 2020

/retest

@LorbusChris
Copy link
Copy Markdown
Contributor Author

LorbusChris commented Feb 26, 2020

@abhinavdahiya I put the go.mod update in the first commit now. Is this what you meant or would you rather have it alongside the go mod vendor changes in the second?

@abhinavdahiya
Copy link
Copy Markdown
Contributor

abhinavdahiya commented Feb 26, 2020

the OpenStack failure is concerning... https://prow.svc.ci.openshift.org/view/gcs/origin-ci-test/pr-logs/pull/openshift_installer/3160/pull-ci-openshift-installer-master-e2e-openstack/4319#1:build-log.txt%3A1257

@pierreprinetti
Copy link
Copy Markdown
Contributor

pierreprinetti commented Feb 26, 2020 •
edited
Loading

/test e2e-openstack

transient network issue?

@LorbusChris
Copy link
Copy Markdown
Contributor Author

LorbusChris commented Feb 26, 2020 •
edited
Loading

nope, the newly added lib casts the MIME type as application/gzip instead of application/x-gzip (apparently they are equivalent) so the contents weren't uncompressed, and the checksum check failed subsequently. Should be fixed now.

@LorbusChris
Copy link
Copy Markdown
Contributor Author

LorbusChris commented Feb 26, 2020 •
edited
Loading

I've updated the commit msg to include a note about the gzip vs x-gzip issue

abhinavdahiya
abhinavdahiya reviewed Feb 26, 2020
View reviewed changes
Comment thread pkg/tfvars/internal/cache/cache.go Outdated
Copy link
Copy Markdown
Contributor

@abhinavdahiya abhinavdahiya Feb 26, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

is it possible to use the consts like https://github.com/h2non/filetype/blob/8241aa2c448465326c0002ccc6df7b910658960d/matchers/archive.go#L8 ?

pierreprinetti
pierreprinetti reviewed Feb 27, 2020
View reviewed changes
Comment thread pkg/tfvars/internal/cache/cache.go Outdated
@LorbusChris
cache: Support artifacts compressed with xz
3748d21 
Switches to "github.com/h2non/filetype/matchers" for MIME type matching
and uses "github.com/ulikunitz/xz" for xz decompression.
@LorbusChris
Update vendor
2832acb
@LorbusChris
Copy link
Copy Markdown
Contributor Author

LorbusChris commented Feb 27, 2020

Updated according to @pierreprinetti's proposal

@abhinavdahiya
Copy link
Copy Markdown
Contributor

abhinavdahiya commented Feb 27, 2020

/approve

@openshift-ci-robot
Copy link
Copy Markdown
Contributor

openshift-ci-robot commented Feb 27, 2020

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: abhinavdahiya

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@openshift-ci-robot openshift-ci-robot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Feb 27, 2020
@LorbusChris
Copy link
Copy Markdown
Contributor Author

LorbusChris commented Feb 28, 2020

/retest

@pierreprinetti
Copy link
Copy Markdown
Contributor

pierreprinetti commented Feb 28, 2020

we're experiencing infra issues on openstack's CI

/test e2e-openstack

@pierreprinetti
Copy link
Copy Markdown
Contributor

pierreprinetti commented Feb 28, 2020

/retest

@mandre
Copy link
Copy Markdown
Member

mandre commented Mar 1, 2020

/lgtm

@openshift-ci-robot openshift-ci-robot added the lgtm Indicates that a PR is ready to be merged. label Mar 1, 2020
@openshift-bot
Copy link
Copy Markdown
Contributor

openshift-bot commented Mar 1, 2020

/retest

Please review the full test history for this PR and help us cut down flakes.

@openshift-merge-robot openshift-merge-robot merged commit d00ba85 into openshift:master Mar 1, 2020
@openshift-ci-robot
Copy link
Copy Markdown
Contributor

openshift-ci-robot commented Mar 1, 2020

@LorbusChris: The following test failed, say /retest to rerun all failed tests:

Test name Commit Details Rerun command
ci/prow/e2e-libvirt 2832acb link /test e2e-libvirt

Full PR test history. Your PR dashboard. Please help us cut down on flakes by linking to an open issue when you hit one in your PR.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@jcpowermac jcpowermac Awaiting requested review from jcpowermac

@jhixson74 jhixson74 Awaiting requested review from jhixson74

@vrutkovs vrutkovs Awaiting requested review from vrutkovs

2 more reviewers

@abhinavdahiya abhinavdahiya abhinavdahiya left review comments

@pierreprinetti pierreprinetti pierreprinetti left review comments

Reviewers whose approvals may not affect merge requirements

Assignees

@mandre mandre

Labels

approved Indicates a PR has been approved by an approver from all required OWNERS files. lgtm Indicates that a PR is ready to be merged. size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files.

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

8 participants

@LorbusChris @abhinavdahiya @vrutkovs @pierreprinetti @openshift-ci-robot @mandre @openshift-bot @openshift-merge-robot