Skip to content

Registry bulk sync#506

Closed
grokspawn wants to merge 27 commits intoopenshift:masterfrom
grokspawn:registry-bulk-sync
Closed

Registry bulk sync#506
grokspawn wants to merge 27 commits intoopenshift:masterfrom
grokspawn:registry-bulk-sync

Conversation

@grokspawn
Copy link
Copy Markdown
Contributor

Current status: cannot make verify because manifests make target is broken due to interaction with helm/k8s dependency bumps.

joelanford and others added 27 commits July 7, 2023 15:59
* opm serve: use pre-existing cache, if set and up-to-date

Signed-off-by: Joe Lanford <joe.lanford@gmail.com>

* refactor to leave NewQuerier function untouched

Signed-off-by: Joe Lanford <joe.lanford@gmail.com>

Signed-off-by: Joe Lanford <joe.lanford@gmail.com>
Upstream-repository: operator-registry
Upstream-commit: 494b68e62a814a891821aeb2bd28f33abc1624ff
Signed-off-by: Jordan Keister <jordan@nimblewidget.com>

Signed-off-by: Jordan Keister <jordan@nimblewidget.com>
Upstream-repository: operator-registry
Upstream-commit: bd3c80489dbf6942005c345a33733fb782e952fe
Signed-off-by: Austin Macdonald <austin@redhat.com>

Signed-off-by: Austin Macdonald <austin@redhat.com>
Upstream-repository: operator-registry
Upstream-commit: 6d762ada58dfd90f41cc002d8f34c01811312f6f
Signed-off-by: jcho02 <jason.cho2@ibm.com>

Signed-off-by: jcho02 <jason.cho2@ibm.com>
Upstream-repository: operator-registry
Upstream-commit: 0271e7847d0b71cd969a2426f5532a4cb1cddd40
Signed-off-by: Jordan Keister <jordan@nimblewidget.com>

Signed-off-by: Jordan Keister <jordan@nimblewidget.com>
Upstream-repository: operator-registry
Upstream-commit: 89594183c5c9ea38c183d2f38c2644dba0f37024
#1043)

Signed-off-by: Joe Lanford <joe.lanford@gmail.com>

Signed-off-by: Joe Lanford <joe.lanford@gmail.com>
Upstream-repository: operator-registry
Upstream-commit: 0080ea0a938fdb91662a6ca36387aa71d3a52b28
Signed-off-by: Joe Lanford <joe.lanford@gmail.com>

Signed-off-by: Joe Lanford <joe.lanford@gmail.com>
Upstream-repository: operator-registry
Upstream-commit: b7307b57c5f67b296098e66bfe38458e762c99aa
* Bump k8s dependencies to 1.26

Signed-off-by: Rashmi Gottipati <chowdary.grashmi@gmail.com>

* Attempting to fix sanity test

Signed-off-by: Rashmi Gottipati <chowdary.grashmi@gmail.com>

---------

Signed-off-by: Rashmi Gottipati <chowdary.grashmi@gmail.com>
Upstream-repository: operator-registry
Upstream-commit: fe5123023244fb997e00d831dee10eaf4d8b5fc0
Bumps [golang.org/x/net](https://github.com/golang/net) from 0.4.0 to 0.7.0.
- [Release notes](https://github.com/golang/net/releases)
- [Commits](golang/net@v0.4.0...v0.7.0)

Upstream-repository: operator-registry
Upstream-commit: 0dff010522f4639d0a6089fdded19b595832f452

---
updated-dependencies:
- dependency-name: golang.org/x/net
  dependency-type: direct:production
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
…dout (#1056)

Signed-off-by: Joe Lanford <joe.lanford@gmail.com>
Upstream-repository: operator-registry
Upstream-commit: 3bfcb23fa7d0523ad187ef2cbb396dcc1fcba5df
* Re-add logic for switching request

Signed-off-by: Ish Shah <ishah@redhat.com>

* string issues

Signed-off-by: Ish Shah <ishah@redhat.com>

* add read after err check

Signed-off-by: Ish Shah <ishah@redhat.com>

* change error outs

Signed-off-by: Ish Shah <ishah@redhat.com>

---------

Signed-off-by: Ish Shah <ishah@redhat.com>
Upstream-repository: operator-registry
Upstream-commit: 49eae7b7fac9ad7d6dd80ffbdd5266eef9feef21
* impose order on maps

Signed-off-by: Jordan Keister <jordan@nimblewidget.com>

* now modding GetBundle

Signed-off-by: Jordan Keister <jordan@nimblewidget.com>

* more terse

Signed-off-by: Jordan Keister <jordan@nimblewidget.com>

* forgot bundles

Signed-off-by: Jordan Keister <jordan@nimblewidget.com>

---------

Signed-off-by: Jordan Keister <jordan@nimblewidget.com>
Upstream-repository: operator-registry
Upstream-commit: 7835ea9c613ad1009cc36df8ad16e091d7775368
….18 (#1062)

Bumps [github.com/containerd/containerd](https://github.com/containerd/containerd) from 1.5.13 to 1.5.18.
- [Release notes](https://github.com/containerd/containerd/releases)
- [Changelog](https://github.com/containerd/containerd/blob/main/RELEASES.md)
- [Commits](containerd/containerd@v1.5.13...v1.5.18)

Upstream-repository: operator-registry
Upstream-commit: 8006f50ca54550bcb894a283138e3b97de7062fa

---
updated-dependencies:
- dependency-name: github.com/containerd/containerd
  dependency-type: direct:production
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Signed-off-by: akihikokuroda <akihikokuroda2020@gmail.com>
Upstream-repository: operator-registry
Upstream-commit: 0aebc3ceb4d2504eb03f470116f5eb2ee8f0a3bc
Removes the vendor directory from source control and attempts to cache the vendor dir when github actions are run. Also removes a couple of Dockerfiles which are no longer used and relied on the vendor directory.
Signed-off-by: dtfranz <dfranz@redhat.com>
Upstream-repository: operator-registry
Upstream-commit: 45f491b932997c20ac9c6e54b4b4fa822994cefb
* refactor to move more processing to the template

Signed-off-by: Jordan Keister <jordan@nimblewidget.com>

* better diagnostics on custom exec error

Signed-off-by: Jordan Keister <jordan@nimblewidget.com>

* Remove obsolete fields

Signed-off-by: Catherine Chan-Tse <cchantse@redhat.com>
Signed-off-by: Jordan Keister <jordan@nimblewidget.com>

* caching wip

Signed-off-by: Jordan Keister <jordan@nimblewidget.com>

* refactor a bit and update unit tests (#2)

* refactor a bit and update unit tests

Signed-off-by: Bryce Palmer <bpalmer@redhat.com>

* add more utests and minor tweaks

Signed-off-by: Bryce Palmer <bpalmer@redhat.com>

---------

Signed-off-by: Bryce Palmer <bpalmer@redhat.com>
Signed-off-by: Jordan Keister <jordan@nimblewidget.com>

* whitespace sanity hell

Signed-off-by: Jordan Keister <jordan@nimblewidget.com>

* drop STDERR for nominal case

Signed-off-by: Jordan Keister <jordan@nimblewidget.com>

---------

Signed-off-by: Jordan Keister <jordan@nimblewidget.com>
Signed-off-by: Catherine Chan-Tse <cchantse@redhat.com>
Signed-off-by: Bryce Palmer <bpalmer@redhat.com>
Co-authored-by: Catherine Chan-Tse <cchantse@redhat.com>
Co-authored-by: Bryce Palmer <everettraven@gmail.com>
Upstream-repository: operator-registry
Upstream-commit: 00a4cce4847d75f71bbcb574e0e7ac364fd9f546
…1090)

* comment out CatalogDestination.BaseImage while we mature the concept

Signed-off-by: Jordan Keister <jordan@nimblewidget.com>

* removed baseImage from test data and validations

Signed-off-by: Jordan Keister <jordan@nimblewidget.com>

---------

Signed-off-by: Jordan Keister <jordan@nimblewidget.com>
Upstream-repository: operator-registry
Upstream-commit: 41fa00f331c0281a3419b2c59504ea35097a46c8
* add WalkMetasFS and WalkMetasReader functions

These function are like WalkFS except that callers provide a function
that handles a Meta object at a time, rather than an entire file's worth
of FBC objects.

For callers that can operate on a single meta at a time, these functions
will provide significant reductions in memory usage.

Signed-off-by: Joe Lanford <joe.lanford@gmail.com>

* address PR feedback

Signed-off-by: Joe Lanford <joe.lanford@gmail.com>

---------

Signed-off-by: Joe Lanford <joe.lanford@gmail.com>
Upstream-repository: operator-registry
Upstream-commit: e1eebae2130c3987eeff7dc9f69672b41dd0ba78
…e otherwise identical (#1095)

* minorly-kludgy approach, but functional

Signed-off-by: Jordan Keister <jordan@nimblewidget.com>

* review comments

Signed-off-by: Jordan Keister <jordan@nimblewidget.com>

* retool flag checks and defaulting

Signed-off-by: Jordan Keister <jordan@nimblewidget.com>

* more review updates

Signed-off-by: Jordan Keister <jordan@nimblewidget.com>

---------

Signed-off-by: Jordan Keister <jordan@nimblewidget.com>
Upstream-repository: operator-registry
Upstream-commit: 7629c6f991fc8dcce91bcb5ee4e998678bbd7779
* render: improve olm.bundle.object rendering for bundles

When rendering individual bundles, only generate olm.bundle.object
properties for the CSV if there is an image reference for the bundle.

Signed-off-by: Joe Lanford <joe.lanford@gmail.com>

* render: improve olm.bundle.object rendering for sqlite dbs

When rendering sqlite-based catalogs, only generate olm.bundle.object
properties for the CSV if there is an image reference for the bundle.

Signed-off-by: Joe Lanford <joe.lanford@gmail.com>

* introduce benchmark for declcfg.LoadFS

Signed-off-by: Joe Lanford <joe.lanford@gmail.com>

* concurrent LoadFS

Signed-off-by: Joe Lanford <joe.lanford@gmail.com>

* add olm.csv.metadata property

1. When rendering sqlite DBs and bundle images, generate an
   "olm.csv.metadata" property instead of a full CSV (so long as
   there is a bundle image reference associated with the corresponding
   bundle)
2. When serving the GRPC interface and a full CSV is not present in an
   "olm.bundle.object" property, generate a CSV from (a) the
   "olm.csv.metadata" property. Also include the bundle's related
   images, and the package's icon, if defined. If there is no
   description in the CSV metadata, also include the package's
   description in the generated CSV.

Signed-off-by: Joe Lanford <joe.lanford@gmail.com>

* Meta: more correct handling of HTML-unescaping

It turns out that straight byte-based replacements of unicode escape
characters back to their ascii representations is invalid if the unicode
escape character itself is escaped (e.g. "\u003c" => "\\u003c" => "\<").

To solve this, we will instead unmarshal Meta objects to
map[string]interface{}, extract the expected Meta fields from the map,
and then use a JSON encoder with SetEscapeHTML(false) to re-encode the
map back to JSON to be stored in Meta.Blob.

Signed-off-by: Joe Lanford <joe.lanford@gmail.com>

* updated LoadFS benchmark to use csv metadata properties instead of olm.bundle.object properties

Signed-off-by: Joe Lanford <joe.lanford@gmail.com>

---------

Signed-off-by: Joe Lanford <joe.lanford@gmail.com>
Upstream-repository: operator-registry
Upstream-commit: 2ee231b3235cb38aeb045845627a7d7ffb98d1b4
Signed-off-by: Jordan <jordan@nimblewidget.com>
Upstream-repository: operator-registry
Upstream-commit: b51aaf05bdafdc91e69c5e2a8c26097bf9ebbb1a
* *: thread through context into FBC load

Signed-off-by: Steve Kuznetsov <skuznets@redhat.com>

* declcfg/load: allow configuring the parallelism

Signed-off-by: Steve Kuznetsov <skuznets@redhat.com>

---------

Signed-off-by: Steve Kuznetsov <skuznets@redhat.com>
Upstream-repository: operator-registry
Upstream-commit: 647537d941108ed88e47b559eb98d134967444ad
Signed-off-by: Joe Lanford <joe.lanford@gmail.com>
Upstream-repository: operator-registry
Upstream-commit: 6c602841934d6e154e38c0574cc140471dc063e6
Signed-off-by: Mikalai Radchuk <mradchuk@redhat.com>
Upstream-repository: operator-registry
Upstream-commit: 337b60ddc37d4cf3f8a8d8c00df7a76c55a5e12d
Signed-off-by: Mikalai Radchuk <mradchuk@redhat.com>
Upstream-repository: operator-registry
Upstream-commit: b854557bd44020f46383e0f5d483bf56c297aaf7
* Bump k8s.io/client-go from v0.26.1 to v0.27.1

Signed-off-by: Edmund Ochieng <ochienged@gmail.com>

* Bump controller-runtime version from v0.14.4 to v0.15.0

Signed-off-by: Edmund Ochieng <ochienged@gmail.com>

---------

Signed-off-by: Edmund Ochieng <ochienged@gmail.com>
Upstream-repository: opeator-registry
Upstream-commit: 43225def6de9c8fd7f8342c822c210911a872d5a
Remove executable bits from file creation

Signed-off-by: Rami <72725910+ramikg@users.noreply.github.com>
Upstream-repository: operator-registry
Upstream-commit: bf549304d8810c55d0953310bd47ad6a489e5379
@openshift-ci openshift-ci Bot requested review from awgreene and perdasilva July 10, 2023 14:32
@openshift-ci
Copy link
Copy Markdown
Contributor

openshift-ci Bot commented Jul 10, 2023

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: grokspawn

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@openshift-ci openshift-ci Bot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Jul 10, 2023
@tmshort
Copy link
Copy Markdown
Contributor

tmshort commented Jul 17, 2023

Please see #510

@openshift-ci
Copy link
Copy Markdown
Contributor

openshift-ci Bot commented Jul 24, 2023

@grokspawn: The following tests failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name Commit Details Required Rerun command
ci/prow/verify b07da9d link true /test verify
ci/prow/e2e-gcp-olm-flaky b07da9d link false /test e2e-gcp-olm-flaky
ci/prow/images b07da9d link true /test images
ci/prow/unit-psm b07da9d link true /test unit-psm
ci/prow/e2e-upgrade b07da9d link true /test e2e-upgrade
ci/prow/e2e-gcp-console-olm b07da9d link true /test e2e-gcp-console-olm
ci/prow/e2e-gcp-olm b07da9d link true /test e2e-gcp-olm
ci/prow/e2e-gcp-ovn b07da9d link true /test e2e-gcp-ovn

Full PR test history. Your PR dashboard.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

@tmshort
Copy link
Copy Markdown
Contributor

tmshort commented Jul 27, 2023

This can be closed due to merging #510

@openshift-merge-robot openshift-merge-robot added the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Jul 27, 2023
@openshift-merge-robot
Copy link
Copy Markdown
Contributor

PR needs rebase.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

@grokspawn grokspawn closed this Aug 1, 2023
@grokspawn grokspawn deleted the registry-bulk-sync branch August 1, 2023 11:51
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

approved Indicates a PR has been approved by an approver from all required OWNERS files. needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD.

Projects

None yet

Development

Successfully merging this pull request may close these issues.