[Epic] Unpack image source using direct registry client

[anik120]

Overview

Currently Catalogd uses a Pod for unpacking catalog image contents and reads the unpacked contents from said Pod’s logs. There are a few problems associated with this:

• As mentioned by the Kubernetes documentation on working with Pods, Pods are designed as disposable entities. Pods that have exited successfully can be garbage collected at any time and thus there is no guarantee that it will still exist by the time we attempt to read the logs or that the logs that are present are lossless.
• It is not recommended to manage Pod resources yourself and instead use something like a Deployment, Job, or StatefulSet.
• When thinking about polling, we will likely be checking if the SHA of an image is different from the last time the image was unpacked. The general approach to doing this with a Pod based unpacking process is to create another pod to pull the same image and check the Pod status to determine if there is a difference in the SHA.

Directly communicating with an image registry allows us to avoid running into these issues altogether.

Related Issues

• Optimize memory usage of unpacking FBC from pod logs #22
• Explore alternative methods for unpacking catalog data #66
• Changing catalog image source doesn't unpack again #132

Design Documents

RFC: https://docs.google.com/document/d/1VHlUAVewIckVMLccNsoNptRsrQ19KcWzQ2HZetYK2h0/edit?usp=sharing

Task List

• (feature): add direct image registry client Unpacker implementation #145
• Make registry client unpacking the default and only Unpack implementation #182

[everettraven]

related to #143

[anik120]

@joelanford's description from #143:

There are several issues related to our use of a pod to unpack images:

https://github.com/operator-framework/catalogd/issues/22
https://github.com/operator-framework/catalogd/issues/66
https://github.com/operator-framework/catalogd/issues/132
I propose that we switch to an image registry client used directly by the controller. I think for the initial implementation we can:

Always resolve tags to SHAs by communicating with image registry
Pull content for tag into an emptyDir volume
i. If the content for a SHA is cached locally, no need to pull again
Unpack content into served filesystem
i. If the desired SHA is already being served, no need to unpack again
Future optimizations could include:

Running full resolve/unpack process in a goroutine, and using a channel to communicate events back to the reconciler to requeue
Rate-limit requests to an image registry (but if a request is rate-limited always put another request in the queue to make sure we don't rest on a stale tag resolution.
Explore cache optimizations (e.g. persistent volume, garbage collection based on LRU + max inactive size)