Store(local dir) and Serve(http server) FBC by anik120 · Pull Request #135 · operator-framework/catalogd

anik120 · 2023-08-03T19:11:59Z

closes #113

codecov · 2023-08-03T19:16:42Z

Codecov Report

Merging #135 (e59cdff) into main (a8f7196) will decrease coverage by 1.07%.
The diff coverage is 53.84%.

@@            Coverage Diff             @@
##             main     #135      +/-   ##
==========================================
- Coverage   78.01%   76.94%   -1.07%     
==========================================
  Files           2        2              
  Lines         282      295      +13     
==========================================
+ Hits          220      227       +7     
- Misses         39       42       +3     
- Partials       23       26       +3

Files Changed	Coverage Δ
pkg/controllers/core/catalog_controller.go	`76.79% <53.84%> (-1.07%)`	⬇️

anik120 · 2023-08-03T19:51:48Z

@ncdc it's a WIP PR 😄

(I didn't think anyone had notifications on for new PRs for this repo yet, hence the skipping of the WIP tag 😮‍💨 )

ncdc · 2023-08-03T19:53:01Z

Sorry, saw the notification come through so I started looking at it.

anik120 · 2023-08-03T19:55:39Z

@ncdc fyi this is just copy paste of https://github.com/operator-framework/rukpak/blob/main/pkg/storage/localdir.go (I was using this for a private conversation), but it looks like we should look into auditing this in rukpak based off of your suggestions

stevekuznetsov

What's the difference between the server and the storage? Why do we need two separate concepts as opposed to one?

everettraven · 2023-08-15T13:07:43Z

        - "--metrics-bind-address=127.0.0.1:8080"
        - "--leader-elect"
-        - "--feature-gates=PackagesBundleMetadataAPIs=true,CatalogMetadataAPI=true"
+        - "--feature-gates=PackagesBundleMetadataAPIs=true,CatalogMetadataAPI=true,HTTPServer=false"


Do we want the http server disabled by default? I figure we probably want all of them enabled so we can start the deprecation/removal process after a new release with these changes.

I'm still thinking this should probably be set to true by default so that all of the serving methods are available by default in the next release and we can remove the Package , BundleMetadata, and CatalogMetadata in the next+1 release. That being said that can be done in a follow up if we want and shouldn't block this PR

I don't think we even have to wait for the next +1 release. We should be fine just removing those, and switching this to true. But yea let's leave that for a follow up

everettraven · 2023-08-15T13:20:49Z

+	if features.CatalogdFeatureGate.Enabled(features.HTTPServer) && existingCatsrc.DeletionTimestamp.IsZero() && !controllerutil.ContainsFinalizer(&existingCatsrc, fbcDeletionFinalizer) {
+		controllerutil.AddFinalizer(&existingCatsrc, fbcDeletionFinalizer)
+		if err := r.Update(ctx, &existingCatsrc); err != nil {
+			return ctrl.Result{}, err
+		}
+	}
+	if features.CatalogdFeatureGate.Enabled(features.HTTPServer) && !existingCatsrc.DeletionTimestamp.IsZero() && controllerutil.ContainsFinalizer(&existingCatsrc, fbcDeletionFinalizer) {
+		if err := r.Storage.Delete(existingCatsrc.Name); err != nil {
+			return ctrl.Result{}, err
+		}
+		controllerutil.RemoveFinalizer(&existingCatsrc, fbcDeletionFinalizer)
+		err := r.Update(ctx, &existingCatsrc)
+		return ctrl.Result{}, err
+	}


Nit: Based on our current code structure, would this logic be better suited in the reconcile() function below? IIUC any changes to the finalizer will be reflected when we issue the update requests at the end of this function. For consistency I would prefer we do our finalizer update logic where we also do the status updates.

I think we need to actually add the finalizer to the object before storing anything into the storage directory. Here's the problematic scenario:

Lookup and get a copy of the object we're reconciling

Add a finalizer to the copy (without actually updating it in etcd)

Store the catalog data

Send the object update, but get a failure on the update

Now, we've stored the data, but a finalizer isn't present to make sure it gets deleted when the object is deleted.

But +1 on moving the finalizer logic to reconcile().

What we could do there is add the finalizer to the copy, but then just return immediately when we've added the finalizer, and let the existing diff-ing update logic here Reconcile handle the actual update call.

ncdc · 2023-08-15T14:07:59Z

-
+	if features.CatalogdFeatureGate.Enabled(features.HTTPServer) && existingCatsrc.DeletionTimestamp.IsZero() && !controllerutil.ContainsFinalizer(&existingCatsrc, fbcDeletionFinalizer) {
+		controllerutil.AddFinalizer(&existingCatsrc, fbcDeletionFinalizer)
+		if err := r.Update(ctx, &existingCatsrc); err != nil {


It's almost always best to use Patch instead of Update. When using Update:

If the running code has an older version of the go struct definition, and

The CRD in the cluster contains new fields, and

There are CRs that have the new fields populated, then

When you issue an Update using the older client, the result is the new fields will be nulled out

This is probably orthogonal to this PR, given we already have some generic diff-then-update logic in the Reconcile() method. I propose we make a separate issue to change the existing Update calls to Patch.

ncdc · 2023-08-15T14:09:34Z

+	}
+	if features.CatalogdFeatureGate.Enabled(features.HTTPServer) && !existingCatsrc.DeletionTimestamp.IsZero() && controllerutil.ContainsFinalizer(&existingCatsrc, fbcDeletionFinalizer) {
+		if err := r.Storage.Delete(existingCatsrc.Name); err != nil {
+			return ctrl.Result{}, err


We should set a condition on the Catalog here

ncdc · 2023-08-15T14:09:45Z

+			return ctrl.Result{}, err
+		}
+		controllerutil.RemoveFinalizer(&existingCatsrc, fbcDeletionFinalizer)
+		err := r.Update(ctx, &existingCatsrc)


Ditto re patch

ncdc · 2023-08-15T14:10:29Z

-
+		if features.CatalogdFeatureGate.Enabled(features.HTTPServer) {
+			if err := r.Storage.Store(catalog.Name, fbc); err != nil {
+				return ctrl.Result{}, updateStatusUnpackFailing(&catalog.Status, fmt.Errorf("error storing fbc: %v", err))


We should set a condition on the Catalog here

stevekuznetsov · 2023-08-16T13:41:05Z

+	}
+}
+
+func (s *Storage) Store(owner string, fbc *declcfg.DeclarativeConfig) error {


One other thing we glossed over - when a catalog is updated, we will call Store() again with the same name as before. I think the most seamless behavior here is:

write contents to a new file

move new file to the location of the old one

don't touch the HTTP handlers

This will ensure zero-downtime for the callers.

This is important.

I've captured this as a follow up item. We need an entire story for handling updates to the content in the remote registry (ie what we currently call the polling strategy in v0), and for handling changes to the spec.image field of the CR.

Does the controller not handle update events for CatalogSource objects? Is there no place where this can slot in already?

No. It's unimplemented. So far what we have is "creation and deletion once". We haven't gotten to updates at all.

obsoletes operator-framework#113 Signed-off-by: Anik <anikbhattacharya93@gmail.com>

Signed-off-by: Anik <anikbhattacharya93@gmail.com>

everettraven · 2023-08-16T21:01:11Z

+						Expect(res).To(Equal(ctrl.Result{}))
+						Expect(err).ToNot(HaveOccurred())
+					})
+					It("the catalog should become available at addr/catalogs", func() {


Could be done in a follow up, but should we also validate that the returned JSON stream from the catalog being stored matches what we expect? IIRC, the tests for the other methods of storing/serving validate that the actual content == expected content

Signed-off-by: Anik <anikbhattacharya93@gmail.com>

everettraven

Overall this looks good to me. Have one nit and a couple comments that I think would be nice to have addressed either as part of this PR or a follow-up.

everettraven · 2023-08-17T16:22:18Z

+	fs := http.FileServer(http.FS(os.DirFS(dir)))
+	return fs


Nit: Can reduce this to a one liner:

Suggested change

fs := http.FileServer(http.FS(os.DirFS(dir)))

return fs

return http.FileServer(http.FS(os.DirFS(dir)))

I would also expect http.StripPath to enter into the equation somewhere.

The URL paths will come in with /catalogs/<catalogName>/all.json, but our http.FileSystem will only see <catalogName>/all.json, so we need to strip /catalogs from the path before the http.FileSystem sees the request.

What's the benefit of making the path /catalogs/catalogName/all.json? Why not just keep it simple to /catalogName.json?

The original intention behind /catalogs/catalogName/all.json was to leave room for expanding the endpoints (files being served) if desired without impacting clients wanting everything. For example, if we wanted to expand via file serving for getting only all the bundles for the catalog we could add /catalogs/{catalogName}/bundles.json without changing the API for clients that always want to fetch everything. It becomes purely additive (and is IMO intuitive) to add new file based endpoints with that structure

everettraven · 2023-08-17T16:25:23Z

        - "--metrics-bind-address=127.0.0.1:8080"
        - "--leader-elect"
-        - "--feature-gates=PackagesBundleMetadataAPIs=true,CatalogMetadataAPI=true"
+        - "--feature-gates=PackagesBundleMetadataAPIs=true,CatalogMetadataAPI=true,HTTPServer=false"


I'm still thinking this should probably be set to true by default so that all of the serving methods are available by default in the next release and we can remove the Package , BundleMetadata, and CatalogMetadata in the next+1 release. That being said that can be done in a follow up if we want and shouldn't block this PR

everettraven · 2023-08-17T16:28:37Z

+					It("the catalog should become available at server endpoint", func() {
+						resp, err := httpclient.Do(httpRequest)
+						Expect(err).To(Not(HaveOccurred()))
+						defer resp.Body.Close()
+
+						catalogs, err := io.ReadAll(resp.Body)
+						Expect(err).To(Not(HaveOccurred()))
+						Expect(string(catalogs)).To(Equal(fmt.Sprintf(httpResponse, fmt.Sprintf("\n<a href=\"%s\">%s</a>", catalogKey.Name+".json", catalogKey.Name+".json"))))
+					})


I'd like to see this validate the response matches what we expect when we request the catalog contents rather than checking that there is a link to the file when visiting the index endpoint, but I'm okay with that being a follow-up.

Seems like we should test that this returns the content we want, if not, what are we confident that we've written here?

+1, See https://github.com/operator-framework/catalogd/pull/135/files#r1297527024

stevekuznetsov · 2023-08-17T16:48:35Z

+	}
+	// if the ShutdownTimeout is zero, wait forever to shutdown
+	// otherwise force shut down when timeout expires
+	sc := context.Background()


nit: sc and scc are opaque names that I'm not sure everyone will unpack (I struggled!)

How bout shutdownCtx and shutdownCancel?

stevekuznetsov · 2023-08-17T16:50:19Z

+						req, err := http.NewRequest("GET", testServer.URL, nil)
+						Expect(err).To(Not(HaveOccurred()))
+						req.Header.Set("Accept", "text/html")
+						httpRequest = req


Why do we need a side-effect mutation to bring this up a scope? Why not create the request in the test closure that uses it?

creating that request in multiple closures is just repeating the same code

What's wrong with that?

A little bit less tongue-in-cheek: tests must be as independent as possible. The more that tests are intertwined, it's harder to break just one and grok all of it at once while reviewing changes. Repeating two or three lines of code is really not something we should be optimizing for.

stevekuznetsov · 2023-08-17T16:51:04Z

+					It("the catalog should become available at server endpoint", func() {
+						resp, err := httpclient.Do(httpRequest)
+						Expect(err).To(Not(HaveOccurred()))
+						defer resp.Body.Close()
+
+						catalogs, err := io.ReadAll(resp.Body)
+						Expect(err).To(Not(HaveOccurred()))
+						Expect(string(catalogs)).To(Equal(fmt.Sprintf(httpResponse, fmt.Sprintf("\n<a href=\"%s\">%s</a>", catalogKey.Name+".json", catalogKey.Name+".json"))))
+					})


Seems like we should test that this returns the content we want, if not, what are we confident that we've written here?

stevekuznetsov · 2023-08-17T16:51:43Z

+	}
+}
+
+func (s *Storage) Store(owner string, fbc *declcfg.DeclarativeConfig) error {


This is important.

joelanford · 2023-08-17T17:15:50Z

 						Expect(err).To(Not(HaveOccurred()))
-						//omitting trailing new line char from response
-						Expect(string(catalogs[:len(catalogs)-1])).To(Equal(catalogKey.Name))
+						Expect(string(catalogs)).To(Equal(fmt.Sprintf(httpResponse, fmt.Sprintf("\n<a href=\"%s\">%s</a>", catalogKey.Name+".json", catalogKey.Name+".json"))))


The RFC does not mention anything about serving index.html pages for directory listings. I would expect this request to be a 404 Not Found response.

Follow-up for clarity. I would expect tests for (pseudocode):

http.Get("<host>/catalogs/<catalogName>/all.json => response code 200 and valid FBC

http.Get("<host>/catalogs/<catalogName>/non-exist.json => response code 404

http.Get("<host>/catalogs/<catalogName>/ => response code 404

http.Get("<host>/catalogs/ => response code 404

I am not sure this file is the right place for these tests. This file contains tests for what Reconcile does. In the case of catalog contents, Reconcile only stores the contents into the storage directory. So I would expect tests here to just assert that the expected files exist in the storage directory after Reconcile is called.

I think we should have unit tests in pkg/catalogserver that use httptest package.

And I think we should have an e2e test that actually queries the pod endpoint after a Catalog says it is unpacked.

Steve pointed out to me that we can probably get by without the Reconcile()-specific test that would assert on filesystem contents if we have an e2e. That would make it easier to make internal changes and still be assured they don't break users.

@joelanford the catalogserver just starts a server. If the tests are moved to pkg/catalogserver then you'd have to simulate a Storage, at which point you're just duplicating work being done in the reconciler tests.

Steve pointed out to me that we can probably get by without the Reconcile()-specific test that would assert on filesystem contents if we have an e2e. That would make it easier to make internal changes and still be assured they don't break users.

I don't understand the difference between the e2e test being suggested here vs the Reconcile test. In fact the whole idea of this test set up was to avoid adding e2e tests unless we absolutely need them wasn't it?

This is different than what http.FileServer does though. FileServer just serves up the entire directory, and there's not much configurability when it comes to access points.

I think that this is partially true. The http.FileServer is less flexible than rolling our own with http.ServeFile, but I think we can achieve what @joelanford suggested by doing something similar to rukpak's implementation that creates a filesystem wrapper that forces 404 not found on requests that would return a directory list

Okay, we're onto something now.....

Again, rukpak is exposing an arbitrary list of arbitrary artifacts stored in the directory. Using that implementation to expose just FBC in our case is definitely overkill. The argument put forward for using http.FileServer was also something along the lines of "less configuration/less code required for standing up the server", but using that implementation to expose all.json is looking more like a long winded way of shoehorning everything in for http.FileServer.

I don't know that I personally feel strongly about @joelanford requirements, but in any case the FilesOnlyFilesystrem is twenty lines of code you can import and re-use. You do not need to manage everything else that http.FileServer does for you.

The requirements are from the RFC and I do agree with those requirements since they adhere to design best practices. We all +1-ed the requirements in the RFC too.

FileServer is a wrapper around filehandler, and ServerHTTP ultimately calls serveFile.

ServeFile just takes the file name we pass as a parameter, splits the name into "dir name" and "file name", and uses those variables to call the exact same serveFile.

What is this "everything else" you're alluding to?

I'm confused by rukpak's code. Could someone point me to where ServeHTTP is actually used? How are the files really exposed? I searched in the project and the only caller of this function is a test function

Neither here nor there on the requirements. Importing the 20LoC or copy-pasting it is low cost.

As for what "everything else" is - I would recommend going a bit deeper there. From the management of which files are and are not served, to how how paths are cleaned and block traversal attacks, etc.

Signed-off-by: Anik <anikbhattacharya93@gmail.com>

anik120 · 2023-08-24T20:11:09Z

@stevekuznetsov @operator-framework/catalogd-maintainers closing this PR in favor of splitting it up into smaller PRs so that it's easier to review and get things moving along. Look out for a series of PRs that'll be based off of this PR, but with some additional stuff to address remaining open comments for the work being done in this PR. First one starts with #144 to store the FBC in a local Dir.
Second PR will be to introduce an http Handler.
A third PR will tie the first two PRs (and will contain integration tests)

ncdc reviewed Aug 3, 2023

View reviewed changes

Comment thread pkg/util/fs.go Outdated

ncdc reviewed Aug 3, 2023

View reviewed changes

Comment thread pkg/storage/localdir.go Outdated

anik120 force-pushed the storage-implementation branch from afffbe6 to 1c21eb3 Compare August 3, 2023 19:19

ncdc reviewed Aug 3, 2023

View reviewed changes

anik120 changed the title ~~Implement rukpak storage interface~~ WIP: Implement rukpak storage interface Aug 3, 2023

openshift-ci Bot added the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Aug 3, 2023

anik120 force-pushed the storage-implementation branch 4 times, most recently from 046fd14 to a9c9482 Compare August 11, 2023 16:26

anik120 force-pushed the storage-implementation branch 6 times, most recently from 7610d7c to ad474b3 Compare August 12, 2023 20:25

anik120 changed the title ~~WIP: Implement rukpak storage interface~~ WIP: Store(local dir) and Serve(http server) FBC Aug 12, 2023

anik120 force-pushed the storage-implementation branch from ad474b3 to baeb9be Compare August 14, 2023 13:45

openshift-merge-robot added the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Aug 14, 2023

anik120 force-pushed the storage-implementation branch 2 times, most recently from 3496d8a to 4c0f6b6 Compare August 14, 2023 13:47

openshift-merge-robot removed the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Aug 14, 2023

stevekuznetsov reviewed Aug 14, 2023

View reviewed changes

Comment thread pkg/catalogserver/server.go Outdated

Comment thread pkg/controllers/core/catalog_controller_test.go Outdated

Comment thread cmd/manager/main.go Outdated

Comment thread cmd/manager/main.go Outdated

Comment thread pkg/storage/storage.go Outdated

anik120 force-pushed the storage-implementation branch 2 times, most recently from c512ed8 to bddb98c Compare August 14, 2023 21:25

anik120 changed the title ~~WIP: Store(local dir) and Serve(http server) FBC~~ Store(local dir) and Serve(http server) FBC Aug 14, 2023

joelanford reviewed Aug 15, 2023

View reviewed changes

Comment thread cmd/manager/main.go Outdated

joelanford reviewed Aug 15, 2023

View reviewed changes

Comment thread cmd/manager/main.go Outdated

joelanford reviewed Aug 15, 2023

View reviewed changes

Comment thread pkg/controllers/core/catalog_controller.go Outdated

joelanford reviewed Aug 15, 2023

View reviewed changes

Comment thread pkg/storage/storage.go Outdated

everettraven reviewed Aug 15, 2023

View reviewed changes

ncdc reviewed Aug 15, 2023

View reviewed changes

everettraven mentioned this pull request Aug 15, 2023

WIP: (feature): Update controller to use Storage interface #129

Closed

3 tasks

stevekuznetsov reviewed Aug 16, 2023

View reviewed changes

anik120 requested a review from a team as a code owner August 16, 2023 16:21

openshift-merge-robot added the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Aug 16, 2023

anik120 added 6 commits August 16, 2023 12:23

Implement storage for fbc files

5d7eea7

obsoletes operator-framework#113 Signed-off-by: Anik <anikbhattacharya93@gmail.com>

Serve stored FBC content

8afa23a

Signed-off-by: Anik <anikbhattacharya93@gmail.com>

put storage and server behind featuregate

9a2a909

Signed-off-by: Anik <anikbhattacharya93@gmail.com>

add unit tests

e6b10ff

Signed-off-by: Anik <anikbhattacharya93@gmail.com>

start server in goroutine and shutdown gracefully

9d60f83

Signed-off-by: Anik <anikbhattacharya93@gmail.com>

store catalogs in /var/cache/catalogs dir

4e304f9

Signed-off-by: Anik <anikbhattacharya93@gmail.com>

anik120 force-pushed the storage-implementation branch from 6e71f82 to 4e304f9 Compare August 16, 2023 16:34

openshift-merge-robot removed the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Aug 16, 2023

anik120 added 2 commits August 16, 2023 16:13

use httptest package to test server

d9069e9

Signed-off-by: Anik <anikbhattacharya93@gmail.com>

switch to using errgroup

50842c6

Signed-off-by: Anik <anikbhattacharya93@gmail.com>

anik120 mentioned this pull request Aug 16, 2023

Follow up to-do list for #135 #142

Closed

everettraven reviewed Aug 16, 2023

View reviewed changes

switch to http.FileServer

2d64104

Signed-off-by: Anik <anikbhattacharya93@gmail.com>

anik120 force-pushed the storage-implementation branch from eee8ebd to 2d64104 Compare August 17, 2023 16:06

everettraven approved these changes Aug 17, 2023

View reviewed changes

stevekuznetsov reviewed Aug 17, 2023

View reviewed changes

joelanford suggested changes Aug 17, 2023

View reviewed changes

minor refractors

e59cdff

Signed-off-by: Anik <anikbhattacharya93@gmail.com>

anik120 closed this Aug 24, 2023

	fs := http.FileServer(http.FS(os.DirFS(dir)))
	return fs
	return http.FileServer(http.FS(os.DirFS(dir)))

Conversation

anik120 commented Aug 3, 2023

Uh oh!

Uh oh!

codecov Bot commented Aug 3, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

anik120 commented Aug 3, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

ncdc commented Aug 3, 2023

Uh oh!

anik120 commented Aug 3, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

stevekuznetsov left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

everettraven left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

codecov Bot commented Aug 3, 2023 •

edited

Loading

anik120 commented Aug 3, 2023 •

edited

Loading

anik120 commented Aug 3, 2023 •

edited

Loading

anik120 Aug 17, 2023 •

edited

Loading