diff --git a/docs/design/SYSTEM_DESIGN.md b/docs/design/SYSTEM_DESIGN.md index dfc137f944..46bdf54723 100644 --- a/docs/design/SYSTEM_DESIGN.md +++ b/docs/design/SYSTEM_DESIGN.md @@ -40,6 +40,9 @@ defined as Kubernetes resources that provide the following services: ### Components +Envoy Gateway is made up of several components that communicate in-process; how this communication happens is described +in [watching.md][]. + #### Provider A Provider is an infrastructure component that Envoy Gateway calls to establish its runtime configuration, resolve services, persist data, etc. Kubernetes and File are the only supported providers. However, other providers can be added @@ -141,3 +144,4 @@ The draft for this document is [here][draft_design]. [be]: https://gateway-api.sigs.k8s.io/v1alpha2/references/spec/#gateway.networking.k8s.io/v1alpha2.BackendObjectReference [svc]: https://kubernetes.io/docs/concepts/services-networking/service/ [issue_95]: https://github.com/envoyproxy/gateway/pull/95 +[watching.md]: ./watching.md diff --git a/docs/design/watching.md b/docs/design/watching.md new file mode 100644 index 0000000000..b76ba57690 --- /dev/null +++ b/docs/design/watching.md @@ -0,0 +1,112 @@ +## Watching the results of other components + +Envoy Gateway is made up of several components that communicate in-process. Some of them (namely providers) watch +external resources, and "publish" what they see for other components to consume; others watch what another publishes and +act on it (such as the resource translator watches what the providers publish, and then publishes its own results that +are watched by another component). Some of these internally published results are consumed by multiple components. + +To facilitate this communication use the [watchable][] library. The `watchable.Map` type is very similar to the +standard library's `sync.Map` type, but supports a `.Subscribe` (and `.SubscribeSubset`) method that promotes a pub/sub +pattern. + +### pub + +Many of the things we communicate around are naturally named, either by a bare "name" string or by a "name"/"namespace" +tuple. And because `watchable.Map` is typed, it makes sense to have one map for each type of thing (very similar to if +we were using native Go `map`s). For example, a struct that might be written to by the Kubernetes provider, and read by +the IR translator: + + ```go + type ResourceTable struct { + // gateway classes are cluster-scoped; no namespace + GatewayClasses watchable.Map[string, *gwapiv1b1.GatewayClass] + + // gateways are namespace-scoped, so use a k8s.io/apimachinery/pkg/types.NamespacedName as the map key. + Gateways watchable.Map[types.NamespacedName, *gwapiv1b1.Gateway] + + HTTPRoutes watchable.Map[types.NamespacedName, *gwapiv1b1.HTTPRoute] + } + ``` + +The Kubernetes provider updates the table by calling `table.Thing.Store(name, val)` and `table.Thing.Delete(name)`; +updating a map key with a value that is deep-equal (usually `reflect.DeepEqual`, but you can implement your own `.Equal` +method) the current value is a no-op; it won't trigger an event for subscribers. This is handy so that the publisher +doesn't have as much state to keep track of; it doesn't need to know "did I already publish this thing", it can just +`.Store` its data and `watchable` will do the right thing. + +### sub + +Meanwhile, the translator and other interested components subscribe to it with `table.Thing.Subscribe` (or +`table.Thing.SubscrubeSubset` if they only care about a few "Thing"s). So the translator goroutine might look like: + + ```go + func(ctx context.Context) error { + for snapshot := range k8sTable.HTTPRoutes.Subscribe(ctx) { + fullState := irInput{ + GatewayClasses: k8sTable.GatewayClasses.LoadAll(), + Gateways: k8sTable.Gateways.LoadAll(), + HTTPRoutes: snapshot.State, + } + translate(irInput) + } + } + ``` + +Or, to watch multiple maps in the same loop: + + ```go + func worker(ctx context.Context) error { + classCh := k8sTable.GatewayClasses.Subscribe(ctx) + gwCh := k8sTable.Gateways.Subscribe(ctx) + routeCh := k8sTable.HTTPRoutes.Subscribe(ctx) + for ctx.Err() == nil { + var arg irInput + select { + case snapshot := <-classCh: + arg.GatewayClasses = snapshot.State + case snapshot := <-gwCh: + arg.Gateways = snapshot.State + case snapshot := <-routeCh: + arg.Routes = snapshot.State + } + if arg.GateWayClasses == nil { + arg.GatewayClasses = k8sTable.GateWayClasses.LoadAll() + } + if arg.GateWays == nil { + arg.Gateways = k8sTable.GateWays.LoadAll() + } + if arg.HTTPRoutes == nil { + arg.HTTPRoutes = k8sTable.HTTPRoutes.LoadAll() + } + translate(irInput) + } + } + ``` + +From the updates it gets from `.Subscribe`, it can get a full view of the map being subscribed to via `snapshot.State`; +but it must read the other maps explicitly. Like `sync.Map`, `watchable.Map`s are thread-safe; while `.Subscribe` is a +handy way to know when to run, `.Load` and friends can be used without subscribing. + +There can be any number of subscribers. For that matter, there can be any number of publishers `.Store`ing things, but +it's probably wise to just have one publisher for each map. + +The channel returned from `.Subscribe` is immediately readable with a snapshot of the map as it existed when +`.Subscribe` was called; after that initial read it becomes readable again whenever `.Store` or `.Delete` mutates the +map. If multiple mutations happen between reads, they are coalesced in to one snapshot to be read; the `snapshot.State` +is the most-recent full state, and `snapshot.Updates` is a listing of each of the mutations that cause this snapshot to +be different than the last-read one. This way subscribers don't need to worry about a backlog accumulating if they +can't keep up with the rate of changes from the publisher. + +### other notes + +The common pattern will likely be that the entrypoint that launches the goroutines for each component instantiates the +map, and passes them to the appropriate publishers and subscribers; same as if they were communicating via a dumb +`chan`. + +A limitation of `watchable.Map` is that in order to ensure safety between goroutines, it does require that value types +be deep-copiable; either by having a `DeepCopy` method, being a `proto.Message`, or by containing no reference types and +so can be deep-copied by naive assignment. Fortunately, we're using `controller-gen` anyway, and `controller-gen` can +generate `DeepCopy` methods for us: just stick a `// +k8s:deepcopy-gen=true` on the types that you want it to generate +methods for. + +[watchable]: https://pkg.go.dev/github.com/telepresenceio/watchable diff --git a/go.mod b/go.mod index 3bc7ad4875..6c4d353e99 100644 --- a/go.mod +++ b/go.mod @@ -5,11 +5,12 @@ go 1.18 require ( github.com/envoyproxy/go-control-plane v0.10.3 github.com/go-logr/zapr v1.2.0 - github.com/google/go-cmp v0.5.7 + github.com/google/go-cmp v0.5.8 github.com/spf13/cobra v1.4.0 - github.com/stretchr/testify v1.7.1 + github.com/stretchr/testify v1.8.0 + github.com/telepresenceio/watchable v0.0.0-20220726211108-9bb86f92afa7 go.uber.org/zap v1.19.1 - google.golang.org/grpc v1.45.0 + google.golang.org/grpc v1.46.2 k8s.io/api v0.24.2 k8s.io/apimachinery v0.24.2 k8s.io/client-go v0.24.2 @@ -65,18 +66,18 @@ require ( github.com/spf13/pflag v1.0.5 // indirect go.uber.org/atomic v1.7.0 // indirect go.uber.org/multierr v1.6.0 // indirect - golang.org/x/net v0.0.0-20220127200216-cd36cc0744dd // indirect + golang.org/x/net v0.0.0-20220526153639-5463443f8c37 // indirect golang.org/x/oauth2 v0.0.0-20211104180415-d3ed0bb246c8 // indirect - golang.org/x/sys v0.0.0-20220209214540-3681064d5158 // indirect + golang.org/x/sys v0.0.0-20220520151302-bc2c85ada10a // indirect golang.org/x/term v0.0.0-20210927222741-03fcf44c2211 // indirect - golang.org/x/text v0.3.7 // indirect + golang.org/x/text v0.3.8-0.20211105212822-18b340fc7af2 // indirect golang.org/x/time v0.0.0-20220210224613-90d013bbcef8 // indirect gomodules.xyz/jsonpatch/v2 v2.2.0 // indirect google.golang.org/appengine v1.6.7 // indirect - google.golang.org/genproto v0.0.0-20220329172620-7be39ac1afc7 // indirect + google.golang.org/genproto v0.0.0-20220505152158-f39f71e6c8f3 // indirect gopkg.in/inf.v0 v0.9.1 // indirect gopkg.in/yaml.v2 v2.4.0 // indirect - gopkg.in/yaml.v3 v3.0.0-20210107192922-496545a6307b // indirect + gopkg.in/yaml.v3 v3.0.1 // indirect k8s.io/apiextensions-apiserver v0.24.2 // indirect k8s.io/component-base v0.24.2 // indirect k8s.io/klog/v2 v2.60.1 // indirect diff --git a/go.sum b/go.sum index 9bf546d211..0978f96825 100644 --- a/go.sum +++ b/go.sum @@ -96,6 +96,7 @@ github.com/cncf/udpa/go v0.0.0-20210930031921-04548b0d99d4/go.mod h1:6pvJx4me5XP github.com/cncf/xds/go v0.0.0-20210312221358-fbca930ec8ed/go.mod h1:eXthEFrGJvWHgFFCl3hGmgk+/aYT6PnTQLykKQRLhEs= github.com/cncf/xds/go v0.0.0-20210805033703-aa0b78936158/go.mod h1:eXthEFrGJvWHgFFCl3hGmgk+/aYT6PnTQLykKQRLhEs= github.com/cncf/xds/go v0.0.0-20210922020428-25de7278fc84/go.mod h1:eXthEFrGJvWHgFFCl3hGmgk+/aYT6PnTQLykKQRLhEs= +github.com/cncf/xds/go v0.0.0-20211001041855-01bcc9b48dfe/go.mod h1:eXthEFrGJvWHgFFCl3hGmgk+/aYT6PnTQLykKQRLhEs= github.com/cncf/xds/go v0.0.0-20211011173535-cb28da3451f1/go.mod h1:eXthEFrGJvWHgFFCl3hGmgk+/aYT6PnTQLykKQRLhEs= github.com/cncf/xds/go v0.0.0-20220314180256-7f1daf1720fc h1:PYXxkRUBGUMa5xgMVMDl62vEklZvKpVaxQeN9ie7Hfk= github.com/cncf/xds/go v0.0.0-20220314180256-7f1daf1720fc/go.mod h1:eXthEFrGJvWHgFFCl3hGmgk+/aYT6PnTQLykKQRLhEs= @@ -113,6 +114,7 @@ github.com/cpuguy83/go-md2man/v2 v2.0.0/go.mod h1:maD7wRr/U5Z6m/iR4s+kqSMx2CaBsr github.com/cpuguy83/go-md2man/v2 v2.0.1/go.mod h1:tgQtvFlXSQOSOSIRvRPT7W67SCa46tRHOmNcaadrF8o= github.com/creack/pty v1.1.9/go.mod h1:oKZEueFk5CKHvIhNR5MUki03XCEU+Q6VDXinZuGJ33E= github.com/creack/pty v1.1.11/go.mod h1:oKZEueFk5CKHvIhNR5MUki03XCEU+Q6VDXinZuGJ33E= +github.com/datawire/dlib v1.3.0 h1:KkmyXU1kwm3oPBk1ypR70YbcOlEXWzEbx5RE0iRXTGk= github.com/davecgh/go-spew v1.1.0/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38= github.com/davecgh/go-spew v1.1.1 h1:vj9j/u1bqnvCEfJOwUhtlOARqs3+rkHYY13jYWTU97c= github.com/davecgh/go-spew v1.1.1/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38= @@ -132,6 +134,7 @@ github.com/envoyproxy/go-control-plane v0.9.9-0.20201210154907-fd9021fe5dad/go.m github.com/envoyproxy/go-control-plane v0.9.9-0.20210217033140-668b12f5399d/go.mod h1:cXg6YxExXjJnVBQHBLXeUAgxn2UodCpnH306RInaBQk= github.com/envoyproxy/go-control-plane v0.9.9-0.20210512163311-63b5d3c536b0/go.mod h1:hliV/p42l8fGbc6Y9bQ70uLwIvmJyVE5k4iMKlh8wCQ= github.com/envoyproxy/go-control-plane v0.9.10-0.20210907150352-cf90f659a021/go.mod h1:AFq3mo9L8Lqqiid3OhADV3RfLJnjiw63cSpi+fDTRC0= +github.com/envoyproxy/go-control-plane v0.10.2-0.20220325020618-49ff273808a1/go.mod h1:KJwIaB5Mv44NWtYuAOFCVOjcI94vtpEz2JU/D2v6IjE= github.com/envoyproxy/go-control-plane v0.10.3 h1:xdCVXxEe0Y3FQith+0cj2irwZudqGYvecuLB1HtdexY= github.com/envoyproxy/go-control-plane v0.10.3/go.mod h1:fJJn/j26vwOu972OllsvAgJJM//w9BV6Fxbg2LuVd34= github.com/envoyproxy/protoc-gen-validate v0.1.0/go.mod h1:iSmxcyjqTsJpI2R4NaDN7+kN2VEUnK/pcBlmesArF7c= @@ -235,8 +238,9 @@ github.com/google/go-cmp v0.5.3/go.mod h1:v8dTdLbMG2kIc/vJvl+f65V22dbkXbowE6jgT/ github.com/google/go-cmp v0.5.4/go.mod h1:v8dTdLbMG2kIc/vJvl+f65V22dbkXbowE6jgT/gNBxE= github.com/google/go-cmp v0.5.5/go.mod h1:v8dTdLbMG2kIc/vJvl+f65V22dbkXbowE6jgT/gNBxE= github.com/google/go-cmp v0.5.6/go.mod h1:v8dTdLbMG2kIc/vJvl+f65V22dbkXbowE6jgT/gNBxE= -github.com/google/go-cmp v0.5.7 h1:81/ik6ipDQS2aGcBfIN5dHDB36BwrStyeAQquSYCV4o= github.com/google/go-cmp v0.5.7/go.mod h1:n+brtR0CgQNWTVd5ZUFpTBC8YFBDLK/h/bpaJ8/DtOE= +github.com/google/go-cmp v0.5.8 h1:e6P7q2lk1O+qJJb4BtCQXlK8vWEO8V1ZeuEdJNOqZyg= +github.com/google/go-cmp v0.5.8/go.mod h1:17dUlkBOakJ0+DkrSSNjCkIjxS6bF9zb3elmeNGIjoY= github.com/google/gofuzz v1.0.0/go.mod h1:dBl0BpW6vV/+mYPU4Po3pmUjxk6FQPldtuIdl/M65Eg= github.com/google/gofuzz v1.1.0 h1:Hsa8mG0dQ46ij8Sl2AYJDUv1oA9/d6Vk+3LG99Oe02g= github.com/google/gofuzz v1.1.0/go.mod h1:dBl0BpW6vV/+mYPU4Po3pmUjxk6FQPldtuIdl/M65Eg= @@ -431,6 +435,7 @@ github.com/sirupsen/logrus v1.2.0/go.mod h1:LxeOpSwHxABJmUn/MG1IvRgCAasNZTLOkJPx github.com/sirupsen/logrus v1.4.2/go.mod h1:tLMulIdttU9McNUspp0xgXVQah82FyeX6MwdIuYE2rE= github.com/sirupsen/logrus v1.6.0/go.mod h1:7uNnSEd1DgxDLC74fIahvMZmmYsHGZGEOFrfsX/uA88= github.com/sirupsen/logrus v1.7.0/go.mod h1:yWOB1SBYBC5VeMP7gHvWumXLIWorT60ONWic61uBYv0= +github.com/sirupsen/logrus v1.8.1 h1:dJKuHgqk1NNQlqoA6BTlM1Wf9DOH3NBjQyu0h9+AZZE= github.com/sirupsen/logrus v1.8.1/go.mod h1:yWOB1SBYBC5VeMP7gHvWumXLIWorT60ONWic61uBYv0= github.com/smartystreets/assertions v0.0.0-20180927180507-b2de0cb4f26d/go.mod h1:OnSkiWE9lh6wB0YB77sQom3nweQdgAjqCqsofrRNTgc= github.com/smartystreets/goconvey v1.6.4/go.mod h1:syvi0/a8iFYH4r/RixwvyeAJjdLS9QV7WQ/tjFTllLA= @@ -453,15 +458,20 @@ github.com/spf13/viper v1.7.0/go.mod h1:8WkrPz2fc9jxqZNCJI/76HCieCp4Q8HaLFoCha5q github.com/stoewer/go-strcase v1.2.0/go.mod h1:IBiWB2sKIp3wVVQ3Y035++gc+knqhUQag1KpM8ahLw8= github.com/stretchr/objx v0.1.0/go.mod h1:HFkY916IF+rwdDfMAkV7OtwuqBVzrE8GR6GFx+wExME= github.com/stretchr/objx v0.1.1/go.mod h1:HFkY916IF+rwdDfMAkV7OtwuqBVzrE8GR6GFx+wExME= +github.com/stretchr/objx v0.4.0/go.mod h1:YvHI0jy2hoMjB+UWwv71VJQ9isScKT/TqJzVSSt89Yw= github.com/stretchr/testify v1.2.2/go.mod h1:a8OnRcib4nhh0OaRAV+Yts87kKdq0PP7pXfy6kDkUVs= github.com/stretchr/testify v1.3.0/go.mod h1:M5WIy9Dh21IEIfnGCwXGc5bZfKNJtfHm1UVUgZn+9EI= github.com/stretchr/testify v1.4.0/go.mod h1:j7eGeouHqKxXV5pUuKE4zz7dFj8WfuZ+81PSLYec5m4= github.com/stretchr/testify v1.5.1/go.mod h1:5W2xD1RspED5o8YsWQXVCued0rvSQ+mT+I5cxcmMvtA= github.com/stretchr/testify v1.6.1/go.mod h1:6Fq8oRcR53rry900zMqJjRRixrwX3KX962/h/Wwjteg= github.com/stretchr/testify v1.7.0/go.mod h1:6Fq8oRcR53rry900zMqJjRRixrwX3KX962/h/Wwjteg= -github.com/stretchr/testify v1.7.1 h1:5TQK59W5E3v0r2duFAb7P95B6hEeOyEnHRa8MjYSMTY= github.com/stretchr/testify v1.7.1/go.mod h1:6Fq8oRcR53rry900zMqJjRRixrwX3KX962/h/Wwjteg= +github.com/stretchr/testify v1.8.0 h1:pSgiaMZlXftHpm5L7V1+rVB+AZJydKsMxsQBIJw4PKk= +github.com/stretchr/testify v1.8.0/go.mod h1:yNjHg4UonilssWZ8iaSj1OCr/vHnekPRkoO+kdMU+MU= github.com/subosito/gotenv v1.2.0/go.mod h1:N0PQaV/YGNqwC0u51sEeR/aUtSLEXKX9iv69rRypqCw= +github.com/telepresenceio/telepresence/rpc/v2 v2.6.8 h1:q5V85LBT9bA/c4YPa/kMvJGyKZDgBPJTftlAMqJx7j4= +github.com/telepresenceio/watchable v0.0.0-20220726211108-9bb86f92afa7 h1:GMw3nEaOVyi+tNiGko5kAeRtoiEIpXNHmISyZ7fpw14= +github.com/telepresenceio/watchable v0.0.0-20220726211108-9bb86f92afa7/go.mod h1:ihJ97e2gsd8GuzFF/I3B1qcik3XZLpXjumQifXi8Slg= github.com/tetratelabs/multierror v1.1.0 h1:cKmV/Pbf42K5wp8glxa2YIausbxIraPN8fzru9Pn1Cg= github.com/tetratelabs/multierror v1.1.0/go.mod h1:kH3SzI/z+FwEbV9bxQDx4GiIgE2djuyb8wiB2DaUBnY= github.com/tmc/grpc-websocket-proxy v0.0.0-20190109142713-0ad062ec5ee5/go.mod h1:ncp9v5uamzpCO7NfCPTXjqaC+bZgJeR0sMTm6dMHP7U= @@ -615,8 +625,9 @@ golang.org/x/net v0.0.0-20210813160813-60bc85c4be6d/go.mod h1:9nx3DQGgdP8bBQD5qx golang.org/x/net v0.0.0-20210825183410-e898025ed96a/go.mod h1:9nx3DQGgdP8bBQD5qxJ1jj9UTztislL4KSBs9R2vV5Y= golang.org/x/net v0.0.0-20211015210444-4f30a5c0130f/go.mod h1:9nx3DQGgdP8bBQD5qxJ1jj9UTztislL4KSBs9R2vV5Y= golang.org/x/net v0.0.0-20211112202133-69e39bad7dc2/go.mod h1:9nx3DQGgdP8bBQD5qxJ1jj9UTztislL4KSBs9R2vV5Y= -golang.org/x/net v0.0.0-20220127200216-cd36cc0744dd h1:O7DYs+zxREGLKzKoMQrtrEacpb0ZVXA5rIwylE2Xchk= golang.org/x/net v0.0.0-20220127200216-cd36cc0744dd/go.mod h1:CfG3xpIq0wQ8r1q4Su4UZFWDARRcnwPjda9FqA0JpMk= +golang.org/x/net v0.0.0-20220526153639-5463443f8c37 h1:lUkvobShwKsOesNfWWlCS5q7fnbG1MEliIzwu886fn8= +golang.org/x/net v0.0.0-20220526153639-5463443f8c37/go.mod h1:XRhObCWvk6IyKnWLug+ECip1KBveYUHfp+8e9klMJ9c= golang.org/x/oauth2 v0.0.0-20180821212333-d2e6202438be/go.mod h1:N/0e6XlmueqKjAGxoOufVs8QHGRruUQn6yWY3a++T0U= golang.org/x/oauth2 v0.0.0-20190226205417-e64efc72b421/go.mod h1:gOpvHmFTYa4IltrdGE7lF6nIHvwfUNPOp7c8zoXwtLw= golang.org/x/oauth2 v0.0.0-20190604053449-0f29369cfe45/go.mod h1:gOpvHmFTYa4IltrdGE7lF6nIHvwfUNPOp7c8zoXwtLw= @@ -707,8 +718,9 @@ golang.org/x/sys v0.0.0-20210831042530-f4d43177bf5e/go.mod h1:oPkhp1MJrh7nUepCBc golang.org/x/sys v0.0.0-20211019181941-9d821ace8654/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg= golang.org/x/sys v0.0.0-20211216021012-1d35b9e2eb4e/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg= golang.org/x/sys v0.0.0-20220114195835-da31bd327af9/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg= -golang.org/x/sys v0.0.0-20220209214540-3681064d5158 h1:rm+CHSpPEEW2IsXUib1ThaHIjuBVZjxNgSKmBLFfD4c= golang.org/x/sys v0.0.0-20220209214540-3681064d5158/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg= +golang.org/x/sys v0.0.0-20220520151302-bc2c85ada10a h1:dGzPydgVsqGcTRVwiLJ1jVbufYwmzD3LfVPLKsKg+0k= +golang.org/x/sys v0.0.0-20220520151302-bc2c85ada10a/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg= golang.org/x/term v0.0.0-20201126162022-7de9c90e9dd1/go.mod h1:bj7SfCRtBDWHUb9snDiAeCFNEtKQo2Wmx5Cou7ajbmo= golang.org/x/term v0.0.0-20210927222741-03fcf44c2211 h1:JGgROgKl9N8DuW20oFS5gxc+lE67/N3FcwmBPMe7ArY= golang.org/x/term v0.0.0-20210927222741-03fcf44c2211/go.mod h1:jbD1KX2456YbFQfuXm/mYQcufACuNUgVhRMnK/tPxf8= @@ -720,8 +732,9 @@ golang.org/x/text v0.3.3/go.mod h1:5Zoc/QRtKVWzQhOtBMvqHzDpF6irO9z98xDceosuGiQ= golang.org/x/text v0.3.4/go.mod h1:5Zoc/QRtKVWzQhOtBMvqHzDpF6irO9z98xDceosuGiQ= golang.org/x/text v0.3.5/go.mod h1:5Zoc/QRtKVWzQhOtBMvqHzDpF6irO9z98xDceosuGiQ= golang.org/x/text v0.3.6/go.mod h1:5Zoc/QRtKVWzQhOtBMvqHzDpF6irO9z98xDceosuGiQ= -golang.org/x/text v0.3.7 h1:olpwvP2KacW1ZWvsR7uQhoyTYvKAupfQrRGBFM352Gk= golang.org/x/text v0.3.7/go.mod h1:u+2+/6zg+i71rQMx5EYifcz6MCKuco9NR6JIITiCfzQ= +golang.org/x/text v0.3.8-0.20211105212822-18b340fc7af2 h1:GLw7MR8AfAG2GmGcmVgObFOHXYypgGjnGno25RDwn3Y= +golang.org/x/text v0.3.8-0.20211105212822-18b340fc7af2/go.mod h1:EFNZuWvGYxIRUEX+K8UmCFwYmZjqcrnq15ZuVldZkZ0= golang.org/x/time v0.0.0-20181108054448-85acf8d2951c/go.mod h1:tRJNPiyCQ0inRvYxbN9jk5I+vvW/OXSQhTDSoE431IQ= golang.org/x/time v0.0.0-20190308202827-9d24e82272b4/go.mod h1:tRJNPiyCQ0inRvYxbN9jk5I+vvW/OXSQhTDSoE431IQ= golang.org/x/time v0.0.0-20191024005414-555d28b269f0/go.mod h1:tRJNPiyCQ0inRvYxbN9jk5I+vvW/OXSQhTDSoE431IQ= @@ -789,7 +802,6 @@ golang.org/x/tools v0.1.10-0.20220218145154-897bd77cd717/go.mod h1:Uh6Zz+xoGYZom golang.org/x/xerrors v0.0.0-20190717185122-a985d3407aa7/go.mod h1:I/5z698sn9Ka8TeJc9MKroUUfqBBauWjQqLJ2OPfmY0= golang.org/x/xerrors v0.0.0-20191011141410-1b5146add898/go.mod h1:I/5z698sn9Ka8TeJc9MKroUUfqBBauWjQqLJ2OPfmY0= golang.org/x/xerrors v0.0.0-20191204190536-9bdfabe68543/go.mod h1:I/5z698sn9Ka8TeJc9MKroUUfqBBauWjQqLJ2OPfmY0= -golang.org/x/xerrors v0.0.0-20200804184101-5ec99f83aff1 h1:go1bK/D/BFZV2I8cIQd1NKEZ+0owSTG1fDTci4IqFcE= golang.org/x/xerrors v0.0.0-20200804184101-5ec99f83aff1/go.mod h1:I/5z698sn9Ka8TeJc9MKroUUfqBBauWjQqLJ2OPfmY0= gomodules.xyz/jsonpatch/v2 v2.2.0 h1:4pT439QV83L+G9FkcCriY6EkpcK6r6bK+A5FBUMI7qY= gomodules.xyz/jsonpatch/v2 v2.2.0/go.mod h1:WXp+iVDkoLQqPudfQ9GBlwB2eZ5DKOnjQZCYdOS8GPY= @@ -869,8 +881,9 @@ google.golang.org/genproto v0.0.0-20210602131652-f16073e35f0c/go.mod h1:UODoCrxH google.golang.org/genproto v0.0.0-20210831024726-fe130286e0e2/go.mod h1:eFjDcFEctNawg4eG61bRv87N7iHBWyVhJu7u1kqDUXY= google.golang.org/genproto v0.0.0-20211118181313-81c1377c94b1/go.mod h1:5CzLGKJ67TSI2B9POpiiyGha0AjJvZIUgRMt1dSmuhc= google.golang.org/genproto v0.0.0-20220107163113-42d7afdf6368/go.mod h1:5CzLGKJ67TSI2B9POpiiyGha0AjJvZIUgRMt1dSmuhc= -google.golang.org/genproto v0.0.0-20220329172620-7be39ac1afc7 h1:HOL66YCI20JvN2hVk6o2YIp9i/3RvzVUz82PqNr7fXw= google.golang.org/genproto v0.0.0-20220329172620-7be39ac1afc7/go.mod h1:8w6bsBMX6yCPbAVTeqQHvzxW0EIFigd5lZyahWgyfDo= +google.golang.org/genproto v0.0.0-20220505152158-f39f71e6c8f3 h1:q1kiSVscqoDeqTF27eQ2NnLLDmqF0I373qQNXYMy0fo= +google.golang.org/genproto v0.0.0-20220505152158-f39f71e6c8f3/go.mod h1:RAyBrSAP7Fh3Nc84ghnVLDPuV51xc9agzmm4Ph6i0Q4= google.golang.org/grpc v1.19.0/go.mod h1:mqu4LbDTu4XGKhr4mRzUsmM4RtVoemTSY81AxZiDr8c= google.golang.org/grpc v1.20.1/go.mod h1:10oTOabMzJvdu6/UiuZezV6QK5dSlG84ov/aaiqXj38= google.golang.org/grpc v1.21.1/go.mod h1:oYelfM1adQP15Ek0mdvEgi9Df8B9CZIaU1084ijfRaM= @@ -894,8 +907,10 @@ google.golang.org/grpc v1.37.0/go.mod h1:NREThFqKR1f3iQ6oBuvc5LadQuXVGo9rkm5ZGrQ google.golang.org/grpc v1.38.0/go.mod h1:NREThFqKR1f3iQ6oBuvc5LadQuXVGo9rkm5ZGrQdJfM= google.golang.org/grpc v1.40.0/go.mod h1:ogyxbiOoUXAkP+4+xa6PZSE9DZgIHtSpzjDTB9KAK34= google.golang.org/grpc v1.42.0/go.mod h1:k+4IHHFw41K8+bbowsex27ge2rCb65oeWqe4jJ590SU= -google.golang.org/grpc v1.45.0 h1:NEpgUqV3Z+ZjkqMsxMg11IaDrXY4RY6CQukSGK0uI1M= google.golang.org/grpc v1.45.0/go.mod h1:lN7owxKUQEqMfSyQikvvk5tf/6zMPsrK+ONuO11+0rQ= +google.golang.org/grpc v1.46.0/go.mod h1:vN9eftEi1UMyUsIF80+uQXhHjbXYbm0uXoFCACuMGWk= +google.golang.org/grpc v1.46.2 h1:u+MLGgVf7vRdjEYZ8wDFhAVNmhkbJ5hmrA1LMWK1CAQ= +google.golang.org/grpc v1.46.2/go.mod h1:vN9eftEi1UMyUsIF80+uQXhHjbXYbm0uXoFCACuMGWk= google.golang.org/protobuf v0.0.0-20200109180630-ec00e32a8dfd/go.mod h1:DFci5gLYBciE7Vtevhsrf46CRTquxDuWsQurQQe4oz8= google.golang.org/protobuf v0.0.0-20200221191635-4d8936d0db64/go.mod h1:kwYJMbMJ01Woi6D6+Kah6886xMZcty6N08ah7+eCXa0= google.golang.org/protobuf v0.0.0-20200228230310-ab0ca4ff8a60/go.mod h1:cfTl7dwQJ+fmap5saPgwCLgHXTUD7jkjRqWcaiX5VyM= @@ -939,8 +954,9 @@ gopkg.in/yaml.v2 v2.4.0 h1:D8xgwECY7CYvx+Y2n4sBz93Jn9JRvxdiyyo8CTfuKaY= gopkg.in/yaml.v2 v2.4.0/go.mod h1:RDklbk79AGWmwhnvt/jBztapEOGDOx6ZbXqjP6csGnQ= gopkg.in/yaml.v3 v3.0.0-20200313102051-9f266ea9e77c/go.mod h1:K4uyk7z7BCEPqu6E+C64Yfv1cQ7kz7rIZviUmN+EgEM= gopkg.in/yaml.v3 v3.0.0-20200615113413-eeeca48fe776/go.mod h1:K4uyk7z7BCEPqu6E+C64Yfv1cQ7kz7rIZviUmN+EgEM= -gopkg.in/yaml.v3 v3.0.0-20210107192922-496545a6307b h1:h8qDotaEPuJATrMmW04NCwg7v22aHH28wwpauUhK9Oo= gopkg.in/yaml.v3 v3.0.0-20210107192922-496545a6307b/go.mod h1:K4uyk7z7BCEPqu6E+C64Yfv1cQ7kz7rIZviUmN+EgEM= +gopkg.in/yaml.v3 v3.0.1 h1:fxVm/GzAzEWqLHuvctI91KS9hhNmmWOoWu0XTYJS7CA= +gopkg.in/yaml.v3 v3.0.1/go.mod h1:K4uyk7z7BCEPqu6E+C64Yfv1cQ7kz7rIZviUmN+EgEM= gotest.tools/v3 v3.0.2/go.mod h1:3SzNCllyD9/Y+b5r9JIKQ474KzkZyqLqEfYqMsX94Bk= gotest.tools/v3 v3.0.3/go.mod h1:Z7Lb0S5l+klDB31fvDQX8ss/FlKDxtlFlw3Oa8Ymbl8= honnef.co/go/tools v0.0.0-20190102054323-c2f93a96b099/go.mod h1:rf3lG4BRIbNafJWhAfAdb/ePZxsR/4RtNHQocxwk9r4= diff --git a/internal/cmd/server.go b/internal/cmd/server.go index 846e626944..67c99a942e 100644 --- a/internal/cmd/server.go +++ b/internal/cmd/server.go @@ -3,10 +3,10 @@ package cmd import ( "os" - "github.com/envoyproxy/gateway/api/config/v1alpha1" - "github.com/envoyproxy/gateway/internal/envoygateway/config" "github.com/spf13/cobra" + "github.com/envoyproxy/gateway/api/config/v1alpha1" + "github.com/envoyproxy/gateway/internal/envoygateway/config" "github.com/envoyproxy/gateway/internal/provider" ) @@ -56,8 +56,12 @@ func server() error { cfg.EnvoyGateway = eg } - if err := provider.Start(cfg); err != nil { + k8sTable := new(provider.ResourceTable) + if err := provider.Start(cfg, k8sTable); err != nil { return err } + // TODO: while the provider.Start goroutine writes to the k8sTable, a (not-yet-existent) + // translator goroutine will read from it. + return nil } diff --git a/internal/provider/kubernetes/gateway.go b/internal/provider/kubernetes/gateway.go index 0b25ad8cbc..bce90bd712 100644 --- a/internal/provider/kubernetes/gateway.go +++ b/internal/provider/kubernetes/gateway.go @@ -6,6 +6,7 @@ package kubernetes import ( "context" "fmt" + "sync" "github.com/go-logr/logr" metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" @@ -27,16 +28,21 @@ type gatewayReconciler struct { // classController is the configured gatewayclass controller name. classController gwapiv1b1.GatewayController log logr.Logger + + initializeOnce sync.Once + resourceTable *ResourceTable } // newGatewayController creates a gateway controller. The controller will watch for // Gateway objects across all namespaces and reconcile those that match the configured // gatewayclass controller name. -func newGatewayController(mgr manager.Manager, cfg *config.Server) error { +func newGatewayController(mgr manager.Manager, cfg *config.Server, resourceTable *ResourceTable) error { + resourceTable.Initialized.Add(1) r := &gatewayReconciler{ client: mgr.GetClient(), classController: gwapiv1b1.GatewayController(cfg.EnvoyGateway.Gateway.ControllerName), log: cfg.Logger, + resourceTable: resourceTable, } c, err := controller.New("gateway", mgr, controller.Options{Reconciler: r}) @@ -97,8 +103,9 @@ func (r *gatewayReconciler) Reconcile(ctx context.Context, request reconcile.Req if acceptedClass == nil { r.log.Info("No accepted gatewayclass found for gateway", "namespace", request.Namespace, "name", request.Name) - // TODO: Delete gateway from the IR. - // xref: https://github.com/envoyproxy/gateway/issues/38 + for namespacedName := range r.resourceTable.Gateways.LoadAll() { + r.resourceTable.Gateways.Delete(namespacedName) + } return reconcile.Result{}, nil } @@ -106,19 +113,29 @@ func (r *gatewayReconciler) Reconcile(ctx context.Context, request reconcile.Req if err := r.client.List(ctx, allGateways); err != nil { return reconcile.Result{}, fmt.Errorf("error listing gateways") } + // Get all the Gateways for the Accepted=true GatewayClass. acceptedGateways := gatewaysOfClass(acceptedClass, allGateways) if len(acceptedGateways) == 0 { r.log.Info("No gateways found for accepted gatewayclass") - // TODO: Delete gateway from the IR. - // xref: https://github.com/envoyproxy/gateway/issues/34 - // xref: https://github.com/envoyproxy/gateway/issues/38 - return reconcile.Result{}, nil + } + found := false + for i := range acceptedGateways { + key := types.NamespacedName{ + Name: acceptedGateways[i].GetName(), + Namespace: acceptedGateways[i].GetNamespace(), + } + r.resourceTable.Gateways.Store(key, &acceptedGateways[i]) + if key == request.NamespacedName { + found = true + } + } + if !found { + r.resourceTable.Gateways.Delete(request.NamespacedName) } - // TODO: Add gateway to the IR. - // xref: https://github.com/envoyproxy/gateway/issues/34 - // xref: https://github.com/envoyproxy/gateway/issues/38 + // Once we've processed `allGateways`, record that we've fully initialized. + defer r.initializeOnce.Do(r.resourceTable.Initialized.Done) return reconcile.Result{}, nil } diff --git a/internal/provider/kubernetes/gatewayclass.go b/internal/provider/kubernetes/gatewayclass.go index e31490728b..c4b74eb985 100644 --- a/internal/provider/kubernetes/gatewayclass.go +++ b/internal/provider/kubernetes/gatewayclass.go @@ -6,6 +6,7 @@ package kubernetes import ( "context" "fmt" + "sync" "github.com/go-logr/logr" "k8s.io/apimachinery/pkg/types" @@ -27,23 +28,28 @@ type gatewayClassReconciler struct { controller gwapiv1b1.GatewayController statusUpdater status.Updater log logr.Logger + + initializeOnce sync.Once + resourceTable *ResourceTable } // newGatewayClassController creates the gatewayclass controller. The controller // will be pre-configured to watch for cluster-scoped GatewayClass objects with // a controller field that matches name. -func newGatewayClassController(mgr manager.Manager, cfg *config.Server) error { +func newGatewayClassController(mgr manager.Manager, cfg *config.Server, resourceTable *ResourceTable) error { cli := mgr.GetClient() uh := status.NewUpdateHandler(cfg.Logger, cli) if err := mgr.Add(uh); err != nil { return fmt.Errorf("failed to add status update handler %v", err) } + resourceTable.Initialized.Add(1) r := &gatewayClassReconciler{ client: cli, controller: gwapiv1b1.GatewayController(cfg.EnvoyGateway.Gateway.ControllerName), statusUpdater: uh.Writer(), log: cfg.Logger, + resourceTable: resourceTable, } c, err := controller.New("gatewayclass", mgr, controller.Options{Reconciler: r}) @@ -96,21 +102,27 @@ func (r *gatewayClassReconciler) Reconcile(ctx context.Context, request reconcil var cc controlledClasses + found := false for i := range gatewayClasses.Items { if gatewayClasses.Items[i].Spec.ControllerName == r.controller { cc.addMatch(&gatewayClasses.Items[i]) + if gatewayClasses.Items[i].GetName() == request.Name { + found = true + } } } + if !found { + r.resourceTable.GatewayClasses.Delete(request.Name) + } // no controlled gatewayclasses, trigger a delete if len(cc.matchedClasses) == 0 { r.log.Info("failed to find gatewayclass", "name", request.Name) - // TODO: Delete gatewayclass from the IR. - // xref: https://github.com/envoyproxy/gateway/issues/38 return reconcile.Result{}, nil } updater := func(gc *gwapiv1b1.GatewayClass, accepted bool) error { + r.resourceTable.GatewayClasses.Store(gc.GetName(), gc) if r.statusUpdater != nil { r.statusUpdater.Send(status.Update{ NamespacedName: types.NamespacedName{Name: gc.Name}, @@ -144,9 +156,8 @@ func (r *gatewayClassReconciler) Reconcile(ctx context.Context, request reconcil if err := updater(cc.acceptedClass(), true); err != nil { return reconcile.Result{}, err } - - // TODO: Add gatewayclass to the IR. - // xref: https://github.com/envoyproxy/gateway/issues/38 + // Once we've iterated over all listed classes, mark that we've fully initialized. + r.initializeOnce.Do(r.resourceTable.Initialized.Done) r.log.WithName(request.Name).Info("reconciled gatewayclass") return reconcile.Result{}, nil diff --git a/internal/provider/kubernetes/kubernetes.go b/internal/provider/kubernetes/kubernetes.go index 7936eaf727..e7b5c7d386 100644 --- a/internal/provider/kubernetes/kubernetes.go +++ b/internal/provider/kubernetes/kubernetes.go @@ -3,26 +3,42 @@ package kubernetes import ( "context" "fmt" + "sync" + "github.com/telepresenceio/watchable" + "k8s.io/apimachinery/pkg/types" "k8s.io/client-go/rest" ctrl "sigs.k8s.io/controller-runtime" "sigs.k8s.io/controller-runtime/pkg/client" "sigs.k8s.io/controller-runtime/pkg/manager" + gwapiv1b1 "sigs.k8s.io/gateway-api/apis/v1beta1" "github.com/envoyproxy/gateway/internal/envoygateway" "github.com/envoyproxy/gateway/internal/envoygateway/config" ) +// ResourceTable is a listing of all of the Kubernetes resources bing +// watched. +type ResourceTable struct { + // Initialized.Wait() will return once each of the maps in the + // table have been initialized at startup. + Initialized sync.WaitGroup + + GatewayClasses watchable.Map[string, *gwapiv1b1.GatewayClass] + Gateways watchable.Map[types.NamespacedName, *gwapiv1b1.Gateway] +} + // Provider is the scaffolding for the Kubernetes provider. It sets up dependencies // and defines the topology of the provider and its managed components, wiring // them together. type Provider struct { - client client.Client - manager manager.Manager + client client.Client + manager manager.Manager + resourceTable *ResourceTable } // New creates a new Provider from the provided EnvoyGateway. -func New(cfg *rest.Config, svr *config.Server) (*Provider, error) { +func New(cfg *rest.Config, svr *config.Server, resourceTable *ResourceTable) (*Provider, error) { // TODO: Decide which mgr opts should be exposed through envoygateway.provider.kubernetes API. mgrOpts := manager.Options{ Scheme: envoygateway.GetScheme(), @@ -37,18 +53,19 @@ func New(cfg *rest.Config, svr *config.Server) (*Provider, error) { } // Create and register the controllers with the manager. - if err := newGatewayClassController(mgr, svr); err != nil { + if err := newGatewayClassController(mgr, svr, resourceTable); err != nil { return nil, fmt.Errorf("failed to create gatewayclass controller: %w", err) } - if err := newGatewayController(mgr, svr); err != nil { + if err := newGatewayController(mgr, svr, resourceTable); err != nil { return nil, fmt.Errorf("failed to create gateway controller: %w", err) } // TODO: Add httproute controllers. // xref: https://github.com/envoyproxy/gateway/issues/163 return &Provider{ - manager: mgr, - client: mgr.GetClient(), + manager: mgr, + client: mgr.GetClient(), + resourceTable: resourceTable, }, nil } @@ -67,3 +84,9 @@ func (p *Provider) Start(ctx context.Context) error { return err } } + +// Resources returns an updating table of all of the resources being +// watched. +func (p *Provider) Resources() *ResourceTable { + return p.resourceTable +} diff --git a/internal/provider/provider.go b/internal/provider/provider.go index 76f804f0e5..663711b242 100644 --- a/internal/provider/provider.go +++ b/internal/provider/provider.go @@ -10,18 +10,24 @@ import ( "github.com/envoyproxy/gateway/internal/provider/kubernetes" ) -func Start(svr *config.Server) error { +type ResourceTable = kubernetes.ResourceTable + +func Start(svr *config.Server, k8sTable *ResourceTable) error { log := svr.Logger if svr.EnvoyGateway.Provider.Type == v1alpha1.ProviderTypeKubernetes { log.Info("Using provider", "type", v1alpha1.ProviderTypeKubernetes) - cfg := ctrl.GetConfigOrDie() - provider, err := kubernetes.New(cfg, svr) + cfg, err := ctrl.GetConfig() + if err != nil { + return fmt.Errorf("failed to get kubeconfig: %w", err) + } + provider, err := kubernetes.New(cfg, svr, k8sTable) if err != nil { return fmt.Errorf("failed to create provider %s", v1alpha1.ProviderTypeKubernetes) } - if err := provider.Start(ctrl.SetupSignalHandler()); err != nil { + if err := provider.Start(ctrl.SetupSignalHandler()); err != nil { //lint:ignore SA4023 provider.Start currently never returns non-nil return fmt.Errorf("failed to start provider %s", v1alpha1.ProviderTypeKubernetes) } + return nil } // Unsupported provider. return fmt.Errorf("unsupported provider type %v", svr.EnvoyGateway.Provider.Type) diff --git a/internal/provider/provider_test.go b/internal/provider/provider_test.go index acdf649f58..8e7a1bdca8 100644 --- a/internal/provider/provider_test.go +++ b/internal/provider/provider_test.go @@ -37,7 +37,7 @@ func TestStart(t *testing.T) { for _, tc := range testCases { t.Run(tc.name, func(t *testing.T) { - err := Start(tc.cfg) + err := Start(tc.cfg, new(ResourceTable)) if tc.expect { require.NoError(t, err) } else { diff --git a/internal/xds/types/resourceversiontable.go b/internal/xds/types/resourceversiontable.go index 91153df22a..5c24a7cbaa 100644 --- a/internal/xds/types/resourceversiontable.go +++ b/internal/xds/types/resourceversiontable.go @@ -6,7 +6,12 @@ import ( ) // XdsResources represents all the xds resources -type XdsResources map[resource.Type][]types.Resource +// +// This is the type that +// github.com/envoyproxy/go-control-plane/pkg/cache/v3.NewSnapshot +// takes; if we decide that we want to change this type, then we'd +// have to do conversion. +type XdsResources = map[resource.Type][]types.Resource // ResourceVersionTable holds all the translated xds resources type ResourceVersionTable struct {