-
Notifications
You must be signed in to change notification settings - Fork 630
GCP PubSub Channel #618
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Merged
Merged
GCP PubSub Channel #618
Changes from all commits
Commits
Show all changes
38 commits
Select commit
Hold shift + click to select a range
14a4842
Initial work on the GCP PubSub Channel Controller.
Harwayne f99bf21
Switch to the more standard model, a single global dispatcher.
Harwayne 72bb3eb
Args are now defaulted on the controller, removed the ability to cust…
Harwayne b3b9275
YAML and scaffold for the dispatcher.
Harwayne 03c69e6
Created the 'receive' side of the dispatcher. The dispatch side is to…
Harwayne c1d1074
Initial work on the dispatcher.
Harwayne ac685a2
More work on the dispatcher.
Harwayne a6ff217
Dispatcher mostly working.
Harwayne 8f5756c
Handle the reconcileChan
Harwayne 7660fe3
Merge branch 'master' into pb-channel
Harwayne 43d6e06
Minor improvements.
Harwayne 7bb74b0
Touch ups.
Harwayne 9032baf
Merge branch 'master' into pb-channel
Harwayne 85c3aca
Rename Run->Start.
Harwayne 08e0320
Fake PubSub.
Harwayne 7acb792
Util unit tests.
Harwayne c05f46b
Unit tests for the channel controller.
Harwayne c32b5ab
Beginning unit tests for dispatcher.
Harwayne 667ba1a
update-deps
Harwayne 9542a6d
Unit tests for dispatcher/reconcile.go
Harwayne 6901012
Unit test dispatcher/receiver.
Harwayne abaa872
Add a README.
Harwayne ba2e849
DeepCopy the sub
Harwayne 5e0ce22
Merge branch 'master' into pb-channel
Harwayne e67a304
Add the ServiceEntry to the Channel's config.
Harwayne 4cff962
Typo
Harwayne 5b77a10
PR comments
Harwayne d59873d
Merge branch 'master' into pb-channel
Harwayne 3906810
buses -> provisioners
Harwayne 0b73e4c
Merge remote-tracking branch 'origin/pb-channel' into pb-channel
Harwayne 15784c9
Switch to the common utils.
Harwayne 864c20d
Add VirtualService.Update RBAC now that VirtualServices are updated i…
Harwayne 2e8bf1d
PR comments
Harwayne b2d0d9f
Move the contoller packages inside the controller folder.
Harwayne 5a38f65
Merge branch 'master' into pb-channel
Harwayne a3ea34d
PR comments (mostly log levels).
Harwayne 49f47a3
Desugar to get the fields right.
Harwayne 74a1b6f
Switch service name, to match #626.
Harwayne File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.
Oops, something went wrong.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,91 @@ | ||
| # GCP PubSub Channels | ||
|
|
||
| GCP PubSub channels are production-quality Channels that are backed by | ||
| [GCP PubSub](https://cloud.google.com/pubsub/). | ||
|
|
||
| They offer: | ||
| * Persistence | ||
| - If the Channel's Pod goes down, all events already ACKed by the Channel will persist and be | ||
| retransmitted when the Pod restarts. | ||
| * Redelivery attempts | ||
| - If downstream rejects an event, that request is attempted again. | ||
|
|
||
| They do not offer: | ||
| * Ordering guarantees | ||
| - Events seen downstream may not occur in the same order they were inserted into the Channel. | ||
|
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. I find this particular formatting of a list of titles plus descriptions tough to take in at a glance. I prefer something more like this:
|
||
|
|
||
| ### Deployment Steps | ||
|
|
||
| #### Prerequisites | ||
|
|
||
| 1. Create a [Google Cloud Project](https://cloud.google.com/resource-manager/docs/creating-managing-projects). | ||
| 1. Enable the 'Cloud Pub/Sub API' on that project. | ||
|
|
||
| ```shell | ||
| gcloud services enable pubsub.googleapis.com | ||
| ``` | ||
|
|
||
| 1. Create a GCP [Service Account](https://console.cloud.google.com/iam-admin/serviceaccounts/project). | ||
| 1. Determine the Service Account to use, or create a new one. | ||
| 1. Give that Service Account the 'Pub/Sub Editor' role on your GCP project. | ||
| 1. Download a new JSON private key for that Service Account. | ||
| 1. Create a secret for the downloaded key: | ||
|
|
||
| ```shell | ||
| kubectl -n knative-sources create secret generic gcppubsub-channel-key --from-file=key.json=PATH_TO_KEY_FILE.json | ||
| ``` | ||
|
|
||
| 1. Setup [Knative Eventing](../../../DEVELOPMENT.md). | ||
|
Harwayne marked this conversation as resolved.
|
||
|
|
||
| #### Deployment | ||
|
|
||
| 1. Set the shell variable with the correct value: | ||
|
|
||
| ```shell | ||
| export GCP_PROJECT=REPLACE_ME | ||
| ``` | ||
|
|
||
| 1. Apply `gcppubsub.yaml`. | ||
|
|
||
| ```shell | ||
| sed "s/REPLACE_WITH_GCP_PROJECT/$GCP_PROJECT/" config/provisioners/gcppubsub/gcppubsub.yaml | ko apply -f - | ||
| ``` | ||
|
|
||
| 1. Create Channels that reference the `gcp-pubsub` Channel. | ||
|
|
||
| ```yaml | ||
| apiVersion: eventing.knative.dev/v1alpha1 | ||
| kind: Channel | ||
| metadata: | ||
| name: foo | ||
| spec: | ||
| provisioner: | ||
| apiVersion: eventing.knative.dev/v1alpha1 | ||
| kind: ClusterChannelProvisioner | ||
| name: gcp-pubsub | ||
| ``` | ||
|
|
||
| ### Components | ||
|
|
||
| The major components are: | ||
| * [Channel Controller](../../../pkg/provisioners/gcppubsub/controller) | ||
| - [ClusterChannelProvisioner Controller](../../../pkg/provisioners/gcppubsub/clusterchannelprovisioner) | ||
| - [Channel Controller](../../../pkg/provisioners/gcppubsub/channel) | ||
| * [Channel Dispatcher](../../../pkg/provisioners/gcppubsub/dispatcher/cmd) | ||
| - [Dispatcher](../../../pkg/provisioners/gcppubsub/dispatcher/dispatcher) | ||
| - [Receiver](../../../pkg/provisioners/gcppubsub/dispatcher/receiver) | ||
|
|
||
| The `Channel Controller` controls all the Kubernetes resources and creates `Topic`s and | ||
| `Subscription`s in GCP PubSub. It runs in the Deployment: | ||
|
|
||
| ```shell | ||
| kubectl get deployment -n knative-eventing gcp-pubsub-channel-controller | ||
| ``` | ||
|
|
||
| The `Channel Dispatcher` handles all the data plane portions of the `Channel`. It receives events | ||
| from the cluster and writes them to GCP PubSub `Topic`s. It also polls `Subscriptions`s and sends | ||
| events back into the cluster. It runs in the Deployment: | ||
|
|
||
| ```shell | ||
| kubectl get deployment -n knative-eventing gcp-pubsub-channel-dispatcher | ||
| ``` | ||
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.