GCP Pub/Sub Knative Bus#98
Conversation
google-prow-robot
added
the
size/XXL
google-prow-robot
added
size/XL
|
Rebased on master now that #88 is merged |
A Knative Bus implementation backed by GCP Cloud Pub/Sub. Deployment steps: 1. Setup Knative Eventing 2. [Create a service account](https://console.cloud.google.com/iam-admin/serviceaccounts/project) with the 'Pub/Sub Editor' role, and download a new JSON private key. 3. Create a secret for the downloaded key `kubectl create secret generic gcppubsub-bus-key --from-file=key.json=PATH-TO-KEY-FILE.json` 4. Replace `$PROJECT_ID` in `config/buses/gcppubsub.yaml` with your GCP Project ID 5. Apply the 'gcppubsub' Bus `ko apply -f config/buses/gcppubsub.yaml` 6. Create Channels that reference the 'gcppubsub' Bus The bus has an independent provisioner and dispatcher. The provisioner will create GCP Pub/Sub Topics and Subscriptions for each Knative Channel and Subscription (respectively) targeting the Bus. Clients should avoid interacting with topics and subscriptions provisioned by the Bus. The dispatcher can receive events via the Channel Service from inside the cluster. Events on the Pub/Sub topic for an active subscription are forwarded via HTTP to the subscriber. HTTP responses with a 2xx status code are acknowledge while all other status code will have delivery reattempted. Note: Cloud Pub/Sub does not guarantee exactly once delivery, subscribers must guard against multiple deliveries of the same event. To view logs: - for the dispatcher `kail -d gcppubsub-bus -c dispatcher` - for the provisioner `kail -d gcppubsub-bus-provisioner -c provisioner`
| @@ -0,0 +1,62 @@ | |||
| # Copyright 2018 Google, Inc. All rights reserved. | |||
| topicID := b.topicID(channel) | ||
| topic := b.pubsubClient.Topic(topicID) | ||
|
|
||
| // check if topic exists before creating |
There was a problem hiding this comment.
I'm just curious about the pattern of checking for existence before trying these operations and why it's being done in every case. With this check then do something else, there's a race condition anyways, so like I said, just curious about this pattern.
There was a problem hiding this comment.
The call to create a topic will return an error if the topic already exists. Rather than try to inspect the error I found it easier to check if the topic exists before creating.
There was a problem hiding this comment.
Sorry, I meant the pattern in each of these, like delete for example and so forth. Same deal there?
There was a problem hiding this comment.
Each operation will return an error unless the client was directly responsible for achieving the desired effect. So deleting a topic that doesn't exist will return an error. The published doc doesn't distinguish between different types of errors. There are a few hint in the codebase that NOT_FOUND is distinguished from ALREADY_EXISTS and other errors, but it's not part of the public interface from what I can see.
There was a problem hiding this comment.
There was a problem hiding this comment.
To your comment about a race condition, the provisioner is a singleton, and the workqueue guarantees that any given resource will never be processed more than once concurrently.
|
/lgtm |
|
[APPROVALNOTIFIER] This PR is APPROVED This pull-request has been approved by: scothis, vaikas-google The full list of commands accepted by this bot can be found here. The pull request process is described here DetailsNeeds approval from an approver in each of these files:
Approvers can indicate their approval by writing |
google-prow-robot
added
the
approved
Adding new approvers
5 participants
A Knative Bus implementation backed by GCP Cloud Pub/Sub.
Deployment steps:
kubectl create secret generic gcppubsub-bus-key --from-file=key.json=PATH-TO-KEY-FILE.json$PROJECT_IDinconfig/buses/gcppubsub.yamlwith your GCP Project IDko apply -f config/buses/gcppubsub.yamlThe bus has an independent provisioner and dispatcher.
The provisioner will create GCP Pub/Sub Topics and Subscriptions for each Knative Channel and Subscription (respectively) targeting the Bus. Clients should avoid interacting with topics and subscriptions provisioned by the Bus.
The dispatcher can receive events via the Channel Ingress from outside the cluster, or the Channel Service from inside the cluster. Events on the Pub/Sub topic for an active subscription are forwarded via HTTP to the subscriber. HTTP responses with a 2xx status code are acknowledge while all other status code will have delivery reattempted.
Note: Cloud Pub/Sub does not guarantee exactly once delivery, subscribers must guard against multiple deliveries of the same event.
To view logs:
kail -d gcppubsub-bus -c dispatcherkail -d gcppubsub-bus-provisioner -c provisionerResolves #80
This PR is effectively adds a single commit on top of #88, and will be rebased after it merges.