Support AWS China install by wanghaoran1988 · Pull Request #209 · openshift/enhancements

wanghaoran1988 · 2020-02-13T14:43:09Z

openshift-ci-robot · 2020-02-13T14:44:05Z

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: wanghaoran1988
To complete the pull request process, please assign soltysh
You can assign the PR to them by writing /assign @soltysh in a comment when ready.

The full list of commands accepted by this bot can be found here.

Details

Needs approval from an approver in each of these files:

OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

abhinavdahiya · 2020-02-13T17:36:24Z

+### Non-Goals
+
+* It's not a goal to detail how to request and setup a AWS account in AWS China.
+* It's not a goal to detail how to do UPI install.


I think another goal would be supporting any endpoints for services that are not part of the SDK for those region.

abhinavdahiya · 2020-02-13T17:37:48Z

+#### Setup AWS China Account
+
+The AMIs in global regions are not useable in AWS China regions, we need setup an AWS China Account to host our RHCOS AMIs, so that installer can use them to setup the cluster.
+
+#### Push RHCOS AMIs to AWS China account, and share them to public.
+
+Currently, we have CI jobs push the AMIs to public regions, after the AWS China account setup is ready, we should make our CI job start push our AMIs to AWS China regions, and share them to all accounts in AWS China regions.
+
+#### OpenShift installer support AWS China regions
+
+The OpenShift installer should be able to use the AMIs that pushed to AWS China regions to provision clusters, and use the correct api endpoints and ARNs, Notable difference for AWS China:
+
+* AWS resources ARNs in China regions are prefixed with "arn:arn-cn"
+* Ec2 service endpoint is "ec2.amazonaws.com.cn"
+* Route53 currently is not GA, we can use api endpoint "route53.amazonaws.com.cn" or "api.route53.cn" in AWS China.
+
+#### Cloud credential operator support AWS China regions
+
+Cloud credential operator will create AWS client and use IAM service to validate the permission for provided AWS credential, to support AWS China, it should use IAM api endpoint "iam.amazonaws.com.cn" for AWS China regions.
+
+#### Ingress operator support AWS China regions
+
+Ingress operator will create ELBs and using route53 service to update related DNS records, to support AWS China, it should use
+"route53.amazonaws.com.cn" or "api.route53.cn" api endpoint. And for the resource groups tagging api, it should use "tagging.cn-northwest-1.amazonaws.com.cn"


these don't seem to correctly belong to User Stories

What do you mean "these"? Would you mind tell me where I should put these?

Comment on lines 58 to 82

These sound like implementation/proposal and not user stories.

Done, moved to "implemetations" part

abhinavdahiya · 2020-02-13T17:38:26Z

+
+* AWS resources ARNs in China regions are prefixed with "arn:arn-cn"
+* Ec2 service endpoint is "ec2.amazonaws.com.cn"
+* Route53 currently is not GA, we can use api endpoint "route53.amazonaws.com.cn" or "api.route53.cn" in AWS China.


I think we should mention that AWS SDK is missing the endpoint, so we should decide what endpoint is going to be hard coded.

abhinavdahiya · 2020-02-13T17:39:22Z

+The OpenShift installer should be able to use the AMIs that pushed to AWS China regions to provision clusters, and use the correct api endpoints and ARNs, Notable difference for AWS China:
+
+* AWS resources ARNs in China regions are prefixed with "arn:arn-cn"
+* Ec2 service endpoint is "ec2.amazonaws.com.cn"


AWS SDK already has endpoint information for this region, so i think we should probably focus on

make sure you are using SDK defaults

define how the SDK should be configured so that (1) is achievable.

abhinavdahiya · 2020-02-13T17:39:51Z

+
+The AMIs in global regions are not useable in AWS China regions, we need setup an AWS China Account to host our RHCOS AMIs, so that installer can use them to setup the cluster.
+
+#### Push RHCOS AMIs to AWS China account, and share them to public.


cc @cgwalters @ashcrow

abhinavdahiya · 2020-02-13T17:40:47Z

+
+#### Cloud credential operator support AWS China regions
+
+Cloud credential operator will create AWS client and use IAM service to validate the permission for provided AWS credential, to support AWS China, it should use IAM api endpoint "iam.amazonaws.com.cn" for AWS China regions.


Again i don't think we should be hard coding things that are already in SDK..

Something along the lines of https://github.com/openshift/enhancements/pull/209/files#r379014913 would make more sense.

abhinavdahiya · 2020-02-13T17:43:04Z

+
+#### Ingress operator support AWS China regions
+
+Ingress operator will create ELBs and using route53 service to update related DNS records, to support AWS China, it should use


For tagging same as https://github.com/openshift/enhancements/pull/209/files#r379015589

For route 53, i think https://github.com/openshift/enhancements/pull/209/files#r379014462

And this also needs to define, the route53 resources behavior when using resourcetaggingapi

the resourcetaggingapi returns no results for route53 zone, unless the region being used is some predetermined one.. like us-east-1 for all public regions.

Also ingress operator doesn't create ELB, the kube-controller-manager does.

abhinavdahiya · 2020-02-13T17:43:15Z

+* Ec2 service endpoint is "ec2.amazonaws.com.cn"
+* Route53 currently is not GA, we can use api endpoint "route53.amazonaws.com.cn" or "api.route53.cn" in AWS China.
+
+#### Cloud credential operator support AWS China regions


cc @dgoodwin @joelddiaz

abhinavdahiya · 2020-02-13T17:43:46Z

+
+Cloud credential operator will create AWS client and use IAM service to validate the permission for provided AWS credential, to support AWS China, it should use IAM api endpoint "iam.amazonaws.com.cn" for AWS China regions.
+
+#### Ingress operator support AWS China regions


cc @ironcladlou

abhinavdahiya · 2020-02-13T17:44:44Z

+
+### Test Plan
+
+Our testing CI should include one AWS China Region, and run the installer and e2e tests in AWS China account.


I think we should include that DP team must setup the account before we can begin testing and have it available to devs in openshift. Calling that out explicitly should help us.

These requests have been made.

cgwalters · 2020-02-13T18:44:08Z

This intersects with #201 in that if we had that it'd be a bit more obvious how to upload an AMI to this region manually.

wanghaoran1988 · 2020-02-18T13:43:28Z

@abhinavdahiya Comments should be addressed, could you take another look again ?

crawford · 2020-02-18T21:05:50Z

+* It's not a goal to detail how to do UPI install.
+* It's not a goal to support all AWS service endpoints that are not part of the SDK for AWS China Regions.
+
+## Proposal


Do we need to worry about the Great Firewall? Quay uses S3 as it’s backing store and China blocks it from time to time. It’s been a while since I’ve had to pay attention to that, so maybe it’s not an issue anymore.

Quay.io is slow, but it's still workable at the moment.

Okay, that doubles my concern then. Slow download speeds are going to have a large impact on cluster install times and their ability to scale quickly. And there still seems to be the possibility that the firewall will block the traffic outright.

Quay has a georeplication project underway which might help, but we cannot depend on that as a prerequisite for this China bringup, especially since so far in China testing we haven't had any failures. Also, we're not do not expect a large volume of cluster installs or scaling activity in China.

Wait quay really uses S3 and not a proper CDN that would have regional distribution? Hmm, AFAICS I'm not getting any redirects for the blobs indeed.

jeremyeder · 2020-02-25T21:19:00Z

@abhinavdahiya @crawford hey - looking to make sure we've addressed all the comments on this to your satisfaction. If not, can you point us to where we need to update the PR?

Who can merge this once we've got it all buttoned up?

abhinavdahiya · 2020-02-25T21:21:14Z

@abhinavdahiya @crawford hey - looking to make sure we've addressed all the comments on this to your satisfaction. If not, can you point us to where we need to update the PR?

Who can merge this once we've got it all buttoned up?

will try to take a look at this again. I think I can lgtm and merge this when it's ready.

dofinn · 2020-03-05T04:31:29Z

@abhinavdahiya @crawford hey - looking to make sure we've addressed all the comments on this to your satisfaction. If not, can you point us to where we need to update the PR?
Who can merge this once we've got it all buttoned up?

will try to take a look at this again. I think I can lgtm and merge this when it's ready.

@abhinavdahiya Any update on this review? Is there anything blocking or that we can clarify/assist with?

openshift-bot · 2020-10-01T23:47:17Z

Issues go stale after 90d of inactivity.

Mark the issue as fresh by commenting /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.
Exclude this issue from closing by commenting /lifecycle frozen.

If this issue is safe to close now please do so with /close.

/lifecycle stale

openshift-bot · 2020-11-01T01:36:54Z

Stale issues rot after 30d of inactivity.

Mark the issue as fresh by commenting /remove-lifecycle rotten.
Rotten issues close after an additional 30d of inactivity.
Exclude this issue from closing by commenting /lifecycle frozen.

If this issue is safe to close now please do so with /close.

/lifecycle rotten
/remove-lifecycle stale

openshift-bot · 2020-12-01T03:31:04Z

Rotten issues close after 30d of inactivity.

Reopen the issue by commenting /reopen.
Mark the issue as fresh by commenting /remove-lifecycle rotten.
Exclude this issue from closing again by commenting /lifecycle frozen.

/close

openshift-ci-robot · 2020-12-01T03:31:21Z

@openshift-bot: Closed this PR.

Details

In response to this:

Rotten issues close after 30d of inactivity.

Reopen the issue by commenting /reopen.
Mark the issue as fresh by commenting /remove-lifecycle rotten.
Exclude this issue from closing again by commenting /lifecycle frozen.

/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

Support AWS China install

9ec8883

openshift-ci-robot requested review from abhinavdahiya and jeremyeder February 13, 2020 14:43

openshift-ci-robot added the size/L Denotes a PR that changes 100-499 lines, ignoring generated files. label Feb 13, 2020

abhinavdahiya reviewed Feb 13, 2020

View reviewed changes

Address comments

6b8c205

wanghaoran1988 force-pushed the aws_china branch from c403d80 to 6b8c205 Compare February 17, 2020 08:39

crawford reviewed Feb 18, 2020

View reviewed changes

openshift-ci-robot added the lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. label Oct 1, 2020

openshift-ci-robot added lifecycle/rotten Denotes an issue or PR that has aged beyond stale and will be auto-closed. and removed lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. labels Nov 1, 2020

openshift-ci-robot closed this Dec 1, 2020


		The AMIs in global regions are not useable in AWS China regions, we need setup an AWS China Account to host our RHCOS AMIs, so that installer can use them to setup the cluster.

		#### Push RHCOS AMIs to AWS China account, and share them to public.


		#### Cloud credential operator support AWS China regions

		Cloud credential operator will create AWS client and use IAM service to validate the permission for provided AWS credential, to support AWS China, it should use IAM api endpoint "iam.amazonaws.com.cn" for AWS China regions.


		#### Ingress operator support AWS China regions

		Ingress operator will create ELBs and using route53 service to update related DNS records, to support AWS China, it should use


		### Test Plan

		Our testing CI should include one AWS China Region, and run the installer and e2e tests in AWS China account.

Conversation

wanghaoran1988 commented Feb 13, 2020

Uh oh!

openshift-ci-robot commented Feb 13, 2020

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

cgwalters commented Feb 13, 2020

Uh oh!

wanghaoran1988 commented Feb 18, 2020

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

jeremyeder commented Feb 25, 2020

Uh oh!

abhinavdahiya commented Feb 25, 2020

Uh oh!

dofinn commented Mar 5, 2020

Uh oh!

openshift-bot commented Oct 1, 2020

Uh oh!

openshift-bot commented Nov 1, 2020

Uh oh!

openshift-bot commented Dec 1, 2020

Uh oh!

openshift-ci-robot commented Dec 1, 2020

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

8 participants