Add Cross Repository CI Relay (CRCR) infrastructure(L1 Only)

[fffrog]

Note

Due to the restrictions on secret injection in the fork Repo scenario on GitHub, a new PR needs to be created to replace the old one (#415). Please refer to the old PR for a detailed discussion.

Summary

Please refer to this comment for the overall implementation.

• Add Terraform infrastructure for CRCR (Cross-Repository CI Relay), a GitHub webhook relay service for PyTorch out-of-tree backends that receives upstream webhook events via a GitHub App and forwards repository_dispatch events to registered downstream repositories
• Infrastructure includes: Lambda function (webhook handler), ElastiCache Redis (allowlist caching), dedicated VPC, IAM roles, and Lambda Function URL
• Add two GitHub Actions workflows: crcr-on-pr.yml and crcr-deploy-prod.yml

Notes:

This PR need to wait this merged first for purpose of updating tag field in Terrafile.

Architecture

GitHub App → Lambda webhook (Function URL) → repository_dispatch → downstream repos

AWS Resources (us-east-1, account 391835788720):

• Lambda function (cross_repo_ci_webhook) with Python 3.10 runtime
• ElastiCache Redis replication group for allowlist caching
• VPC with private subnets for Lambda ↔ Redis connectivity
• IAM role with Secrets Manager, VPC networking, and CloudWatch Logs permissions
• S3 backend for Terraform state

Test

Multiple deployments and verifications have been completed on personal AWS environment.

[fffrog]

Hi @zxiiro, I apologize for bothering you again.

The failed job named TFLint & Plan - CRCR / CRCR tflint + terraform plan was due to missing S3 and DynamoDB. Therefore, we need to manually create S3 and DynamoDB first, and then ensure that the IAM role has access to the new bucket. After that, I think the job should succeed.

Could you help me create S3/DynamoDB? section should be a good reference. Thank you very much for your help!

[zxiiro]

Hi @zxiiro, I apologize for bothering you again.

The failed job named TFLint & Plan - CRCR / CRCR tflint + terraform plan was due to missing S3 and DynamoDB. Therefore, we need to manually create S3 and DynamoDB first, and then ensure that the IAM role has access to the new bucket. After that, I think the job should succeed.

Could you help me create S3/DynamoDB? section should be a good reference. Thank you very much for your help!

I think its failing because the bucket tfstate-pyt-crcr-prod already exists. I'm not able to create it manually either it says bucket already exists but i don't see it in the PyTorch Foundation account.

So In AWS S3 buckets are globally unique. If your team created a bucket with the same name in your own account when testing then we cannot create this bucket name because you've already used it. You'll need to delete your bucket in your test account to release the name for us to deploy or we will need to update the terraform config in this PR to use a different bucket name.

[fffrog]

I think its failing because the bucket tfstate-pyt-crcr-prod already exists. I'm not able to create it manually either it says bucket already exists but i don't see it in the PyTorch Foundation account.

Thanks for pointing this out. I was previously unaware that AWS S3 buckets are unique across different accounts, which is a bit unbelievable to me :D.

You'll need to delete your bucket in your test account to release the name for us to deploy or we will need to update the terraform config in this PR to use a different bucket name.

I prefer to keep the name tfstate-pyt-crcr-prod for production, so I have already freed the name from my personal AWS account, and have confirmed that the name has been freed by rerunning the failed job. If you have time, please help create the S3 and DynamoDB instances; thank you very much.

[fffrog]

Hi @ZainRizvi @zxiiro, sorry to bother you again.

I’ve introduced a few additional changes in my latest commits. Below is a brief summary of the rationale behind them. Please let me know if you have any feedback.

Change 1: Sync with Upstream VPC Module

• Rationale: Based on the discussion here.
• Modification: Replaced jeanschmidt/terraform-aws-vpc with the official upstream terraform-aws-modules/terraform-aws-vpc (v6.6.1).

Change 2: Simplify Lambda Permissions

• Rationale: With the AWS provider upgraded to v6.28+ (required by the new VPC module), we can now use native invoked_via_function_url support.
• Modification: Replaced the previous aws_cloudformation_stack workaround with native aws_lambda_permission resources to simplify the stack.

Change 3: Flatten Directory Structure

• Rationale:
  • The previous aws/<account>/<region>/ structure added unnecessary complexity for canary testing in personal accounts (requiring manual renaming of S3 buckets and directories to avoid conflicts).
  • Given that CRCR does not require multi-region deployment, a flatter structure is more maintainable.
• Modification: Flattened the directory structure to aws/.

[zxiiro]

I prefer to keep the name tfstate-pyt-crcr-prod for production, so I have already freed the name from my personal AWS account, and have confirmed that the name has been freed by rerunning the failed job. If you have time, please help create the S3 and DynamoDB instances; thank you very much.

Alright S3 and DynamoDB has been created. The Terraform Plan jobs appear to be working now. I'll take a look at your newer changes later today.

[fffrog]

Alright S3 and DynamoDB has been created. The Terraform Plan jobs appear to be working now. I'll take a look at your newer changes later today.

Thank you lot, and the CI have passed :D

Please let me know if you have any questions and we have carefully tested in my personal AWS with the environment suffix "canary" 😀

[ZainRizvi]

Looking good!

[zxiiro]

Approved. I have 2 suggestions if you want to handle them in this PR; otherwise I will merge this tomorrow during my daytime.