Request Driven Backend Service deployment fails with YAML Validation error when secrets are added

[jorgeavaldez]

We initially created a Request Driven Backend service without secrets that was successfully deployed, both manually and via a pipeline.

We then started adding secrets to SSM Parameter store using copilot secret init --cli-input-yaml. I successfully was able to inject these into the pipeline and a scheduled job. When attempting to deploy the backend service, I receive the following error:

✘ Proposing infrastructure changes for stack drakula-staging-backend
✘ execute deployment 1 of 1 in group 1: deploy service backend to environment staging: deploy service: check if changeset is empty: create 
change set copilot-fd308a18-2202-4477-bad3-507021bf4c25 for stack drakula-staging-backend: ValidationError: Template format error: YAML not
 well-formed. (line 215, column 13)
        status code: 400, request id: 10e7a64f-683f-4baa-a4bb-8e7f89b6e75d: describe change set copilot-fd308a18-2202-4477-bad3-507021bf4c2
5 for stack drakula-staging-backend: ChangeSetNotFound: ChangeSet [copilot-fd308a18-2202-4477-bad3-507021bf4c25] does not exist
        status code: 404, request id: e2729cbf-dfa4-42fd-97dd-32c61b54a38c

This is my manifest:

# The manifest for the "backend" service.
# Read the full specification for the "Request-Driven Web Service" type at:
# https://aws.github.io/copilot-cli/docs/manifest/rd-web-service/

# Your service name will be used in naming your resources like log groups, App Runner services, etc.
name: backend
# The "architecture" of the service you're running.
type: Request-Driven Web Service
image:
  # Docker build arguments.
  # For additional overrides: https://aws.github.io/copilot-cli/docs/manifest/rd-web-service/#image-build
  build: Dockerfile
  # Port exposed through your container to route traffic to it.
  port: 8151
# http:
#   healthcheck:
#     path: /
#     healthy_threshold: 3
#     unhealthy_threshold: 5
#     interval: 10s
#     timeout: 5s

# Number of CPU units for the task.
cpu: 1024
# Amount of memory in MiB used by the task.
memory: 2048
# Connect your App Runner service to your environment's VPC.
#
network:
  vpc:
    placement: private

# Enable tracing for the service.
# observability:
#   tracing: awsxray
# Optional fields for more advanced use-cases.
#
# variables:                    # Pass environment variables as key value pairs.
#   LOG_LEVEL: info
#
# tags:                         # Pass tags as key value pairs.
#   project: project-name
secrets:
  NFT_STORAGE_API_KEY: NFT_STORAGE_API_KEY
  BUNNY_STREAM_API_KEY: BUNNY_STREAM_API_KEY
  BUNNY_STORAGE_ZONE_PASSWORD: BUNNY_STORAGE_ZONE_PASSWORD
  SENTRY_AUTH_TOKEN: SENTRY_AUTH_TOKEN
  PRIVY_APP_SECRET: PRIVY_APP_SECRET

I have also tried using the arn, the ${COPILOT_APPLICATION_NAME} substitution, the full name with no interpolated values, wrapping in single quotes, and wrapping in double quotes. Removing the secrets section deploys fine.

[CaptainCarpensir]

Because it gives the error YAML not well formed my guess is there's some bug in the generated yaml. Can you display it by running copilot svc package and post it here to look at?

[jorgeavaldez]

I've been getting this error output:

❯ copilot svc package -n backend -e staging
✘ generate workload backend template against environment staging: generate stack template parameters: determine image repository type: image is not supported by App Runner:

[jorgeavaldez]

I noticed this section in the template source: https://github.com/aws/copilot-cli/blob/mainline/internal/pkg/template/templates/workloads/services/rd-web/cf.yml#L101

I do have an RDS instance as an addon, the above looks like it's injecting the secret here after the custom manifest secrets

[CaptainCarpensir]

I'm still struggling to figure out exactly what's going on here. In the meantime svc package not working is its own bug so I made an issue for that one here #5634. In order to get svc package to work you can simply change the image field in the manifest like so:

image:
  # Docker build arguments.
  # For additional overrides: https://aws.github.io/copilot-cli/docs/manifest/rd-web-service/#image-build
  # build: Dockerfile
  location: public.ecr.aws/

This isn't needed for the deployment, just a workaround for the bug. Then we can see the secrets it's generating to try and figure this out!

[jorgeavaldez]

that workaround worked! thank you 😄

here's the relevant section:

  Service:
    Metadata:
      'aws:copilot:description': 'An App Runner service to run and manage your containers'
    Type: AWS::AppRunner::Service
    Properties:
      ServiceName: !Sub '${AppName}-${EnvName}-${WorkloadName}'
      SourceConfiguration:
        AuthenticationConfiguration: !If
          - NeedsAccessRole
          - AccessRoleArn: !GetAtt AccessRole.Arn
          - !Ref AWS::NoValue
        AutoDeploymentsEnabled: false
        ImageRepository:
          ImageIdentifier: !Ref ContainerImage
          ImageRepositoryType: !Ref ImageRepositoryType
          ImageConfiguration:
            Port: !Ref ContainerPort
            RuntimeEnvironmentSecrets:
              - Name: REDACTED_1
                Value:  REDACTED_1
              - Name: REDACTED_2
                Value:  REDACTED_2
            - Name: BACKENDCLUSTER_SECRET
              Value:
                Fn::GetAtt: [AddonsStack, Outputs.backendclusterSecret]
            RuntimeEnvironmentVariables:
              - Name: COPILOT_APPLICATION_NAME
                Value: !Ref AppName
              - Name: COPILOT_ENVIRONMENT_NAME
                Value: !Ref EnvName
              - Name: COPILOT_SERVICE_NAME
                Value: !Ref WorkloadName
              - Name: COPILOT_SERVICE_DISCOVERY_ENDPOINT
                Value: staging.drakula.local
              - Name: BACKENDCLUSTER_SECRET_ARN
                Value:
                  Fn::GetAtt: [ AddonsStack, Outputs.backendclusterSecret]
      InstanceConfiguration:
        Cpu: !Ref InstanceCPU
        Memory: !Ref InstanceMemory
        InstanceRoleArn: !GetAtt InstanceRole.Arn
      NetworkConfiguration:
        EgressConfiguration:
          EgressType: VPC
          VpcConnectorArn: !Ref VpcConnector
      Tags:
        - Key: copilot-application
          Value: !Ref AppName
        - Key: copilot-environment
          Value: !Ref EnvName
        - Key: copilot-service
          Value: !Ref WorkloadName

I have an addon stack for an rds cluster. it looks like that interpolated value BACKENDCLUSTER_SECRET list item under RuntimeEnvironmentSecrets has incorrect indentation

[CaptainCarpensir]

Yep! This is just indented incorrectly in the code, super quick fix I'll make for this. Sorry for the inconvenience this has caused.

            RuntimeEnvironmentSecrets:
            {{- range $name, $secret := .Secrets}}
              - Name: {{$name}}
            {{- if $secret.RequiresImport}}
                Value:
                  Fn::ImportValue: {{ quote $secret.ValueFrom }}
            {{- else}}
                Value: {{if not $secret.RequiresSub }} {{$secret.ValueFrom}} {{- else}} !Sub 'arn:${AWS::Partition}:{{$secret.Service}}:${AWS::Region}:${AWS::AccountId}:{{$secret.ValueFrom}}' {{- end}}
            {{- end}}   
            {{- end}}
            {{- end}}
            {{- if .NestedStack}}{{$stackName := .NestedStack.StackName}}
            {{- range $secret := .NestedStack.SecretOutputs}}
            - Name: {{toSnakeCase $secret}}
              Value:
                Fn::GetAtt: [{{$stackName}}, Outputs.{{$secret}}]
            {{- end}}
            {{- end}}

[jorgeavaldez]

thank you!!!

[KollaAdithya]

Hello @jorgeavaldez

Fix is now released in copilot v1.33.1
https://github.com/aws/copilot-cli/releases/tag/v1.33.1 🎉🚀

[jorgeavaldez]

Great thank you!!!