Error while running plan with this GA | Terraform 1.0.8

[Sunil-Jacob]

TL;DR

Getting the below error:

Error: storage.NewClient() failed: dialing: google: error getting credentials using GOOGLE_APPLICATION_CREDENTIALS environment variable: open /home/runner/work/_temp/651d8f6a44067dceb917e2cf: no such file or directory

Expected behavior
The plan should run successfully

Observed behavior
Getting the below error:

Error: storage.NewClient() failed: dialing: google: error getting credentials using GOOGLE_APPLICATION_CREDENTIALS environment variable: open /home/runner/work/_temp/651d8f6a44067dceb917e2cf: no such file or directory

Reproduction

Action YAML

steps:
      # Checkout the repository to the GitHub Actions runner
      - name: Checkout
        uses: actions/checkout@v2

      - name: Set up Cloud SDK
        uses: google-github-actions/setup-gcloud@master
        with:
          project_id: ${{ secrets.GCP_PROJECT_ID }}
          service_account_key: ${{ secrets.GOOGLE_PRIVATE_KEY }}
          export_default_credentials: true

      - name: Bucket creation DEV
        shell: bash
        run: |
          gsutil ls -b gs://${{ secrets.GCP_PROJECT_NAME}} || (gcloud services enable storage-api.googleapis.com && gsutil mb gs://${{ secrets.GCP_PROJECT_NAME}} && gsutil versioning set on gs://${{ secrets.GCP_PROJECT_NAME}} )
      - name: plan-dev
        uses: dflook/terraform-plan@v1
        env:
          GITHUB_TOKEN : ${{ secrets.GITHUB_TOKEN }}
          TERRAFORM_HTTP_CREDENTIALS: |
            github.com/ebomart=${{ secrets.CI_USER}}:${{ secrets.CI_TOKEN}}
        with:
          path: ./
          var_file: environment/dev/variables.tfvars
          label: dev
          backend_config_file: environment/dev/gcs-bucket.tfvars

Additional information
Are you running custom workers? Doing something atypical? No

[Sunil-Jacob]

Hi Team,

This is working when I used SHA google-github-actions/setup-gcloud@b8f95eb7d716bf9a715eafd169162b93f69ed520

[aurecchia]

I just noticed the same failure. It seems #405 accidentally introduced a breaking change.

Just like @Sunil-Jacob I changed my workflow file to explicitly reference a commit SHA (the one used in the last successful build I saw) and then everything is fine.

PS: The output of the setup-gcloud action is the following:

Run google-github-actions/setup-gcloud@master
  with:
    service_account_key: ***
    export_default_credentials: true
    version: latest
  env:
    GITHUB_TOKEN: ***
/usr/bin/tar xz --warning=no-unknown-keyword --overwrite -C /home/runner/work/_temp/2a979573-6694-440f-9081-3c34bd88da32 -f /home/runner/work/_temp/2fd56f81-541b-431f-976b-14d8bc768283
Successfully exported Default Application Credentials

and the error, in the next step of the same job:

Error: storage.NewClient() failed: dialing: google: error getting credentials using GOOGLE_APPLICATION_CREDENTIALS environment variable: open /home/runner/work/_temp/e647589b58297bd498fda21d: no such file or directory

[sethvargo]

Can you share which step the error is coming from? Is it the Bucket creation DEV or the plan-dev step?

[Mowee]

@sethvargo We experience a similar issue. In our case the Default Application Credentials are passed as an environment variable to a docker image but can't be accessed inside the image. For right now I fixed the issue by not using the latest commit anymore and going back a few versions.

[sethvargo]

@Mowee can you please share your actions.yml file? I don't think the filesystem is shared with docker containers, so your situation makes sense to me

[sethvargo]

It looks like dflook/terraform-plan is also a Docker-based action

[timcooijmans]

Same issue here, working with the commit mentioned above, broken at master.

actions.yaml:

name: 'Terragrunt Review@Test'
on:
  - pull_request
env:
  tf_version: 'latest'
  tg_version: 'latest'
  tf_working_dir: './test/'
jobs:
  terragrunt:
    name: 'Terragrunt'
    runs-on: ubuntu-latest
    steps:
      - name: 'Checkout'
        uses: actions/checkout@master
      - name: Set up Google Cloud access
        uses: google-github-actions/setup-gcloud@master
        with:
          service_account_key: ${{ secrets.TEST_GCP_SERVICE_ACCOUNT_KEY }}
          export_default_credentials: true
      - name: 'Terragrunt Init'
        uses: the-commons-project/terragrunt-github-actions@master
        with:
          tf_actions_version: ${{ env.tf_version }}
          tg_actions_version: ${{ env.tg_version }}
          tf_actions_subcommand: 'init'
          tf_actions_working_dir: ${{ env.tf_working_dir }}
          tf_actions_comment: false
        env:
          TF_INPUT: false

[sethvargo]

It looks like Docker-based actions don't mount RUNNER_TEMP, so the credentials aren't available on the shared filesystem.

[timcooijmans]

Oh, I agree this is a difficult case, but I don't think it should suddenly break like this. Is there an alternative by using the credentials_file_path option to force it back?

[Mowee]

@sethvargo Sure we are using this workflow:

name: Rotate GCP service account key
#on:
#  schedule:
#    - cron: '0 12 * * 1'
on: [workflow_dispatch]
jobs:
  rotate:
    name: rotate gcp key
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v2.0.0
      - name: Setup GCP Service Account
        uses: google-github-actions/setup-gcloud@master
        with:
          service_account_key: ${{ secrets.GCP_SA_INFRA_KEY }}
          export_default_credentials: true
      - name: rotate gcp keys
        uses: miklosn/github-action-rotate-gcp-key@main
        with:
          projectId: "example"
          serviceAccount: "github-actions@example.iam.gserviceaccount.com"
          personalAccessToken: "${{ secrets.PERSONAL_ACCESS_TOKEN }}"
          repositories: ${{ github.repository }}
          secretName: "GCP_SA_INFRA_KEY"

The docker image runs as follows: /usr/bin/docker run [..] -e GOOGLE_APPLICATION_CREDENTIALS

I switched our workflow to google-github-actions/setup-gcloud@b8f95eb7d716bf9a715eafd169162b93f69ed520 and it works.

[sethvargo]

@timcooijmans yea - I'm just documenting my findings here. The reason for the change in #405 was largely around security: on self-hosted runners, the GITHUB_WORKSPACE is not removed. So you can accidentally re-use credentials, or, worse, have credentials available inside a run where they should not be present. Current workarounds are to pin to the older sha (google-github-actions/setup-gcloud@b8f95eb7d716bf9a715eafd169162b93f69ed520) or to set credentials_file_path to something inside ${{github.workspace}}.

We'll figure something out by EOD. I think the most likely path forward is to revert using RUNNER_TEMP and put it back in GITHUB_WORKSPACE, but then add a post-step to ensure the file is deleted. But I want to wait for some folks on the team to wake up and validate that approach 😄

[aurecchia]

Thank you for your help and explanation @sethvargo 👍