unexpected end of JSON input error when trying to register driver

[bunyk]

Bug Description

Hello! I'm trying to connect to Google cloudsql Mariadb database using worload identity federation from AWS. Here is the code I run:

func connectWithProxy(conf config.GCPDB) (*sqlx.DB, error) {
	if os.Getenv("GOOGLE_APPLICATION_CREDENTIALS") == "" {
		// just a check to make sure credentials.DetectDefault will detect correct file
		return nil, fmt.Errorf("GOOGLE_APPLICATION_CREDENTIALS environment variable not set")
	}

	creds, err := credentials.DetectDefault(&credentials.DetectOptions{
		Scopes: []string{storage.CloudPlatformScope},
	})
	if err != nil {
		return nil, errl.Source(err)
	}

	if _, err := mysql.RegisterDriver("cloudsql-mysql", cloudsqlconn.WithCredentials(creds)); err != nil {
		return nil, errl.Source(err)
	}
	dbConn := fmt.Sprintf(
		"%s:%s@cloudsql-mysql(%s)/%s?parseTime=true",
		conf.User, conf.Password, conf.Host, conf.Name,
	)
	db, err := sql.Open(
		"cloudsql-mysql",
		dbConn,
	)
	if err != nil {
		return nil, errl.Source(err)
	}
	return sqlx.NewDb(db, "mysql"), nil
}

But it gives me error failed to get instance: refresh error: failed to get instance metadata (connection name = "ccs-wfm:europe-west3:ccs-wfm"): Get "https://sqladmin.googleapis.com/sql/v1beta4/projects/ccs-wfm/instances/ccs-wfm/connectSettings?alt=json&prettyPrint=false\": unexpected end of JSON input

Example code (or command)

Stacktrace

Steps to reproduce?

1. ?
2. ?
3. ?
   ...

Environment

1. OS type and version: alpine-3.12:latest
2. Go version: 1.24.4
3. Cloud SQL Go Connector version: v1.18.1

Additional Details

No response

[kgala2]

Hi @bunyk,

Based on the request flow, one possibility for the unexpected end of JSON input error is that the request from your application is being blocked at the Google Front End (GFE), preventing it from reaching the Cloud SQL Admin API. This can happen if there's an issue with the Workload Identity Federation token or the project configuration.

A typical flow looks like

client app -- [token, request] -- ... -- GFE  -- SqlAdminGlobal -- SqlAdmin Regional -- Spanner 

To verify this, could you please perform the following checks?

1. Ensure the Cloud SQL Admin API is enabled: Make sure that sqladmin.googleapis.com is enabled for your project.

2. Check for interfering proxies: Confirm that there isn't a local proxy or firewall rule that might be redirecting or blocking traffic to the sqladmin.googleapis.com endpoint.

3. Validate your authentication token: A simple way to test if your token and permissions are valid is to list the Cloud SQL instances in your project using gcloud.
   gcloud sql instances list --project=proj

If you are able to see the correct output for these commands, it would indicate that the fundamental configuration is likely correct. In that case, the next step would be to enable debug logging in the connector, you can do this using the cloudsqlconn.WithDebugContextLogger(...) option. With those logs, we can help you debug this further.

[bunyk]

Thing is, I managed to connect using same IDENTITY_POOL_PROVIDER config as pointed by GOOGLE_APPLICATION_CREDENTIALS in python (guided by my companys GCP team internal documentation) and it worked fine:

import os
from boto3 import Session
from google.auth import aws
from google.auth.transport import requests as google_requests
from typing import List
from google.cloud.sql.connector import Connector
import sqlalchemy
import pymysql

# DB_* creds and  IDENTITY_POOL_PROVIDER value omitted for brevity & security

def main():
    gcp_credentials = get_gcp_credentials_via_aws_federation()

    connector = Connector(credentials=gcp_credentials)
    
    engine = sqlalchemy.create_engine(
        "mysql+pymysql://",
        creator=lambda: connector.connect(
            DB_HOST,
            "pymysql",
            user=DB_USER,
            password=DB_PASS,
            db=DB_NAME,
        ),
    )
    
    with engine.connect() as connection:
        result = connection.execute(sqlalchemy.text("SHOW TABLES"))
        tables = result.fetchall()
        for t in tables:
	        print(t)

def get_gcp_credentials_via_aws_federation(
        scopes: List[str] = ["https://www.googleapis.com/auth/cloud-platform"]
) -> aws.Credentials:
    _set_aws_credentials_to_environment()
    credentials = aws.Credentials.from_info(IDENTITY_POOL_PROVIDER).with_scopes(scopes=scopes)
    request = google_requests.Request()
    credentials.refresh(request)
    return credentials


def _set_aws_credentials_to_environment():
    session = Session()
    credentials = session.get_credentials()
    
    os.environ["AWS_ACCESS_KEY_ID"] = credentials.access_key
    os.environ["AWS_SECRET_ACCESS_KEY"] = credentials.secret_key
    os.environ["AWS_SESSION_TOKEN"] = credentials.token
    os.environ["AWS_REGION"] = session.region_name or "eu-central-1"

main()

It breaks on credentials.refresh(request) through in similar way, trying to parse JSON from empty response, if I don't setup this AWS_* env stuff.

Now I just need to translate this to Golang, since that's the language of all the microservices. I assume Go code breaks too because it is not authenticated with AWS access key, just not sure how to obtain it here.

[bunyk]

As for more detailed logs, they just show that it fails ruing connection info refresh:

time="2025-09-09T06:45:07Z" level=debug msg="[ccs-wfm:europe-west3:ccs-wfm] Connection info added to cache"
time="2025-09-09T06:45:07Z" level=debug msg="[ccs-wfm:europe-west3:ccs-wfm] Connection info refresh operation started"
time="2025-09-09T06:45:08Z" level=debug msg="[ccs-wfm:europe-west3:ccs-wfm] Connection info refresh operation failed, err = failed to get instance: refresh error: failed to get instance metadata (connection name = \"ccs-wfm:europe-west3:ccs-wfm\"): Get \"https://sqladmin.googleapis.com/sql/v1beta4/projects/ccs-wfm/instances/ccs-wfm/connectSettings?alt=json&prettyPrint=false\": unexpected end of JSON input"
time="2025-09-09T06:45:08Z" level=debug msg="[ccs-wfm:europe-west3:ccs-wfm] Connection info refresh operation scheduled immediately"
time="2025-09-09T06:45:08Z" level=debug msg="[ccs-wfm:europe-west3:ccs-wfm] Connection info refresh operation started"
time="2025-09-09T06:45:08Z" level=debug msg="[ccs-wfm:europe-west3:ccs-wfm] Removing connection info from cache: failed to get instance: refresh error: failed to get instance metadata (connection name = \"ccs-wfm:europe-west3:ccs-wfm\"): Get \"https://sqladmin.googleapis.com/sql/v1beta4/projects/ccs-wfm/instances/ccs-wfm/connectSettings?alt=json&prettyPrint=false\": unexpected end of JSON input"

[kgala2]

Hi @bunyk ,

One of the reasons I could see this error coming up is if the environment and application default credentials are not set properly in Go, can you verify if

Go equivalent of this in your application is setting the environment correctly?

def get_gcp_credentials_via_aws_federation(
        scopes: List[str] = ["https://www.googleapis.com/auth/cloud-platform"]
) -> aws.Credentials:
    _set_aws_credentials_to_environment()
    credentials = aws.Credentials.from_info(IDENTITY_POOL_PROVIDER).with_scopes(scopes=scopes)
    request = google_requests.Request()
    credentials.refresh(request)
    return credentials


def _set_aws_credentials_to_environment():
    session = Session()
    credentials = session.get_credentials()
    
    os.environ["AWS_ACCESS_KEY_ID"] = credentials.access_key
    os.environ["AWS_SECRET_ACCESS_KEY"] = credentials.secret_key
    os.environ["AWS_SESSION_TOKEN"] = credentials.token
    os.environ["AWS_REGION"] = session.region_name or "eu-central-1"