Skip to content

AioRpcError in DataprocCreateBatchOperator #32069

@HaeSe0ng

Description

@HaeSe0ng

Apache Airflow version

Other Airflow 2 version (please specify below)

What happened

Airflow version: 2.3.4 (Composer 2.1.12)

I've been using the DataprocCreateBatchOperator with the deferrable=True option. It worked well for the past few months, but an error started appearing on June 21, 2023, at 16:51 UTC. The error message is as follows:

grpc.aio._call.AioRpcError: <AioRpcError of RPC that terminated with:
	status = StatusCode.INVALID_ARGUMENT
	details = "Request contains an invalid argument."
	debug_error_string = "UNKNOWN:Error received from peer ipv4:74.125.69.95:443 {grpc_message:"Request contains an invalid argument.", grpc_status:3, created_time:"2023-06-21T16:51:22.992951359+00:00"}"

What you think should happen instead

The name argument in the hook code follows the format "projects/PROJECT_ID/regions/DATAPROC_REGION/batches/BATCH_ID". However, according to the Google Cloud DataProc API Reference, it should be in the format "projects/PROJECT_ID/locations/DATAPROC_REGION/batches/BATCH_ID .

How to reproduce

just run a dataproc operator like this

create_batch = DataprocCreateBatchOperator(
        task_id="create_batch",
        batch_id="batch_test",
        deferrable=True,
)

Operating System

Ubuntu 20.04

Versions of Apache Airflow Providers

No response

Deployment

Google Cloud Composer

Deployment details

Composer 2.1.12

Anything else

No response

Are you willing to submit PR?

  • Yes I am willing to submit a PR!

Code of Conduct

Metadata

Metadata

Assignees

Labels

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions