-
Notifications
You must be signed in to change notification settings - Fork 16.4k
Closed
Labels
area:providerskind:bugThis is a clearly a bugThis is a clearly a bugprovider:googleGoogle (including GCP) related issuesGoogle (including GCP) related issues
Description
Apache Airflow version
Other Airflow 2 version (please specify below)
What happened
Airflow version: 2.3.4 (Composer 2.1.12)
I've been using the DataprocCreateBatchOperator with the deferrable=True option. It worked well for the past few months, but an error started appearing on June 21, 2023, at 16:51 UTC. The error message is as follows:
grpc.aio._call.AioRpcError: <AioRpcError of RPC that terminated with:
status = StatusCode.INVALID_ARGUMENT
details = "Request contains an invalid argument."
debug_error_string = "UNKNOWN:Error received from peer ipv4:74.125.69.95:443 {grpc_message:"Request contains an invalid argument.", grpc_status:3, created_time:"2023-06-21T16:51:22.992951359+00:00"}"
What you think should happen instead
The name argument in the hook code follows the format "projects/PROJECT_ID/regions/DATAPROC_REGION/batches/BATCH_ID". However, according to the Google Cloud DataProc API Reference, it should be in the format "projects/PROJECT_ID/locations/DATAPROC_REGION/batches/BATCH_ID .
How to reproduce
just run a dataproc operator like this
create_batch = DataprocCreateBatchOperator(
task_id="create_batch",
batch_id="batch_test",
deferrable=True,
)
Operating System
Ubuntu 20.04
Versions of Apache Airflow Providers
No response
Deployment
Google Cloud Composer
Deployment details
Composer 2.1.12
Anything else
No response
Are you willing to submit PR?
- Yes I am willing to submit a PR!
Code of Conduct
- I agree to follow this project's Code of Conduct
Metadata
Metadata
Assignees
Labels
area:providerskind:bugThis is a clearly a bugThis is a clearly a bugprovider:googleGoogle (including GCP) related issuesGoogle (including GCP) related issues