BigQueryToGCSOperator issue

[madhukar05]

Apache Airflow version:

Kubernetes version (if you are using kubernetes) (use kubectl version):

Environment:

• Cloud provider or hardware configuration:
• OS (e.g. from /etc/os-release):
• Kernel (e.g. uname -a):
• Install tools:
• Others:

What happened:

What you expected to happen:
Using Google composer and trying to save data to GCS location from Bigquery for that using BigQueryToGCSOperator. task is completed successfully but non of the file was crated in GCS.

How to reproduce it:

Anything else we need to know:

[boring-cyborg]

Thanks for opening your first issue here! Be sure to follow the issue template!

[TobKed]

@madhukar05 which version of Composer to you use?
Do you use google backport packages?
Also some example of usage BigQueryToGCSOperator (anonymised) would be very helpful to reproduce it.

[madhukar05]

here is the code i'm using...

below mentioned task completed successfully but didn't find the output file in GCS

bq_to_gcs_dag = BigQueryToGCSOperator(
task_id='bq_to_gcs_dag', gcp_conn_id='my_gcp_connection' , source_project_dataset_table=XXXXXX ,
destination_cloud_storage_uris=f"gs://XXX",
export_format='CSV',
print_header=False ,
dag=dag
)

[madhukar05]

Composer Version : composer-1.12.2-airflow-1.10.9

Here is the complete scripts which i'm using

import os

from airflow import models
from airflow.utils.dates import days_ago
from airflow.operators import bash_operator
from airflow.providers.google.cloud.transfers.bigquery_to_gcs import BigQueryToGCSOperator
from airflow.providers.google.cloud.operators.bigquery import BigQueryExecuteQueryOperator

dag = models.DAG(
dag_id='wrapper_script_for_dag',
start_date=days_ago(2),
schedule_interval=None,

)

bq_to_gcs_dag = BigQueryToGCSOperator(
task_id='bq_to_gcs_dag', gcp_conn_id='my_gcp_connection' , source_project_dataset_table=XXXXXX ,
destination_cloud_storage_uris=f"gs://XXX",
export_format='CSV',
print_header=False ,
dag=dag
)

[turbaszek]

See #9897. Solved in #9899, so it will be fixed in new providers release 🚀