Skip to content

Conversation

@VladaZakharova
Copy link
Contributor

When trying to attach to RUNNING (but also any other state too) BQ Job using BigQueryInsertJobOperator, even though reattach_states={'PENDING', 'RUNNING'} parameter is set and Job is still running, ValueError: Job already begun is thrown.

Attaching to DONE job, which according to documentation shouldn't be possible (from airflow doc: if job with given id already exists then it tries to reattach to the job if its not done and its state is in reattach_states. If the job is done the operator will raise AirflowException.), but throws an error, as well as with any other Job type because of the validation in job._begin() call and fails if it's different than None.

This PR fixes this by adding checks of the current state before going to execute the Job in deferrable or synch mode.


^ Add meaningful description above
Read the Pull Request Guidelines for more information.
In case of fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in a newsfragment file, named {pr_number}.significant.rst or {issue_number}.significant.rst, in newsfragments.

@boring-cyborg boring-cyborg bot added area:providers provider:google Google (including GCP) related issues labels Jul 9, 2024
@VladaZakharova
Copy link
Contributor Author

Hi @potiuk @Lee-W ! Can you please check changes in this PR? Thanks!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area:providers provider:google Google (including GCP) related issues

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants