Skip to content

Conversation

@Lee-W
Copy link
Member

@Lee-W Lee-W commented Jul 3, 2023

What's the issue?

When waiting for a long-running data factory pipeline run, it's possible that the Azure connection is expired and will need to be refreshed. This happens in (AzureDataFactoryPipelineRunStatusSensor(..., deferrable=True), and AzureDataFactoryRunPipelineOperator which will need changes in the following code sections.

AzureDataFactoryPipelineRunStatusSensor(..., deferrable=True)

pipeline_status = await hook.get_adf_pipeline_run_status(
run_id=self.run_id,
resource_group_name=self.resource_group_name,
factory_name=self.factory_name,
)

AzureDataFactoryRunPipelineOperator(..., deferrable=True)

pipeline_status = await hook.get_adf_pipeline_run_status(
run_id=self.run_id,
resource_group_name=self.resource_group_name,
factory_name=self.factory_name,
)

AzureDataFactoryRunPipelineOperator(..., deferrable=False)

pipeline_run_status = self.get_pipeline_run_status(**pipeline_run_info)

What's changed?

Re-establish connection if an exception is caught in these sections, and set executed_after_refresh_token to False. If the exception does not relate to connection refresh, then the next round of status checking will still raise an exception and won't refresh the toekn


^ Add meaningful description above

Read the Pull Request Guidelines for more information.
In case of fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in a newsfragment file, named {pr_number}.significant.rst or {issue_number}.significant.rst, in newsfragments.

@Lee-W Lee-W force-pushed the refresh-conn-for-data-factory-operator branch 2 times, most recently from 26248fb to 36620f4 Compare July 3, 2023 11:02
@Lee-W Lee-W marked this pull request as ready for review July 3, 2023 11:05
@Lee-W Lee-W force-pushed the refresh-conn-for-data-factory-operator branch from 36620f4 to 42d81b7 Compare July 3, 2023 11:29
@Lee-W Lee-W changed the title refresh connection if exception is caugh in "AzureDataFactory" refresh connection if an exception is caught in "AzureDataFactory" Jul 3, 2023
…FactoryAsyncHook get_pipeline_run and get_adf_pipeline_run_status

These exception should be caught in the triggerer instead.
@Lee-W Lee-W force-pushed the refresh-conn-for-data-factory-operator branch from d6323f9 to 79195e7 Compare July 4, 2023 04:56
@potiuk potiuk merged commit 1d6f613 into apache:main Jul 4, 2023
@pankajastro pankajastro deleted the refresh-conn-for-data-factory-operator branch July 4, 2023 20:09
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants