Updated logic to allow AWS Batch Hook get_job_description retries to be more effective
#37552
+11
−5
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Update logic in
Amazon/awsproviders forbatch_clienthook'sparse_job_descriptionfunction. This function is ONLY called within theget_job_descriptionfunction that allows for retries; however, there is only handling ofbotocore.exceptions.ClientErrorerror in this function.This means, that if AWS simply fails to provide the job description - if there is a delay in AWS to register job information, this function will immediate fail instead of trying. This may happen under certain conditions, especially when using the
batchoperators for submitting jobs which immediately use returnedjob_idto query for the job information - but again, if AWS fails to register the job description in time - this will cause an unnecessary failure in Airflow side of things.The proposed change allows
parse_job_descriptionto returnNoneinstead of raising if zero jobs are returned by AWS. Similarly, if more than one matching job is returned, the function will provide the description of the first job.