[AIRFLOW-6245] Add custom waiters for AWS batch jobs #6811

dazza-codes · 2019-12-13T01:58:42Z

Design / PR notes and RFC

open to commentary on the general design and approach applied in this PR
- the intention is to preserve existing functionality, as is, with minor tweaks
AFAICT, the public API is backwards compatible, unless noted in UPDATING.md
- most changes are refactoring private methods
- this adds new features to the public API
for RFC, please ignore most details of the implementation, just add commentary to the conversation; if a slack discussion would help, ping me to setup a time (Pacific standard time)
if it mostly looks OK, go ahead and give it a review
- LMK if it would be easier to split this out into stacked PRs, but hopefully it's clear enough, although large
- LMK if this requires additional integration testing (and how to do that easily)

Jira

My PR addresses the following Airflow Jira issues and references them in the PR title.
- https://issues.apache.org/jira/browse/AIRFLOW-6245

Related issues

Custom callable for waiter boto/botocore#1915
- [1915] Use delay function in waiter boto/botocore#1921
- While working on this PR, I also pushed up [1915] Use delay function in waiter boto/botocore#1921 and I'm not entirely sure how this PR and that one can change the landscape of waiting/polling on batch jobs, but this PR for Airflow will likely stand on its own for Airflow custom options
While developing this PR, thoughts about custom options for async waiters lingered in the background. This PR may not be the place to introduce that (it might follow on from this PR), but it's worth noting here that work on a base async operator is in [AIRFLOW-5567] BaseReschedulePokeOperator #6210

Description

Here are some details about my PR:

This PR improves the batch job status polling by using exponential backoff with jitter, see

https://aws.amazon.com/blogs/architecture/exponential-backoff-and-jitter/

This PR creates a utility for generating and using custom waiters for the AWS batch service; see also:

https://www.2ndwatch.com/blog/use-waiters-boto3-write/
https://boto3.amazonaws.com/v1/documentation/api/latest/guide/clients.html#waiters
This PR removes a dependency on a botocore change to the waiter for AWS batch jobs that has not been merged and released for several years, i.e.
- Adding AWSBatch JobExists and JobExecutionComplete waiter boto/botocore#1307

The utility for creating custom waiters can accept a waiter config to generate a custom waiter model. This should allow users to specify any waiter model required for their use cases, which should satisfy https://issues.apache.org/jira/browse/AIRFLOW-6245 when the custom waiter can be plugged into the AWS batch operator.

Tests

My PR adds unit tests for the generator of custom waiters for the AWS batch service

Commits

My commits all reference Jira issues in their subject lines, and I have squashed multiple commits if they address the same issue. In addition, my commits follow the guidelines from "How to write a good git commit message"

Documentation

In case of new functionality, my PR adds documentation that describes how to use it.
- All the public functions and the classes in the PR contain docstrings that explain what it does
- If you implement backwards incompatible changes, please leave a note in the Updating.md so we can assign it to a appropriate release

turbaszek

There is too much changes. Could you please split the Pr into two separate:

move AWS from contrib to providers
add custom waiters

In this way it will be easier to review and chance that we will miss something will be lower ;)

tests/providers/amazon/aws/operators/test_batch.py

dazza-codes · 2019-12-16T17:46:15Z

@nuclearpinguin - this is already a stacked PR, please read the PR description (esp. WIP status) and linked PRs. If your interested, please take a look at #6764

dazza-codes · 2019-12-17T17:50:12Z

#6764 is merged - rebased this PR on latest master today

still WIP to consider and test new options for custom waiters (updates to WIP status are in the PR title and description)
pushed up [1915] Use delay function in waiter boto/botocore#1921 and not sure exactly how this PR and that one can change the landscape of waiting/polling on batch jobs, but this PR will likely stand on its own for Airflow custom options; both PRs need more work to sort out CI-test issues

dazza-codes · 2019-12-27T15:37:12Z

Rebased and fixed test imports after the merge of #6919 - only changes to imports, no substantive code changes required.

airflow/providers/amazon/aws/hooks/batch_client.py

airflow/providers/amazon/aws/hooks/batch_waiters.py

tests/test_core_to_contrib.py

airflow/providers/amazon/aws/operators/batch.py

tests/providers/amazon/aws/hooks/test_batch_waiters.py

dazza-codes · 2020-01-06T23:36:04Z

Appreciate the reviews - thanks @feluelle. I've rebased on master (went OK) and pushed up some requested changes (some open comments remain).

dazza-codes · 2020-01-07T18:21:38Z

Rebased on master and pushed again because some test failures were not related to this PR and might be fixed in the latest master.

feluelle · 2020-01-07T19:32:35Z

I restarted the failed check since the error was a known one: travis machine ran out of memory -.-

dazza-codes · 2020-01-07T23:04:02Z

Rebased and pushed again, travis failed one test with a timeout or something, not something specific to this PR.

- add AwsBatchWaiters - the waiters are based on botocore, but not yet available for AWS batch services - refactor AwsBatchOperator: - use an optional waiters object to wait for batch job status indicators - split execute into submit_job and monitor_job - use job_id with an optional init-parameter; discard jobId and jobName (already has job_name) - inherit from AwsBatchClient - add notes to UPDATING.md - extract class for AwsBatchClient - move responsibility for batch API calls and response parsing to this client - move responsibility for default wait and polling to this client - rename BatchProtocol to AwsBatchProtocol [AIP-21] - test backward compatibility - add PROTOCOLS to tests/test_core_to_contrib.py - add notes to UPDATING.md - split up polling for job status into steps: - poll for a JobExists - poll for a JobRunning - poll for a JobComplete - use random jitter for wait-polling delays for high concurrency job polling - modify the exponential backoff delay for the existing polling functions - revise and update unit tests

dazza-codes · 2020-01-08T04:42:32Z

Did another self-review and most things look OK and test OK. There is a minor revision to how the client polls the AWS Batch job description, so that it will fail-fast when it encounters most client errors (except one for too-many-requests, then it will retry). I've briefly considered creating an extract-class refactor to pull out an AwsBatchJob class that captures only the job monitoring responsibility (possibly a persistent state-machine for the job), but skipping it because this PR already trims down the operator enough and that work could follow this PR if it helps later on.

feluelle

Great Work @dazza-codes ! 👍

- add AwsBatchWaiters - the waiters are based on botocore, but not yet available for AWS batch services - refactor AwsBatchOperator: - use an optional waiters object to wait for batch job status indicators - split execute into submit_job and monitor_job - use job_id with an optional init-parameter; discard jobId and jobName (already has job_name) - inherit from AwsBatchClient - add notes to UPDATING.md - extract class for AwsBatchClient - move responsibility for batch API calls and response parsing to this client - move responsibility for default wait and polling to this client - rename BatchProtocol to AwsBatchProtocol [AIP-21] - test backward compatibility - add PROTOCOLS to tests/test_core_to_contrib.py - add notes to UPDATING.md - split up polling for job status into steps: - poll for a JobExists - poll for a JobRunning - poll for a JobComplete - use random jitter for wait-polling delays for high concurrency job polling - modify the exponential backoff delay for the existing polling functions - revise and update unit tests

dazza-codes · 2021-04-15T18:21:14Z

In https://issues.apache.org/jira/browse/AIRFLOW-6245 - this work is attached to a 2.0.0 release. How can we request that this is included in the next 1.x release?

eladkal · 2021-04-15T19:58:01Z

@dazza-codes for 1.10.X you should use https://pypi.org/project/apache-airflow-backport-providers-amazon/

dazza-codes force-pushed the aws-batch-waiters branch from 4da6ba0 to 05a6960 Compare December 13, 2019 02:34

mik-laj added the provider:amazon AWS/Amazon - related issues label Dec 13, 2019

dazza-codes force-pushed the aws-batch-waiters branch 3 times, most recently from afc1be7 to c59d351 Compare December 14, 2019 03:10

turbaszek requested changes Dec 14, 2019

View reviewed changes

tests/providers/amazon/aws/operators/test_batch.py Outdated Show resolved Hide resolved

dazza-codes changed the title ~~[WIP][AIRFLOW-6245] Add custom waiters for AWS batch jobs~~ [WIP, staked on 6764][AIRFLOW-6245] Add custom waiters for AWS batch jobs Dec 16, 2019

dazza-codes changed the title ~~[WIP, staked on 6764][AIRFLOW-6245] Add custom waiters for AWS batch jobs~~ [WIP, stacked on 6764][AIRFLOW-6245] Add custom waiters for AWS batch jobs Dec 16, 2019

dazza-codes force-pushed the aws-batch-waiters branch 2 times, most recently from ec32c66 to dd7f211 Compare December 17, 2019 16:52

dazza-codes changed the title ~~[WIP, stacked on 6764][AIRFLOW-6245] Add custom waiters for AWS batch jobs~~ [WIP][AIRFLOW-6245] Add custom waiters for AWS batch jobs Dec 17, 2019

dazza-codes force-pushed the aws-batch-waiters branch 3 times, most recently from 92b78f0 to c0bc140 Compare December 18, 2019 22:06

dazza-codes changed the title ~~[WIP][AIRFLOW-6245] Add custom waiters for AWS batch jobs~~ [AIRFLOW-6245] Add custom waiters for AWS batch jobs Dec 18, 2019

dazza-codes force-pushed the aws-batch-waiters branch from c0bc140 to 9dde67f Compare December 19, 2019 15:37

dazza-codes changed the title ~~[AIRFLOW-6245] Add custom waiters for AWS batch jobs~~ [WIP][AIRFLOW-6245] Add custom waiters for AWS batch jobs Dec 19, 2019

dazza-codes force-pushed the aws-batch-waiters branch 9 times, most recently from 6fc9fe5 to 64c0232 Compare December 21, 2019 02:55

dazza-codes changed the title ~~[WIP][AIRFLOW-6245] Add custom waiters for AWS batch jobs~~ [WIP][RFC][AIRFLOW-6245] Add custom waiters for AWS batch jobs Dec 21, 2019

dazza-codes changed the title ~~[WIP][RFC][AIRFLOW-6245] Add custom waiters for AWS batch jobs~~ [RFC][AIRFLOW-6245] Add custom waiters for AWS batch jobs Dec 21, 2019

turbaszek requested a review from mik-laj December 24, 2019 08:06

dazza-codes force-pushed the aws-batch-waiters branch from 78ca43c to 2370419 Compare December 27, 2019 15:36

dazza-codes force-pushed the aws-batch-waiters branch from 2370419 to 91bd6a2 Compare December 27, 2019 16:32

feluelle reviewed Dec 27, 2019

View reviewed changes

dazza-codes force-pushed the aws-batch-waiters branch from 91bd6a2 to 1a49bfa Compare December 28, 2019 17:40

dazza-codes requested a review from feluelle December 28, 2019 17:47

dazza-codes force-pushed the aws-batch-waiters branch from 1a49bfa to 4153ccd Compare December 28, 2019 20:47

dazza-codes changed the title ~~[RFC][AIRFLOW-6245] Add custom waiters for AWS batch jobs~~ [AIRFLOW-6245] Add custom waiters for AWS batch jobs Dec 28, 2019

dazza-codes force-pushed the aws-batch-waiters branch from 4153ccd to ba14be3 Compare December 31, 2019 18:53

github-actions bot added the area:docs label Jan 5, 2020

feluelle reviewed Jan 6, 2020

View reviewed changes

dazza-codes force-pushed the aws-batch-waiters branch from ba14be3 to f0e849c Compare January 6, 2020 23:35

dazza-codes force-pushed the aws-batch-waiters branch from f0e849c to b6af4cd Compare January 7, 2020 18:21

dazza-codes force-pushed the aws-batch-waiters branch from b6af4cd to fd2ed5a Compare January 7, 2020 20:33

dazza-codes force-pushed the aws-batch-waiters branch from fd2ed5a to 302d536 Compare January 8, 2020 04:27

feluelle approved these changes Jan 8, 2020

View reviewed changes

feluelle merged commit 78d8fe6 into apache:master Jan 8, 2020

dazza-codes deleted the aws-batch-waiters branch January 8, 2020 15:20

dazza-codes mentioned this pull request Feb 7, 2020

Adding AWSBatch JobExists and JobExecutionComplete waiter boto/botocore#1307

Closed

[AIRFLOW-6245] Add custom waiters for AWS batch jobs #6811

[AIRFLOW-6245] Add custom waiters for AWS batch jobs #6811

Uh oh!

Conversation

dazza-codes commented Dec 13, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Design / PR notes and RFC

Jira

Related issues

Description

Tests

Commits

Documentation

Uh oh!

turbaszek left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

dazza-codes commented Dec 16, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

dazza-codes commented Dec 17, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

dazza-codes commented Dec 27, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

dazza-codes commented Jan 6, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

dazza-codes commented Jan 7, 2020

Uh oh!

feluelle commented Jan 7, 2020

Uh oh!

dazza-codes commented Jan 7, 2020

Uh oh!

dazza-codes commented Jan 8, 2020

Uh oh!

feluelle left a comment

Choose a reason for hiding this comment

Uh oh!

dazza-codes commented Apr 15, 2021

Uh oh!

eladkal commented Apr 15, 2021

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

dazza-codes commented Dec 13, 2019 •

edited

Loading

turbaszek left a comment •

edited

Loading

dazza-codes commented Dec 16, 2019 •

edited

Loading

dazza-codes commented Dec 17, 2019 •

edited

Loading

dazza-codes commented Dec 27, 2019 •

edited

Loading

dazza-codes commented Jan 6, 2020 •

edited

Loading