Skip to content

Doc-only: mention minimum boto3 1.34.52 for AWS provider when using Batch ecs_properties_override #39983

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
ferruzzi merged 6 commits into from
Jun 4, 2024
Merged

Doc-only: mention minimum boto3 1.34.52 for AWS provider when using Batch ecs_properties_override #39983

Changes from all commits
Commits
Show all changes
6 commits
Select commit Hold shift + click to select a range
07f451a
bump minimum boto3 1.34.52 for AWS provider
yehoshuadimarsky May 31, 2024
f2adb9d
update some deps
yehoshuadimarsky May 31, 2024
07c9e3b
Doc-only change: AWS Batch operator requires boto3 1.34.52+ for 'ecs_…
yehoshuadimarsky Jun 3, 2024
5006e79
Merge branch 'main' into aws-bump-boto3
yehoshuadimarsky Jun 4, 2024
122f12b
warn if ecsPropertiesOverride not recognized in Batch API call
yehoshuadimarsky Jun 4, 2024
dbeb95a
tweak message
yehoshuadimarsky Jun 4, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
20 changes: 19 additions & 1 deletion airflow/providers/amazon/aws/operators/batch.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -30,6 +30,8 @@
from functools import cached_property
from typing import TYPE_CHECKING, Any, Sequence

import botocore

from airflow.configuration import conf
from airflow.exceptions import AirflowException, AirflowProviderDeprecationWarning
from airflow.models import BaseOperator
Expand All @@ -45,7 +47,11 @@
BatchCreateComputeEnvironmentTrigger,
BatchJobTrigger,
)
from airflow.providers.amazon.aws.utils import trim_none_values, validate_execute_complete_event
from airflow.providers.amazon.aws.utils import (
get_botocore_version,
trim_none_values,
validate_execute_complete_event,
)
from airflow.providers.amazon.aws.utils.task_log_fetcher import AwsTaskLogFetcher
from airflow.utils.types import NOTSET

Expand All @@ -66,6 +72,7 @@ class BatchOperator(BaseOperator):
:param overrides: DEPRECATED, use container_overrides instead with the same value.
:param container_overrides: the `containerOverrides` parameter for boto3 (templated)
:param ecs_properties_override: the `ecsPropertiesOverride` parameter for boto3 (templated)
**NOTE** This requires `boto3` version 1.34.52+
:param node_overrides: the `nodeOverrides` parameter for boto3 (templated)
:param share_identifier: The share identifier for the job. Don't specify this parameter if the job queue
doesn't have a scheduling policy.
Expand Down Expand Up @@ -323,6 +330,17 @@ def submit_job(self, context: Context):

try:
response = self.hook.client.submit_job(**trim_none_values(args))
except botocore.exceptions.ParamValidationError as error:
if (
'Unknown parameter in input: "ecsPropertiesOverride"' in str(error)
) and self.ecs_properties_override:
self.log.error(
"You are attempting to use ecsPropertiesOverride and the botocore API returned an "
"error message which may indicate the need to update botocore to do this. \n"
"Support for using ecsPropertiesOverride was added in botocore 1.34.52 and you are using botocore %s",
".".join(map(str, get_botocore_version())),
)
raise
except Exception as e:
self.log.error(
"AWS Batch job failed submission - job definition: %s - on queue %s",
Expand Down