Skip to content

[AIRFLOW-4095] Add template_fields for S3CopyObjectOperator & S3DeleteObjectsOperator #4920

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
XD-DENG merged 4 commits into from
Mar 16, 2019
Merged

[AIRFLOW-4095] Add template_fields for S3CopyObjectOperator & S3DeleteObjectsOperator #4920

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
a2f042c
[AIRFLOW-4095]Added some params as template fields
yaswanthrajyadiki Mar 15, 2019
8ca0cdf
[AIRFLOW-4095]Added some params as template fields
yaswanthrajyadiki Mar 15, 2019
84dbf28
[AIRFLOW-4095]Removed extra space
yaswanthrajyadiki Mar 15, 2019
443b648
[AIRFLOW-4095] Removed extra space
yaswanthrajyadiki Mar 15, 2019
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
11 changes: 7 additions & 4 deletions airflow/contrib/operators/s3_copy_object_operator.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -29,21 +29,21 @@ class S3CopyObjectOperator(BaseOperator):
Note: the S3 connection used here needs to have access to both
source and destination bucket/key.

:param source_bucket_key: The key of the source object.
:param source_bucket_key: The key of the source object. (templated)

It can be either full s3:// style url or relative path from root level.

When it's specified as a full s3:// url, please omit source_bucket_name.
:type source_bucket_key: str
:param dest_bucket_key: The key of the object to copy to.
:param dest_bucket_key: The key of the object to copy to. (templated)

The convention to specify `dest_bucket_key` is the same as `source_bucket_key`.
:type dest_bucket_key: str
:param source_bucket_name: Name of the S3 bucket where the source object is in.
:param source_bucket_name: Name of the S3 bucket where the source object is in. (templated)

It should be omitted when `source_bucket_key` is provided as a full s3:// url.
:type source_bucket_name: str
:param dest_bucket_name: Name of the S3 bucket to where the object is copied.
:param dest_bucket_name: Name of the S3 bucket to where the object is copied. (templated)

It should be omitted when `dest_bucket_key` is provided as a full s3:// url.
:type dest_bucket_name: str
Expand All @@ -65,6 +65,9 @@ class S3CopyObjectOperator(BaseOperator):
:type verify: bool or str
"""

template_fields = ('source_bucket_key', 'dest_bucket_key',
'source_bucket_name', 'dest_bucket_name')

@apply_defaults
def __init__(
self,
Expand Down
6 changes: 4 additions & 2 deletions airflow/contrib/operators/s3_delete_objects_operator.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -30,9 +30,9 @@ class S3DeleteObjectsOperator(BaseOperator):

Users may specify up to 1000 keys to delete.

:param bucket: Name of the bucket in which you are going to delete object(s)
:param bucket: Name of the bucket in which you are going to delete object(s). (templated)
:type bucket: str
:param keys: The key(s) to delete from S3 bucket.
:param keys: The key(s) to delete from S3 bucket. (templated)

When ``keys`` is a string, it's supposed to be the key name of
the single object to delete.
Expand All @@ -58,6 +58,8 @@ class S3DeleteObjectsOperator(BaseOperator):
:type verify: bool or str
"""

template_fields = ('keys', 'bucket')

@apply_defaults
def __init__(
self,
Expand Down