Skip to content

Convert S3 example DAG to System test (AIP-47) #26535

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
potiuk merged 3 commits into from
Sep 21, 2022
Merged

Convert S3 example DAG to System test (AIP-47) #26535

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
1f4ebe5
S3 system test (AIP-47) using SystemTestContextBuilder
syedahsn Sep 13, 2022
23f0d1f
Move constants to module level
syedahsn Sep 19, 2022
3439e5a
Update s3.rst to new DAG
syedahsn Sep 20, 2022
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
32 changes: 16 additions & 16 deletions docs/apache-airflow-providers-amazon/operators/s3.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -38,7 +38,7 @@ Create an Amazon S3 bucket
To create an Amazon S3 bucket you can use
:class:`~airflow.providers.amazon.aws.operators.s3.S3CreateBucketOperator`.

.. exampleinclude:: /../../airflow/providers/amazon/aws/example_dags/example_s3.py
.. exampleinclude:: /../../tests/system/providers/amazon/aws/example_s3.py
:language: python
:dedent: 4
:start-after: [START howto_operator_s3_create_bucket]
Expand All @@ -52,7 +52,7 @@ Delete an Amazon S3 bucket
To delete an Amazon S3 bucket you can use
:class:`~airflow.providers.amazon.aws.operators.s3.S3DeleteBucketOperator`.

.. exampleinclude:: /../../airflow/providers/amazon/aws/example_dags/example_s3.py
.. exampleinclude:: /../../tests/system/providers/amazon/aws/example_s3.py
:language: python
:dedent: 4
:start-after: [START howto_operator_s3_delete_bucket]
Expand All @@ -66,7 +66,7 @@ Set the tags for an Amazon S3 bucket
To set the tags for an Amazon S3 bucket you can use
:class:`~airflow.providers.amazon.aws.operators.s3.S3PutBucketTaggingOperator`.

.. exampleinclude:: /../../airflow/providers/amazon/aws/example_dags/example_s3.py
.. exampleinclude:: /../../tests/system/providers/amazon/aws/example_s3.py
:language: python
:dedent: 4
:start-after: [START howto_operator_s3_put_bucket_tagging]
Expand All @@ -80,7 +80,7 @@ Get the tag of an Amazon S3 bucket
To get the tag set associated with an Amazon S3 bucket you can use
:class:`~airflow.providers.amazon.aws.operators.s3.S3GetBucketTaggingOperator`.

.. exampleinclude:: /../../airflow/providers/amazon/aws/example_dags/example_s3.py
.. exampleinclude:: /../../tests/system/providers/amazon/aws/example_s3.py
:language: python
:dedent: 4
:start-after: [START howto_operator_s3_get_bucket_tagging]
Expand All @@ -94,7 +94,7 @@ Delete the tags of an Amazon S3 bucket
To delete the tags of an Amazon S3 bucket you can use
:class:`~airflow.providers.amazon.aws.operators.s3.S3DeleteBucketTaggingOperator`.

.. exampleinclude:: /../../airflow/providers/amazon/aws/example_dags/example_s3.py
.. exampleinclude:: /../../tests/system/providers/amazon/aws/example_s3.py
:language: python
:dedent: 4
:start-after: [START howto_operator_s3_delete_bucket_tagging]
Expand All @@ -108,7 +108,7 @@ Create an Amazon S3 object
To create a new (or replace) Amazon S3 object you can use
:class:`~airflow.providers.amazon.aws.operators.s3.S3CreateObjectOperator`.

.. exampleinclude:: /../../airflow/providers/amazon/aws/example_dags/example_s3.py
.. exampleinclude:: /../../tests/system/providers/amazon/aws/example_s3.py
:language: python
:dedent: 4
:start-after: [START howto_operator_s3_create_object]
Expand All @@ -123,7 +123,7 @@ To copy an Amazon S3 object from one bucket to another you can use
:class:`~airflow.providers.amazon.aws.operators.s3.S3CopyObjectOperator`.
The Amazon S3 connection used here needs to have access to both source and destination bucket/key.

.. exampleinclude:: /../../airflow/providers/amazon/aws/example_dags/example_s3.py
.. exampleinclude:: /../../tests/system/providers/amazon/aws/example_s3.py
:language: python
:dedent: 4
:start-after: [START howto_operator_s3_copy_object]
Expand All @@ -137,7 +137,7 @@ Delete Amazon S3 objects
To delete one or multiple Amazon S3 objects you can use
:class:`~airflow.providers.amazon.aws.operators.s3.S3DeleteObjectsOperator`.

.. exampleinclude:: /../../airflow/providers/amazon/aws/example_dags/example_s3.py
.. exampleinclude:: /../../tests/system/providers/amazon/aws/example_s3.py
:language: python
:dedent: 4
:start-after: [START howto_operator_s3_delete_objects]
Expand All @@ -153,7 +153,7 @@ To transform the data from one Amazon S3 object and save it to another object yo
You can also apply an optional [Amazon S3 Select expression](https://docs.aws.amazon.com/AmazonS3/latest/userguide/s3-glacier-select-sql-reference-select.html)
to select the data you want to retrieve from ``source_s3_key`` using ``select_expression``.

.. exampleinclude:: /../../airflow/providers/amazon/aws/example_dags/example_s3.py
.. exampleinclude:: /../../tests/system/providers/amazon/aws/example_s3.py
:language: python
:dedent: 4
:start-after: [START howto_operator_s3_file_transform]
Expand All @@ -169,7 +169,7 @@ To list all Amazon S3 prefixes within an Amazon S3 bucket you can use
See `here <https://docs.aws.amazon.com/AmazonS3/latest/userguide/using-prefixes.html>`__
for more information about Amazon S3 prefixes.

.. exampleinclude:: /../../airflow/providers/amazon/aws/example_dags/example_s3.py
.. exampleinclude:: /../../tests/system/providers/amazon/aws/example_s3.py
:language: python
:dedent: 4
:start-after: [START howto_operator_s3_list_prefixes]
Expand All @@ -184,7 +184,7 @@ To list all Amazon S3 objects within an Amazon S3 bucket you can use
:class:`~airflow.providers.amazon.aws.operators.s3.S3ListOperator`.
You can specify a ``prefix`` to filter the objects whose name begins with such prefix.

.. exampleinclude:: /../../airflow/providers/amazon/aws/example_dags/example_s3.py
.. exampleinclude:: /../../tests/system/providers/amazon/aws/example_s3.py
:language: python
:dedent: 4
:start-after: [START howto_operator_s3_list]
Expand All @@ -208,15 +208,15 @@ Please keep in mind, especially when used to check a large volume of keys, that

To check one file:

.. exampleinclude:: /../../airflow/providers/amazon/aws/example_dags/example_s3.py
.. exampleinclude:: /../../tests/system/providers/amazon/aws/example_s3.py
:language: python
:dedent: 4
:start-after: [START howto_sensor_s3_key_single_key]
:end-before: [END howto_sensor_s3_key_single_key]

To check multiple files:

.. exampleinclude:: /../../airflow/providers/amazon/aws/example_dags/example_s3.py
.. exampleinclude:: /../../tests/system/providers/amazon/aws/example_s3.py
:language: python
:dedent: 4
:start-after: [START howto_sensor_s3_key_multiple_keys]
Expand All @@ -236,13 +236,13 @@ multiple files can match one key. The list of matched S3 object attributes conta

[{"Size": int}]

.. exampleinclude:: /../../airflow/providers/amazon/aws/example_dags/example_s3.py
.. exampleinclude:: /../../tests/system/providers/amazon/aws/example_s3.py
:language: python
:dedent: 4
:start-after: [START howto_sensor_s3_key_function_definition]
:end-before: [END howto_sensor_s3_key_function_definition]

.. exampleinclude:: /../../airflow/providers/amazon/aws/example_dags/example_s3.py
.. exampleinclude:: /../../tests/system/providers/amazon/aws/example_s3.py
:language: python
:dedent: 4
:start-after: [START howto_sensor_s3_key_function]
Expand All @@ -259,7 +259,7 @@ the inactivity period has passed with no increase in the number of objects you c
Note, this sensor will not behave correctly in reschedule mode,
as the state of the listed objects in the Amazon S3 bucket will be lost between rescheduled invocations.

.. exampleinclude:: /../../airflow/providers/amazon/aws/example_dags/example_s3.py
.. exampleinclude:: /../../tests/system/providers/amazon/aws/example_s3.py
:language: python
:dedent: 4
:start-after: [START howto_sensor_s3_keys_unchanged]
Expand Down
Loading