Restructure Amazon provider operators files #20139
Description
Body
Spun out of #19665 (comment)
Info:
Currently operators in Amazon provider has a structure of operator per file (link) A single service S3 has 7 operators files! This makes it very hard to discover classes.
The proposed structure is what we have for GCP (link) where file is created for a service and all operators for that services are in a single file.
Proposed changes:
Operators:
files that don't require changes:
athena.py
batch.py
cloud_formation.py
datasync.py
ecs.py
eks.py
glacier.py
glue.py
glue_crawler.py
redshift.py
sns.py
sqs.py
files that require changes:
| Current | New |
|---|---|
| dms_create_task.py | dms.py |
| dms_delete_task.py | dms.py |
| dms_describe_tasks.py | dms.py |
| dms_start_task.py | dms.py |
| dms_stop_task.py | dms.py |
| ec2_start_instance.py | ec2.py |
| ec2_stop_instance.py | ec2.py |
| emr_add_steps.py | emr.py |
| emr_containers.py | emr.py |
| emr_create_job_flow.py | emr.py |
| emr_modify_cluster.py | emr.py |
| emr_terminate_job_flow.py | emr.py |
| s3_bucket.py | s3.py |
| s3_bucket_tagging.py | s3.py |
| s3_copy_object.py | s3.py |
| s3_delete_objects.py | s3.py |
| s3_file_transform.py | s3.py |
| s3_list.py | s3.py |
| s3_list_prefixes.py | s3.py |
| sagemaker_base.py | sagemaker.py |
| sagemaker_endpoint.py | sagemaker.py |
| sagemaker_endpoint_config.py | sagemaker.py |
| sagemaker_model.py | sagemaker.py |
| sagemaker_processing.py | sagemaker.py |
| sagemaker_training.py | sagemaker.py |
| sagemaker_transform.py | sagemaker.py |
| sagemaker_tuning.py | sagemaker.py |
| step_function_get_execution_output.py | step_function.py |
| step_function_start_execution.py | step_function.py |
Sensors:
files that don't require changes:
athena.py
batch.py
cloud_formation.py
eks.py
glacier.py
glue.py
glue_crawler.py
redshift.py
sqs.py
files that require changes:
| Current | New | Note |
|---|---|---|
| dms_task.py | dms.py | |
| ec2_instance_state.py | ec2.py | |
| emr_base.py | emr.py | |
| emr_containers.py | emr.py | |
| emr_job_flow.py | emr.py | |
| emr_step.py | emr.py | |
| glue_catalog_partition.py | glue.py | New file already exist |
| s3_key.py | s3.py | |
| s3_keys_unchanged.py | s3.py | |
| s3_prefix.py | s3.py | |
| sagemaker_base.py | sagemaker.py | |
| sagemaker_endpoint.py | sagemaker.py | |
| sagemaker_training.py | sagemaker.py | |
| sagemaker_transform.py | sagemaker.py | |
| sagemaker_tuning.py | sagemaker.py | |
| step_function_execution.py | step_function.py |
Action items:
Create new files and deprecate old ones. This is a backward compatible change.
old file should raise deprecation warnings.
- operators/dms.py @bhavaniravi #20139 - Move Dms operators and Sensors #20156
- operators/ec2.py @eladkal Organize EC2 classes in Amazon provider #20157
- operators/emr.py @bhavaniravi 20139 - organize aws emr operators #20160
- operators/s3.py @KennyRich Organise S3 Classes in Amazon Provider #20167
- operators/sagemaker.py @bhavaniravi Organize Sagemaker classes in Amazon provider #20370
- operators/step_function.py @bhavaniravi 20139 - organize aws step_function #20158
- sensors/dms.py @bhavaniravi #20139 - Move Dms operators and Sensors #20156
- sensors/ec2.py @eladkal Organize EC2 classes in Amazon provider #20157
- sensors/emr.py @bhavaniravi 20139 - organize aws emr operators #20160
[ ] sensors/glue.py @eladkal On hold as we don't know if this really required. Will be revisited in the future if needed- sensors/s3.py @KennyRich Organise S3 Classes in Amazon Provider #20167
- sensors/sagemaker.py @bhavaniravi Organize Sagemaker classes in Amazon provider #20370
- sensors/step_function.py @bhavaniravi 20139 - organize aws step_function #20158
FYI @dstandish @ferruzzi @o-nikolas @dbarrundiag
Committer
- I acknowledge that I am a maintainer/committer of the Apache Airflow project.