-
Notifications
You must be signed in to change notification settings - Fork 16.4k
Description
Description
Follow-up to
Restructure Amazon provider operators files #20139 as discussed with @eladkal
Amazon Operators, Hooks, and Sensors are inconsistently named regarding acronym capitalization and whether to prefix with AWS or not.
An incomplete list for example: we have SnsPublishOperator but we capitalize the acronyms in SQSPublishOperator and ECSOperator. We prefix AwsGlueCrawlerOperator and AwsBatchOperator in lowercase, AWSAthenaOperator is prefixed in uppercase, and no prefix at all on GlacierCreateJobOperatoror CloudFormationCreateStackOperator
Correct naming conventions are listed in AIP 21 cases 3-4 and case 6.
The sensors/operators class names should not have Aws/AWS as part of the class name and only the first letter of the acronyms should be capitalized.
So:
SnsPublishOperator -> No change
SQSPublishOperator -> SqsPublishOperator
AwsBatchOperator -> BatchOperator
AWSAthenaOperator -> AthenaOperator
etc...
Use case/motivation
For each task:
-Rename the operator
-Deprecate the old name
--Add to tests/deprecated_classes.py
--Add to the KNOWN_DEPRECATED_MESSAGES section of dev/provider_packages/prepare_provider_packages.py
-Correct any references to the old name in docstrings and docs files
-Correct all references in tests
-Correct references in example dags
Changes Required:
| Current | New Name |
|---|---|
| AWSAthenaOperator | AthenaOperator |
| AwsBatchOperator | BatchOperator |
| AWSDataSyncOperator | DataSyncOperator |
| ECSProtocol | EcsProtocol |
| ECSTaskLogFetcher | EcsTaskLogFetcher |
| ECSOperator | EcsOperator |
| EKS***Operator (all of them) | Eks***Operator |
| SQSPublishOperator | SqsPublishOperator |
| EKS***Sensor (all of them) | Eks***Sensor |
| AwsRedshiftClusterSensor | RedshiftClusterSensor |
| SQSSensor | SqsSensor |
| AWSAthenaHook | AthenaHook |
| AWSCloudFormationHook | CloudFormationHook |
| AWSDataSyncHook | DataSyncHook |
| AwsDynamoDBHook | DynamoDBHook |
| EKSHook | EksHook |
| AwsFirehoseHook (in kinesis.py) | FirehoseHook |
| AwsLambdaHook | LambdaHook |
| SESHook | SesHook |
| AwsSnsHook | SnsHook |
| SQSHook | SqsHook |
Action Items:
- operators/athena.py - @ferruzzi Standardize AWS Athena naming #20305
- operators/batch.py - @ferruzzi Standardize AWS Batch naming #20369
- operators/datasync.py - @ferruzzi Standardize AWS DataSync naming #20328
- operators/ecs.py - @ferruzzi Standardize AWS ECS naming #20332
- operators/eks.py - @ferruzzi Standardize AWS EKS naming #20354
- operators/glue_crawler.py - @ferruzzi Standardize AWS Glue naming #20372
- operators/sqs.py - @eladkal Standardize AWS SQS classes names #20732
- sensors/eks.py - @ferruzzi Standardize AWS EKS naming #20354
- sensors/redshift.py - @ferruzzi Standardize AWS Redshift naming #20374
- sensors/sqs.py - @eladkal Standardize AWS SQS classes names #20732
- hooks/athena.py - @ferruzzi Standardize AWS Athena naming #20305
- hooks/cloud_formation.py @ferruzzi Standardize AWS CloudFormation naming #20357
- hooks/datasync.py - @ferruzzi Standardize AWS DataSync naming #20328
- hooks/dynamodb.py - @ferruzzi Standardize DynamoDB naming #20360
- hooks/emr.py - @bhavaniravi 20139 - move emr_container hook #20375
- hooks/eks.py - @ferruzzi Standardize AWS EKS naming #20354
- hooks/kinesis.py - @ferruzzi Standardize AWS Kinesis/Firehose naming #20362
- hooks/lambda.py - @ferruzzi Standardize AWS Lambda naming #20365
- hooks/ses.py - @ferruzzi Standardize Amazon SES naming #20367
- hooks/sns.py - @ferruzzi Standardize Amazon SNS naming #20368
- hooks/sqs.py - @eladkal Standardize AWS SQS classes names #20732
To be verified (recently changed in the linked campaign):
DMS - good
EC2 - good
EMR - good
Glue - Added above
S3 - good
SageMaker - good
No changes required:
Operators:
CloudFormationCreateStackOperator
CloudFormationDeleteStackOperator
GlacierCreateJobOperator
SnsPublishOperator
StepFunctionGetExecutionOutputOperator
StepFunctionStartExecutionOperator
AthenaSensor
CloudFormationCreateStackSensor
CloudFormationDeleteStackSensor
DmsTaskBaseSensor
DmsTaskCompletedSensor
Sensors:
GlacierJobOperationSensor
StepFunctionExecutionSensor
Hooks:
ElastiCacheReplicationGroupHook
GlacierHook
RedshiftHook
S3Hook
SageMakerHook
SecretsManagerHook
StepFunctionHook
Related issues
Are you willing to submit a PR?
- Yes I am willing to submit a PR!
Code of Conduct
- I agree to follow this project's Code of Conduct