Skip to content

Conversation

@ashb
Copy link
Member

@ashb ashb commented Dec 7, 2020

Using implicit string concatenation in lists, even across multiple lines is almost always a sign of a bug, and Pylint has a check for this we can enable.

This should hopefully catch a few cases like #12880 from happening again.

There may be a few places in the code that his now starts warning about that we will have to fix as a result


^ Add meaningful description above

Read the Pull Request Guidelines for more information.
In case of fundamental code change, Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in UPDATING.md.

Using implicit string concatenation in lists, even across multiple lines
is almost always a sign of a bug, and Pylint has a check for this we can
enable.
@ashb ashb added the full tests needed We need to run full set of tests for this PR to merge label Dec 7, 2020
@ashb ashb requested review from kaxil and potiuk December 7, 2020 13:18
@github-actions
Copy link

github-actions bot commented Dec 7, 2020

The PR is likely ready to be merged. No tests are needed as no important environment files, nor python files were modified by it. However, committers might decide that full test matrix is needed and add the 'full tests needed' label. Then you should rebase it to the latest master or amend the last commit of the PR, and push it with --force-with-lease.

@github-actions github-actions bot added the okay to merge It's ok to merge this PR as it does not require more tests label Dec 7, 2020
@potiuk
Copy link
Member

potiuk commented Dec 7, 2020

OH YEA!

@potiuk
Copy link
Member

potiuk commented Dec 7, 2020

I will run i locally to check /fix. Maybe faster than in GA

@ashb
Copy link
Member Author

ashb commented Dec 7, 2020

I didn't test what else this might catch (i.e. what we need to fix) but did test that it would catch the specific case that bit us.

@potiuk
Copy link
Member

potiuk commented Dec 7, 2020

Running ./breeze static-check pylint -- --all-files

@ashb
Copy link
Member Author

ashb commented Dec 7, 2020

Oh that's a good idea. This was what I checked:

airflow ❯ git di
diff --git a/setup.py b/setup.py
index 1ff20bd21..adb1995dc 100644
--- a/setup.py
+++ b/setup.py
@@ -912,6 +912,10 @@ def do_setup():
             'list_extras': ListExtras,
         },
         test_suite='setup.airflow_test_suite',
+        setup_requires=[
+            'a'
+            'b'
+        ],
         **setup_kwargs,
     )
 

~/code/airflow/airflow pyling-check-missing-commas-in-sequences*
airflow ❯ pylint setup.py
************* Module setup
setup.py:916:0: W1403: Implicit string concatenation found in list (implicit-str-concat-in-sequence)

------------------------------------------------------------------
Your code has been rated at 9.94/10 (previous run: 9.94/10, +0.00)

@potiuk
Copy link
Member

potiuk commented Dec 7, 2020

Those are the errors: fixing them in my fixup:

************* Module tests.providers.amazon.aws.sensors.test_step_function_executiontests/providers/amazon/aws/sensors/test_step_function_execution.py:30:0: W1404: Implicit string concatenation found in assignment (implicit-str-concat)
************* Module tests.providers.google.cloud.utils.test_credentials_provider
tests/providers/google/cloud/utils/test_credentials_provider.py:146:0: W1404: Implicit string concatenation found in list (implicit-str-concat)
tests/providers/google/cloud/utils/test_credentials_provider.py:252:0: W1404: Implicit string concatenation found in list (implicit-str-concat)
tests/providers/google/cloud/utils/test_credentials_provider.py:275:0: W1404: Implicit string concatenation found in list (implicit-str-concat)
************* Module tests.models.test_xcom_arg
tests/models/test_xcom_arg.py:66:0: W1404: Implicit string concatenation found in assignment (implicit-str-concat)
************* Module tests.api_connexion.endpoints.test_dag_run_endpoint
tests/api_connexion/endpoints/test_dag_run_endpoint.py:396:0: W1404: Implicit string concatenation found in tuple (implicit-str-concat)
************* Module tests.plugins.test_plugins_manager
tests/plugins/test_plugins_manager.py:136:0: W1404: Implicit string concatenation found in tuple (implicit-str-concat)
tests/plugins/test_plugins_manager.py:142:0: W1404: Implicit string concatenation found in tuple (implicit-str-concat)
************* Module test_odbc
tests/providers/odbc/hooks/test_odbc.py:50:0: W1404: Implicit string concatenation found in assignment (implicit-str-concat)
tests/providers/odbc/hooks/test_odbc.py:64:0: W1404: Implicit string concatenation found in assignment (implicit-str-concat)
tests/providers/odbc/hooks/test_odbc.py:84:0: W1404: Implicit string concatenation found in assignment (implicit-str-concat)
************* Module tests.providers.amazon.aws.transfers.test_s3_to_sftp
tests/providers/amazon/aws/transfers/test_s3_to_sftp.py:77:0: W1404: Implicit string concatenation found in assignment (implicit-str-concat)
************* Module tests.providers.yandex.operators.test_yandexcloud_dataproc
tests/providers/yandex/operators/test_yandexcloud_dataproc.py:59:0: W1404: Implicit string concatenation found in list (implicit-str-concat)
tests/providers/yandex/operators/test_yandexcloud_dataproc.py:113:0: W1404: Implicit string concatenation found in list (implicit-str-concat)
************* Module tests.providers.microsoft.azure.log.test_wasb_task_handler
tests/providers/microsoft/azure/log/test_wasb_task_handler.py:125:0: W1404: Implicit string concatenation found in tuple (implicit-str-concat)
************* Module tests.utils.test_docs
tests/utils/test_docs.py:32:0: W1404: Implicit string concatenation found in tuple (implicit-str-concat)
tests/utils/test_docs.py:38:0: W1404: Implicit string concatenation found in tuple (implicit-str-concat)
************* Module tests.providers.yandex.hooks.test_yandexcloud_dataproc
tests/providers/yandex/hooks/test_yandexcloud_dataproc.py:57:0: W1404: Implicit string concatenation found in list (implicit-str-concat)
************* Module tests.cli.commands.test_webserver_command
tests/cli/commands/test_webserver_command.py:389:0: W1404: Implicit string concatenation found in assignment (implicit-str-concat)
************* Module airflow.providers.google.cloud.example_dags.example_datacatalog
airflow/providers/google/cloud/example_dags/example_datacatalog.py:294:0: W1404: Implicit string concatenation found in assignment (implicit-str-concat)
************* Module tests.providers.google.cloud.operators.test_compute
tests/providers/google/cloud/operators/test_compute.py:341:0: W1404: Implicit string concatenation found in assignment (implicit-str-concat)
tests/providers/google/cloud/operators/test_compute.py:795:0: W1404: Implicit string concatenation found in assignment (implicit-str-concat)
tests/providers/google/cloud/operators/test_compute.py:800:0: W1404: Implicit string concatenation found in assignment (implicit-str-concat)
tests/providers/google/cloud/operators/test_compute.py:805:0: W1404: Implicit string concatenation found in assignment (implicit-str-concat)
tests/providers/google/cloud/operators/test_compute.py:810:0: W1404: Implicit string concatenation found in assignment (implicit-str-concat)
************* Module tests.providers.amazon.aws.operators.test_step_function_start_execution
tests/providers/amazon/aws/operators/test_step_function_start_execution.py:63:0: W1404: Implicit string concatenation found in assignment (implicit-str-concat)
************* Module tests.api_connexion.endpoints.test_task_instance_endpoint
tests/api_connexion/endpoints/test_task_instance_endpoint.py:320:0: W1404: Implicit string concatenation found in tuple (implicit-str-concat)
tests/api_connexion/endpoints/test_task_instance_endpoint.py:345:0: W1404: Implicit string concatenation found in tuple (implicit-str-concat)
tests/api_connexion/endpoints/test_task_instance_endpoint.py:359:0: W1404: Implicit string concatenation found in tuple (implicit-str-concat)
tests/api_connexion/endpoints/test_task_instance_endpoint.py:384:0: W1404: Implicit string concatenation found in tuple (implicit-str-concat)
************* Module airflow.cli.cli_parser
airflow/cli/cli_parser.py:69:0: W1404: Implicit string concatenation found in assignment (implicit-str-concat)
************* Module tests.jobs.test_scheduler_job
tests/jobs/test_scheduler_job.py:3079:0: W1404: Implicit string concatenation found in assignment (implicit-str-concat)
tests/jobs/test_scheduler_job.py:3108:0: W1404: Implicit string concatenation found in assignment (implicit-str-concat)
tests/jobs/test_scheduler_job.py:3133:0: W1404: Implicit string concatenation found in assignment (implicit-str-concat)
tests/jobs/test_scheduler_job.py:3163:0: W1404: Implicit string concatenation found in assignment (implicit-str-concat)
************* Module tests.providers.sftp.operators.test_sftp
tests/providers/sftp/operators/test_sftp.py:212:0: W1404: Implicit string concatenation found in assignment (implicit-str-concat)
tests/providers/sftp/operators/test_sftp.py:250:0: W1404: Implicit string concatenation found in assignment (implicit-str-concat)
tests/providers/sftp/operators/test_sftp.py:288:0: W1404: Implicit string concatenation found in assignment (implicit-str-concat)
tests/providers/sftp/operators/test_sftp.py:324:0: W1404: Implicit string concatenation found in assignment (implicit-str-concat)
************* Module tests.cli.commands.test_connection_command
tests/cli/commands/test_connection_command.py:351:0: W1404: Implicit string concatenation found in assignment (implicit-str-concat)
tests/cli/commands/test_connection_command.py:390:0: W1404: Implicit string concatenation found in list (implicit-str-concat)
tests/cli/commands/test_connection_command.py:392:0: W1404: Implicit string concatenation found in list (implicit-str-concat)
tests/cli/commands/test_connection_command.py:419:0: W1404: Implicit string concatenation found in list (implicit-str-concat)
tests/cli/commands/test_connection_command.py:421:0: W1404: Implicit string concatenation found in list (implicit-str-concat)
************* Module airflow.jobs.scheduler_job
airflow/jobs/scheduler_job.py:1226:0: W1404: Implicit string concatenation found in assignment (implicit-str-concat)
************* Module airflow.api_connexion.exceptions
airflow/api_connexion/exceptions.py:26:0: W1404: Implicit string concatenation found in assignment (implicit-str-concat)
************* Module tests.providers.elasticsearch.log.test_es_task_handler
tests/providers/elasticsearch/log/test_es_task_handler.py:320:0: W1404: Implicit string concatenation found in assignment (implicit-str-concat)
************* Module tests.serialization.test_dag_serialization
tests/serialization/test_dag_serialization.py:660:0: W1404: Implicit string concatenation found in assignment (implicit-str-concat)
tests/serialization/test_dag_serialization.py:763:0: W1404: Implicit string concatenation found in tuple (implicit-str-concat)
tests/serialization/test_dag_serialization.py:776:0: W1404: Implicit string concatenation found in tuple (implicit-str-concat)
************* Module airflow.providers.snowflake.hooks.snowflake
airflow/providers/snowflake/hooks/snowflake.py:113:0: W1404: Implicit string concatenation found in assignment (implicit-str-concat)
************* Module tests.providers.apache.spark.hooks.test_spark_submit
tests/providers/apache/spark/hooks/test_spark_submit.py:633:0: W1404: Implicit string concatenation found in list (implicit-str-concat)
tests/providers/apache/spark/hooks/test_spark_submit.py:694:0: W1404: Implicit string concatenation found in list (implicit-str-concat)
************* Module tests.providers.apache.hive.hooks.test_hive
tests/providers/apache/hive/hooks/test_hive.py:121:0: W1404: Implicit string concatenation found in assignment (implicit-str-concat)
************* Module airflow.providers.docker.example_dags.example_docker_copy_data
airflow/providers/docker/example_dags/example_docker_copy_data.py:90:0: W1404: Implicit string concatenation found in list (implicit-str-concat)
************* Module tests.providers.amazon.aws.operators.test_step_function_get_execution_output
tests/providers/amazon/aws/operators/test_step_function_get_execution_output.py:30:0: W1404: Implicit string concatenation found in assignment (implicit-str-concat)
************* Module airflow.www.views
airflow/www/views.py:2647:0: W1404: Implicit string concatenation found in assignment (implicit-str-concat)
************* Module kubernetes_tests.test_kubernetes_pod_operator_backcompat
kubernetes_tests/test_kubernetes_pod_operator_backcompat.py:310:0: W1404: Implicit string concatenation found in list (implicit-str-concat)
************* Module tests.www.test_views
tests/www/test_views.py:585:0: W1404: Implicit string concatenation found in assignment (implicit-str-concat)
tests/www/test_views.py:1150:0: W1404: Implicit string concatenation found in list (implicit-str-concat)
tests/www/test_views.py:1287:0: W1404: Implicit string concatenation found in assignment (implicit-str-concat)
tests/www/test_views.py:1314:0: W1404: Implicit string concatenation found in assignment (implicit-str-concat)
tests/www/test_views.py:1383:0: W1404: Implicit string concatenation found in assignment (implicit-str-concat)
tests/www/test_views.py:1402:0: W1404: Implicit string concatenation found in assignment (implicit-str-concat)
tests/www/test_views.py:1421:0: W1404: Implicit string concatenation found in assignment (implicit-str-concat)
tests/www/test_views.py:3285:0: W1404: Implicit string concatenation found in tuple (implicit-str-concat)
************* Module tests.providers.google.cloud.hooks.test_gcs
tests/providers/google/cloud/hooks/test_gcs.py:839:0: W1404: Implicit string concatenation found in assignment (implicit-str-concat)
************* Module airflow.www.widgets
airflow/www/widgets.py:34:0: W1404: Implicit string concatenation found in assignment (implicit-str-concat)
************* Module tests.providers.google.cloud.log.test_gcs_task_handler
tests/providers/google/cloud/log/test_gcs_task_handler.py:164:0: W1404: Implicit string concatenation found in list (implicit-str-concat)
tests/providers/google/cloud/log/test_gcs_task_handler.py:166:0: W1404: Implicit string concatenation found in list (implicit-str-concat)
************* Module tests.models.test_renderedtifields
tests/models/test_renderedtifields.py:88:0: W1404: Implicit string concatenation found in tuple (implicit-str-concat)
tests/models/test_renderedtifields.py:101:0: W1404: Implicit string concatenation found in tuple (implicit-str-concat)
************* Module test_project_structure
tests/always/test_project_structure.py:176:0: W1404: Implicit string concatenation found in set (implicit-str-concat)
tests/always/test_project_structure.py:179:0: W1404: Implicit string concatenation found in set (implicit-str-concat)
************* Module tests.providers.google.cloud.operators.test_cloud_sql
tests/providers/google/cloud/operators/test_cloud_sql.py:723:0: W1404: Implicit string concatenation found in tuple (implicit-str-concat)
tests/providers/google/cloud/operators/test_cloud_sql.py:774:0: W1404: Implicit string concatenation found in assignment (implicit-str-concat)
************* Module tests.deprecated_classes
tests/deprecated_classes.py:466:0: W1404: Implicit string concatenation found in tuple (implicit-str-concat)
tests/deprecated_classes.py:627:0: W1404: Implicit string concatenation found in tuple (implicit-str-concat)
tests/deprecated_classes.py:632:0: W1404: Implicit string concatenation found in tuple (implicit-str-concat)
tests/deprecated_classes.py:634:0: W1404: Implicit string concatenation found in tuple (implicit-str-concat)
tests/deprecated_classes.py:638:0: W1404: Implicit string concatenation found in tuple (implicit-str-concat)
tests/deprecated_classes.py:643:0: W1404: Implicit string concatenation found in tuple (implicit-str-concat)
tests/deprecated_classes.py:672:0: W1404: Implicit string concatenation found in tuple (implicit-str-concat)
tests/deprecated_classes.py:677:0: W1404: Implicit string concatenation found in tuple (implicit-str-concat)
tests/deprecated_classes.py:682:0: W1404: Implicit string concatenation found in tuple (implicit-str-concat)
tests/deprecated_classes.py:687:0: W1404: Implicit string concatenation found in tuple (implicit-str-concat)
tests/deprecated_classes.py:692:0: W1404: Implicit string concatenation found in tuple (implicit-str-concat)
tests/deprecated_classes.py:697:0: W1404: Implicit string concatenation found in tuple (implicit-str-concat)
tests/deprecated_classes.py:702:0: W1404: Implicit string concatenation found in tuple (implicit-str-concat)
tests/deprecated_classes.py:707:0: W1404: Implicit string concatenation found in tuple (implicit-str-concat)
tests/deprecated_classes.py:720:0: W1404: Implicit string concatenation found in tuple (implicit-str-concat)
tests/deprecated_classes.py:722:0: W1404: Implicit string concatenation found in tuple (implicit-str-concat)
tests/deprecated_classes.py:726:0: W1404: Implicit string concatenation found in tuple (implicit-str-concat)
tests/deprecated_classes.py:728:0: W1404: Implicit string concatenation found in tuple (implicit-str-concat)
tests/deprecated_classes.py:732:0: W1404: Implicit string concatenation found in tuple (implicit-str-concat)
tests/deprecated_classes.py:734:0: W1404: Implicit string concatenation found in tuple (implicit-str-concat)
tests/deprecated_classes.py:874:0: W1404: Implicit string concatenation found in tuple (implicit-str-concat)
tests/deprecated_classes.py:1023:0: W1404: Implicit string concatenation found in tuple (implicit-str-concat)
tests/deprecated_classes.py:1362:0: W1404: Implicit string concatenation found in tuple (implicit-str-concat)
tests/deprecated_classes.py:1566:0: W1404: Implicit string concatenation found in tuple (implicit-str-concat)
tests/deprecated_classes.py:1568:0: W1404: Implicit string concatenation found in tuple (implicit-str-concat)
tests/deprecated_classes.py:1692:0: W1404: Implicit string concatenation found in tuple (implicit-str-concat)
tests/deprecated_classes.py:1713:0: W1404: Implicit string concatenation found in tuple (implicit-str-concat)
************* Module kubernetes_tests.test_kubernetes_pod_operator
kubernetes_tests/test_kubernetes_pod_operator.py:376:0: W1404: Implicit string concatenation found in list (implicit-str-concat)
************* Module tests.providers.google.cloud.operators.test_functions
tests/providers/google/cloud/operators/test_functions.py:321:0: W1404: Implicit string concatenation found in tuple (implicit-str-concat)
tests/providers/google/cloud/operators/test_functions.py:330:0: W1404: Implicit string concatenation found in tuple (implicit-str-concat)
tests/providers/google/cloud/operators/test_functions.py:335:0: W1404: Implicit string concatenation found in tuple (implicit-str-concat)
tests/providers/google/cloud/operators/test_functions.py:340:0: W1404: Implicit string concatenation found in tuple (implicit-str-concat)
tests/providers/google/cloud/operators/test_functions.py:349:0: W1404: Implicit string concatenation found in tuple (implicit-str-concat)
************* Module tests.providers.amazon.aws.transfers.test_sftp_to_s3
tests/providers/amazon/aws/transfers/test_sftp_to_s3.py:76:0: W1404: Implicit string concatenation found in assignment (implicit-str-concat)
************* Module tests.models.test_taskinstance
tests/models/test_taskinstance.py:1242:0: W1404: Implicit string concatenation found in assignment (implicit-str-concat)
************* Module tests.providers.google.cloud.hooks.test_bigquery
tests/providers/google/cloud/hooks/test_bigquery.py:89:0: W1404: Implicit string concatenation found in assignment (implicit-str-concat)
tests/providers/google/cloud/hooks/test_bigquery.py:298:0: W1404: Implicit string concatenation found in tuple (implicit-str-concat)
tests/providers/google/cloud/hooks/test_bigquery.py:305:0: W1404: Implicit string concatenation found in tuple (implicit-str-concat)
tests/providers/google/cloud/hooks/test_bigquery.py:312:0: W1404: Implicit string concatenation found in tuple (implicit-str-concat)
tests/providers/google/cloud/hooks/test_bigquery.py:797:0: W1404: Implicit string concatenation found in tuple (implicit-str-concat)
************* Module tests.providers.presto.hooks.test_presto
tests/providers/presto/hooks/test_presto.py:224:0: W1404: Implicit string concatenation found in assignment (implicit-str-concat)
************* Module tests.providers.google.cloud.hooks.test_cloud_sql
tests/providers/google/cloud/hooks/test_cloud_sql.py:1071:0: W1404: Implicit string concatenation found in assignment (implicit-str-concat)
tests/providers/google/cloud/hooks/test_cloud_sql.py:1096:0: W1404: Implicit string concatenation found in assignment (implicit-str-concat)
tests/providers/google/cloud/hooks/test_cloud_sql.py:1105:0: W1404: Implicit string concatenation found in assignment (implicit-str-concat)
tests/providers/google/cloud/hooks/test_cloud_sql.py:1118:0: W1404: Implicit string concatenation found in assignment (implicit-str-concat)
tests/providers/google/cloud/hooks/test_cloud_sql.py:1134:0: W1404: Implicit string concatenation found in assignment (implicit-str-concat)
tests/providers/google/cloud/hooks/test_cloud_sql.py:1153:0: W1404: Implicit string concatenation found in assignment (implicit-str-concat)
tests/providers/google/cloud/hooks/test_cloud_sql.py:1168:0: W1404: Implicit string concatenation found in assignment (implicit-str-concat)
tests/providers/google/cloud/hooks/test_cloud_sql.py:1177:0: W1404: Implicit string concatenation found in assignment (implicit-str-concat)
tests/providers/google/cloud/hooks/test_cloud_sql.py:1190:0: W1404: Implicit string concatenation found in assignment (implicit-str-concat)
tests/providers/google/cloud/hooks/test_cloud_sql.py:1207:0: W1404: Implicit string concatenation found in assignment (implicit-str-concat)
************* Module setup
setup.py:157:0: W1404: Implicit string concatenation found in assignment (implicit-str-concat)

@potiuk
Copy link
Member

potiuk commented Dec 7, 2020

BTW: https://www.python.org/dev/peps/pep-3126 REJECTED :(

@ashb
Copy link
Member Author

ashb commented Dec 7, 2020

Thanks Jarek, I'm just grabbing a bite to eat

@ashb
Copy link
Member Author

ashb commented Dec 7, 2020

Oh hmmm.

I was kind of hoping it wouldn't catch cases like this tests/providers/google/cloud/hooks/test_cloud_sql.py:1207:0: W1404: Implicit string concatenation found in assignment (implicit-str-concat)

        uri = (
            "gcpcloudsql://user:password@127.0.0.1:3200/testdb?database_type=mysql&"
            "project_id=example-project&location=europe-west1&instance=testdb&"
            "use_proxy=True&sql_proxy_use_tcp=True"
        )

@ashb
Copy link
Member Author

ashb commented Dec 7, 2020

I'm not sure this is worth it after all.

@ashb
Copy link
Member Author

ashb commented Dec 7, 2020

What version of pylint do we use in breeze? Because locally I don't get nearly so many warnings.

This is why implicit-str-concat is not enabled, but implicit-str-concat-in-sequence is.

@ashb
Copy link
Member Author

ashb commented Dec 7, 2020

It will break black formatting to -- donesn't it do this implicit str concat in cases.

@potiuk
Copy link
Member

potiuk commented Dec 7, 2020

Fix here: astronomer#1110

@potiuk
Copy link
Member

potiuk commented Dec 7, 2020

It will break black formatting to -- donesn't it do this implicit str concat in cases.

It does not break black. Black at most moves stuff around but it will not remove '+' if it is there.

@potiuk
Copy link
Member

potiuk commented Dec 7, 2020

What version of pylint do we use in breeze? Because locally I don't get nearly so many warnings.

Not sure. I stopped looking at versions once we have the common breeze env which is same for everyone, I just run it there :)

@ashb ashb added this to the Airflow 2.1 milestone Dec 7, 2020
@ashb
Copy link
Member Author

ashb commented Dec 7, 2020

Punting this to 2.1

@ashb ashb marked this pull request as draft December 7, 2020 16:52
…mmas-in-sequences

Pyling check missing commas in sequences
@potiuk
Copy link
Member

potiuk commented Dec 7, 2020

Punting this to 2.1

Fine for me.

@ashb
Copy link
Member Author

ashb commented Jan 7, 2021

I don't like this check.

@ashb ashb closed this Jan 7, 2021
@ashb ashb deleted the pyling-check-missing-commas-in-sequences branch January 7, 2021 11:24
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

full tests needed We need to run full set of tests for this PR to merge okay to merge It's ok to merge this PR as it does not require more tests

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants