AWSGlueJobHook updates job configuration if it exists

[romibuzi]

closes: #27592

---

Rename GlueJobHook.get_or_create_glue_job() into create_or_update_glue_job() and split code into different methods: create_glue_job_config(), has_job(), create_job() and update_job().

It is now similar to the behavior in GlueCrawlerOperator.

[boring-cyborg]

Congratulations on your first Pull Request and welcome to the Apache Airflow community! If you have any issues or are unsure about any anything please check our Contribution Guide (https://github.com/apache/airflow/blob/main/CONTRIBUTING.rst)
Here are some useful points:

• Pay attention to the quality of your code (flake8, mypy and type annotations). Our pre-commits will help you with that.
• In case of a new feature add useful documentation (in docstrings or in docs/ directory). Adding a new operator? Check this short guide Consider adding an example DAG that shows how users should use it.
• Consider using Breeze environment for testing locally, it's a heavy docker but it ships with a working Airflow and a lot of integrations.
• Be patient and persistent. It might take some time to get a review or get the final approval from Committers.
• Please follow ASF Code of Conduct for all communication including (but not limited to) comments on Pull Requests, Mailing list and Slack.
• Be sure to read the Airflow Coding style.
  Apache Airflow is a community-driven project and together we are making it better 🚀.
  In case of doubts contact the developers at:
  Mailing List: dev@airflow.apache.org
  Slack: https://s.apache.org/airflow-slack

[Taragolis]

small nitpick AwsBaseHook already have two ways how to get boto3.client:

• AwsBaseHook.get_conn()
• AwsBaseHook.conn

And both of them cached so... might be better not to create third way?

[romibuzi]

Oh good catch! I have removed it and made use of AwsBaseHook.get_conn() instead

[syedahsn]

This part here can be refactored to be a bit more concise. Rather than have two return statements returning very similar dictionaries, something like this would be cleaner:

ret_config = {
    "Name": self.job_name,
    "Description": self.desc,
    "LogUri": s3_log_path,
    "Role": execution_role["Role"]["Arn"],
    "ExecutionProperty": {"MaxConcurrentRuns": self.concurrent_run_limit},
    "Command": command,
    "MaxRetries": self.retry_limit,
    **self.create_job_kwargs,
}

if "WorkerType" in self.create_job_kwargs and "NumberOfWorkers" in self.create_job_kwargs:
    ret_config["MaxCapacity"] = self.num_of_dpus

return ret_config

Also, it's generally preferable to use {} rather than the dict() function

[romibuzi]

Done! indeed this way it is nicer

[romibuzi]

@Taragolis and @syedahsn Thanks for your reviews! I have done modifications. Feel free to take a look at them :)

[Taragolis]

looks good to me

[vincbeck]

TypeError would make more sense?

[romibuzi]

done, i have checked in other AWS hooks and it is usually ValueError raised in case a parameter is missing so i put this one

[vincbeck]

Should this name provided by the user and not hardcoded?

[romibuzi]

here it is the default command used but the user can specify another one with the "Command" kwarg: https://github.com/apache/airflow/blob/23d33fe4f424b0302a1637a6005185ea4139c04e/airflow/providers/amazon/aws/hooks/glue.py#L103

[vincbeck]

I see, you can ignore this comment then

[vincbeck]

It is a pretty strong assumption to assume that the user wants the execution role as role no? Should not it be specified by the user?

[romibuzi]

execution_role here is the one associated with the role_name argument of the hook: https://github.com/apache/airflow/blob/23d33fe4f424b0302a1637a6005185ea4139c04e/airflow/providers/amazon/aws/hooks/glue.py#L134-L136

[vincbeck]

My bad :)

[vincbeck]

As a convention we usually try to avoid having methods in hooks which just are just wrapper around boto3 APIs. To me, there is no much value having get_job function in the hook since you can use directly hook.conn.get_job

[romibuzi]

Good remark indeed it doesnt make much sense if there is no additionnal logic in these wrappers methods, I have removed them

[vincbeck]

Same comment here, this function is just a wrapper around boto3 create_job API

[romibuzi]

Thank you @vincbeck!!

[boring-cyborg]

Awesome work, congrats on your first merged pull request!