Add log for AWS Glue Job Console URL #28925
Conversation
boring-cyborg
bot
added
area:providers
provider:amazon
|
@IAL32 I would suggest to create external link: You could keep also link within the logs sometime it also useful, however I think external link much useful |
|
I was not aware of the functionality. Neat! I added it and the test for it, let me know what you think 😄 |
| class GlueJobLogsLink(BaseAwsLink): | ||
| """Helper class for constructing AWS Glue Job Logs Link""" | ||
|
|
||
| name = "AWS Glue Job Logs" | ||
| key = "glue_job_logs" |
There was a problem hiding this comment.
As far as i understand you want to add link to AWS Glue Studio ❯ Monitoring ❯ Job run not link to Cloudwatch logs
So better named it appropriate like (it suggested renaming, final choice by you):
GlueJobLogsLink > GlueJobRunDetailsLink
"AWS Glue Job Logs" > "AWS Glue Job Run Details"
"glue_job_logs" > "glue_job_run_details"
There was a problem hiding this comment.
It might also make sense to add the CloudWatch logs link?
Maybe I will add them on a new PR
There was a problem hiding this comment.
Yep, why not? We already add some similar for BatchOperator.
I think for Glue it might required create a bit different link, e.g. Batch use link to a single LogStream, but Glue create multiple different log stream during execution so potentially we need to provide link to filter LogStream by specific Job ID
| glue_job_run_url = ( | ||
| f"{BASE_AWS_CONSOLE_LINK}/gluestudio/home?" | ||
| + f"region={glue_job.conn_region_name}#/job/{urllib.parse.quote(self.job_name, safe='')}/run/" | ||
| + glue_job_run["JobRunId"] | ||
| ) |
There was a problem hiding this comment.
Thanks for the PR!
Could you not reuse the format string from the Link class so as to not duplicate it here? The two could easily get out of sync. Something like (NOTE: untested):
| glue_job_run_url = ( | |
| f"{BASE_AWS_CONSOLE_LINK}/gluestudio/home?" | |
| + f"region={glue_job.conn_region_name}#/job/{urllib.parse.quote(self.job_name, safe='')}/run/" | |
| + glue_job_run["JobRunId"] | |
| ) | |
| glue_job_run_url = GlueJobRunDetailsLink.format_str.format( | |
| region_name=glue_job.conn_region_name, | |
| job_name=urllib.parse.quote(self.job_name, safe=''), | |
| job_run_id=glue_job_run["JobRunId"]) |
There was a problem hiding this comment.
Thanks @o-nikolas for the suggestion! We should consider having a similar signature of .persist to also get the formatted link. My approach seems a bit ugly and forced.
There was a problem hiding this comment.
It may be an interesting PR to work for 😄
There was a problem hiding this comment.
Why not? When I initially create BaseAwsLink I've just grab an idea from BaseGoogleLink
* Log AWS Glue Job Console URL * Added GlueJobLogsLink
4 participants
It is possible to view the current job run in the AWS Console.
I add a log pointing to the URL that shows the current job run for convenience.
I also slightly modify the test to include job use a job name with slash, which is a valid job name.