[SPARK-47323][K8S] Support custom executor log urls

[EnricoMi]

What changes were proposed in this pull request?

Make Kubernetes resource manager support existing config spark.ui.custom.executor.log.url.

Allow for

spark.ui.custom.executor.log.url="https://my.custom.url/logs?app={{APP_ID}}&executor={{EXECUTOR_ID}}"

Supports these variables:

• APP_ID: The unique application id
• EXECUTOR_ID: The executor id (a positive integer larger than zero)
• HOSTNAME: The name of the host where the executor runs
• KUBERNETES_NAMESPACE: The namespace where the executor pods run
• KUBERNETES_POD_NAME: The name of the pod that contains the executor
• FILE_NAME: The name of the log, which is always "log"

Why are the changes needed?

Running Spark on Kubernetes requires persisting the logs elsewhere. Having the Spark UI link to those logs is very useful. This is currently only supported by YARN.

Does this PR introduce any user-facing change?

Spark UI provides links to logs when run on Kubernetes.

How was this patch tested?

Unit test and manually tested on minikube K8S cluster.

Was this patch authored or co-authored using generative AI tooling?

No

[mridulm]

+CC @thejdeep

[pan3793]

the external log service for K8s is likely to use namespace and pod name to query the logs, could you please expose NAMESPACE too?

[EnricoMi]

I have added the namespace.

[pan3793]

@EnricoMi this looks much simpler than my previous attempt #38357

[pan3793]

standalone?

[EnricoMi]

I think the Please check the documentation for your cluster manager to see which patterns are supported, if any. is sufficient, there is no need to list which manager supports this conf and which don't. That list easily gets out-dated.

[EnricoMi]

@EnricoMi this looks much simpler than my previous attempt #38357

@pan3793 Thanks for the pointer! Here is also a PR for driver log support (#45728) which borrows some code from your attempt (#38357).

[EnricoMi]

CC @dongjoon-hyun

[EnricoMi]

@dongjoon-hyun What do you think?

[github-actions]

We're closing this PR because it hasn't been updated in a while. This isn't a judgement on the merit of the PR in any way. It's just a way of keeping the PR queue manageable.
If you'd like to revive this PR, please reopen it and ask a committer to remove the Stale tag!

[dongjoon-hyun]

Sorry for being late, @EnricoMi .

Apache Spark has been supported this feature. So, The configuration document is fixed in master/3.5/3.4. Could you try to follow the updated documentation?

• #47681

spark.executorEnv.SPARK_EXECUTOR_ATTRIBUTE_APP_ID='$(SPARK_APPLICATION_ID)'
spark.executorEnv.SPARK_EXECUTOR_ATTRIBUTE_EXECUTOR_ID='$(SPARK_EXECUTOR_ID)'
spark.ui.custom.executor.log.url='https://log-server/log?appId={{APP_ID}}&execId={{

[EnricoMi]

Looks like this works in master. Which versions before 4.0.0 support this?

[dongjoon-hyun]

Looks like this works in master. Which versions before 4.0.0 support this?

All Apache Spark with K8s GA have been supporting it. So, SPARK-49176 is a documentation fix.