From 674496de9b1d9ae69057c8e019a5789a34f0c2dd Mon Sep 17 00:00:00 2001 From: panbingkun Date: Sat, 30 Dec 2023 10:02:49 +0800 Subject: [PATCH 1/7] [SPARK-46546][DOCS] Fix the formatting of tables in `running-on-kubernetes` and `running-on-yarn` pages --- docs/running-on-kubernetes.md | 573 ++++++++++++++++++---------------- docs/running-on-yarn.md | 325 ++++++++++--------- 2 files changed, 472 insertions(+), 426 deletions(-) diff --git a/docs/running-on-kubernetes.md b/docs/running-on-kubernetes.md index 4b4dc9d304fbd..4f3894f28b8ca 100644 --- a/docs/running-on-kubernetes.md +++ b/docs/running-on-kubernetes.md @@ -592,23 +592,30 @@ See the [configuration page](configuration.html) for information on Spark config #### Spark Properties - - - - - +
Property NameDefaultMeaningSince Version
spark.kubernetes.context(none)
+ + + + + + + + + + + - - + + - - + + - - + + - - + + - - + + - - + + - - + + - - + + - - + + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - - + + - - + + - + - + - + - + - + - + - + - + - + - + - + - + - + - - + + - - + + - - + + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - - + + - - + + - - + + - - + + - - + + - - + + - - + + - - + + - + - + - + - + - + - - + + - - + + - - + + - - + + - - + + - - + + - + - - + + - + - + - + - + - + - + - - + + - - + + - - + + - - + + - - + + - - + + - - + + - - + + - - + + - - + + - + - + - - + + - - + + - - + + - - + + - - + + - - + + - - + + - - + + - - + +
Property NameDefaultMeaningSince Version
spark.kubernetes.context(none) The context from the user Kubernetes configuration file used for the initial auto-configuration of the Kubernetes client library. When not specified then the users current context is used. NB: Many of the auto-configured settings can be overridden by the use of other Spark - configuration properties e.g. spark.kubernetes.namespace. + configuration properties e.g. spark.kubernetes.namespace. 3.0.0
spark.kubernetes.driver.masterhttps://kubernetes.default.svcspark.kubernetes.driver.masterhttps://kubernetes.default.svc The internal Kubernetes master (API server) address to be used for driver to request executors or 'local[*]' for driver-pod-only mode. @@ -616,68 +623,68 @@ See the [configuration page](configuration.html) for information on Spark config 3.0.0
spark.kubernetes.namespacedefaultspark.kubernetes.namespacedefault The namespace that will be used for running the driver and executor pods. 2.3.0
spark.kubernetes.container.image(none)spark.kubernetes.container.image(none) Container image to use for the Spark application. - This is usually of the form example.com/repo/spark:v1.0.0. + This is usually of the form example.com/repo/spark:v1.0.0. This configuration is required and must be provided by the user, unless explicit images are provided for each different container type. 2.3.0
spark.kubernetes.driver.container.image(value of spark.kubernetes.container.image)spark.kubernetes.driver.container.image(value of spark.kubernetes.container.image) Custom container image to use for the driver. 2.3.0
spark.kubernetes.executor.container.image(value of spark.kubernetes.container.image)spark.kubernetes.executor.container.image(value of spark.kubernetes.container.image) Custom container image to use for executors. 2.3.0
spark.kubernetes.container.image.pullPolicyIfNotPresentspark.kubernetes.container.image.pullPolicyIfNotPresent Container image pull policy used when pulling images within Kubernetes. - Valid values are Always, Never, and IfNotPresent. + Valid values are Always, Never, and IfNotPresent. 2.3.0
spark.kubernetes.container.image.pullSecretsspark.kubernetes.container.image.pullSecrets Comma separated list of Kubernetes secrets used to pull images from private image registries. 2.4.0
spark.kubernetes.allocation.batch.size5spark.kubernetes.allocation.batch.size5 Number of pods to launch at once in each round of executor pod allocation. 2.3.0
spark.kubernetes.allocation.batch.delay1sspark.kubernetes.allocation.batch.delay1s Time to wait between each round of executor pod allocation. Specifying values less than 1 second may lead to excessive CPU usage on the spark driver. @@ -685,167 +692,167 @@ See the [configuration page](configuration.html) for information on Spark config 2.3.0
spark.kubernetes.authenticate.submission.caCertFilespark.kubernetes.authenticate.submission.caCertFile (none) Path to the CA cert file for connecting to the Kubernetes API server over TLS when starting the driver. This file must be located on the submitting machine's disk. Specify this as a path as opposed to a URI (i.e. do not provide - a scheme). In client mode, use spark.kubernetes.authenticate.caCertFile instead. + a scheme). In client mode, use spark.kubernetes.authenticate.caCertFile instead. 2.3.0
spark.kubernetes.authenticate.submission.clientKeyFilespark.kubernetes.authenticate.submission.clientKeyFile (none) Path to the client key file for authenticating against the Kubernetes API server when starting the driver. This file must be located on the submitting machine's disk. Specify this as a path as opposed to a URI (i.e. do not provide - a scheme). In client mode, use spark.kubernetes.authenticate.clientKeyFile instead. + a scheme). In client mode, use spark.kubernetes.authenticate.clientKeyFile instead. 2.3.0
spark.kubernetes.authenticate.submission.clientCertFilespark.kubernetes.authenticate.submission.clientCertFile (none) Path to the client cert file for authenticating against the Kubernetes API server when starting the driver. This file must be located on the submitting machine's disk. Specify this as a path as opposed to a URI (i.e. do not - provide a scheme). In client mode, use spark.kubernetes.authenticate.clientCertFile instead. + provide a scheme). In client mode, use spark.kubernetes.authenticate.clientCertFile instead. 2.3.0
spark.kubernetes.authenticate.submission.oauthTokenspark.kubernetes.authenticate.submission.oauthToken (none) OAuth token to use when authenticating against the Kubernetes API server when starting the driver. Note that unlike the other authentication options, this is expected to be the exact string value of the token to use for - the authentication. In client mode, use spark.kubernetes.authenticate.oauthToken instead. + the authentication. In client mode, use spark.kubernetes.authenticate.oauthToken instead. 2.3.0
spark.kubernetes.authenticate.submission.oauthTokenFilespark.kubernetes.authenticate.submission.oauthTokenFile (none) Path to the OAuth token file containing the token to use when authenticating against the Kubernetes API server when starting the driver. This file must be located on the submitting machine's disk. Specify this as a path as opposed to a URI (i.e. do not - provide a scheme). In client mode, use spark.kubernetes.authenticate.oauthTokenFile instead. + provide a scheme). In client mode, use spark.kubernetes.authenticate.oauthTokenFile instead. 2.3.0
spark.kubernetes.authenticate.driver.caCertFilespark.kubernetes.authenticate.driver.caCertFile (none) Path to the CA cert file for connecting to the Kubernetes API server over TLS from the driver pod when requesting executors. This file must be located on the submitting machine's disk, and will be uploaded to the driver pod. Specify this as a path as opposed to a URI (i.e. do not provide a scheme). In client mode, use - spark.kubernetes.authenticate.caCertFile instead. + spark.kubernetes.authenticate.caCertFile instead. 2.3.0
spark.kubernetes.authenticate.driver.clientKeyFilespark.kubernetes.authenticate.driver.clientKeyFile (none) Path to the client key file for authenticating against the Kubernetes API server from the driver pod when requesting executors. This file must be located on the submitting machine's disk, and will be uploaded to the driver pod as a Kubernetes secret. Specify this as a path as opposed to a URI (i.e. do not provide a scheme). - In client mode, use spark.kubernetes.authenticate.clientKeyFile instead. + In client mode, use spark.kubernetes.authenticate.clientKeyFile instead. 2.3.0
spark.kubernetes.authenticate.driver.clientCertFilespark.kubernetes.authenticate.driver.clientCertFile (none) Path to the client cert file for authenticating against the Kubernetes API server from the driver pod when requesting executors. This file must be located on the submitting machine's disk, and will be uploaded to the driver pod as a Kubernetes secret. Specify this as a path as opposed to a URI (i.e. do not provide a scheme). - In client mode, use spark.kubernetes.authenticate.clientCertFile instead. + In client mode, use spark.kubernetes.authenticate.clientCertFile instead. 2.3.0
spark.kubernetes.authenticate.driver.oauthTokenspark.kubernetes.authenticate.driver.oauthToken (none) OAuth token to use when authenticating against the Kubernetes API server from the driver pod when requesting executors. Note that unlike the other authentication options, this must be the exact string value of the token to use for the authentication. This token value is uploaded to the driver pod as a Kubernetes secret. - In client mode, use spark.kubernetes.authenticate.oauthToken instead. + In client mode, use spark.kubernetes.authenticate.oauthToken instead. 2.3.0
spark.kubernetes.authenticate.driver.oauthTokenFilespark.kubernetes.authenticate.driver.oauthTokenFile (none) Path to the OAuth token file containing the token to use when authenticating against the Kubernetes API server from the driver pod when requesting executors. Note that unlike the other authentication options, this file must contain the exact string value of the token to use for the authentication. This token value is uploaded to the driver pod as a secret. In client mode, use - spark.kubernetes.authenticate.oauthTokenFile instead. + spark.kubernetes.authenticate.oauthTokenFile instead. 2.3.0
spark.kubernetes.authenticate.driver.mounted.caCertFilespark.kubernetes.authenticate.driver.mounted.caCertFile (none) Path to the CA cert file for connecting to the Kubernetes API server over TLS from the driver pod when requesting executors. This path must be accessible from the driver pod. Specify this as a path as opposed to a URI (i.e. do not provide a scheme). In client mode, use - spark.kubernetes.authenticate.caCertFile instead. + spark.kubernetes.authenticate.caCertFile instead. 2.3.0
spark.kubernetes.authenticate.driver.mounted.clientKeyFilespark.kubernetes.authenticate.driver.mounted.clientKeyFile (none) Path to the client key file for authenticating against the Kubernetes API server from the driver pod when requesting executors. This path must be accessible from the driver pod. Specify this as a path as opposed to a URI (i.e. do not provide a scheme). In client mode, use - spark.kubernetes.authenticate.clientKeyFile instead. + spark.kubernetes.authenticate.clientKeyFile instead. 2.3.0
spark.kubernetes.authenticate.driver.mounted.clientCertFilespark.kubernetes.authenticate.driver.mounted.clientCertFile (none) Path to the client cert file for authenticating against the Kubernetes API server from the driver pod when requesting executors. This path must be accessible from the driver pod. Specify this as a path as opposed to a URI (i.e. do not provide a scheme). In client mode, use - spark.kubernetes.authenticate.clientCertFile instead. + spark.kubernetes.authenticate.clientCertFile instead. 2.3.0
spark.kubernetes.authenticate.driver.mounted.oauthTokenFilespark.kubernetes.authenticate.driver.mounted.oauthTokenFile (none) Path to the file containing the OAuth token to use when authenticating against the Kubernetes API server from the driver pod when requesting executors. This path must be accessible from the driver pod. Note that unlike the other authentication options, this file must contain the exact string value of the token to use - for the authentication. In client mode, use spark.kubernetes.authenticate.oauthTokenFile instead. + for the authentication. In client mode, use spark.kubernetes.authenticate.oauthTokenFile instead. 2.3.0
spark.kubernetes.authenticate.driver.serviceAccountNamedefaultspark.kubernetes.authenticate.driver.serviceAccountNamedefault Service account that is used when running the driver pod. The driver pod uses this service account when requesting executor pods from the API server. Note that this cannot be specified alongside a CA cert file, client key file, - client cert file, and/or OAuth token. In client mode, use spark.kubernetes.authenticate.serviceAccountName instead. + client cert file, and/or OAuth token. In client mode, use spark.kubernetes.authenticate.serviceAccountName instead. 2.3.0
spark.kubernetes.authenticate.executor.serviceAccountName(value of spark.kubernetes.authenticate.driver.serviceAccountName)spark.kubernetes.authenticate.executor.serviceAccountName(value of spark.kubernetes.authenticate.driver.serviceAccountName) Service account that is used when running the executor pod. If this parameter is not setup, the fallback logic will use the driver's service account. @@ -853,7 +860,7 @@ See the [configuration page](configuration.html) for information on Spark config 3.1.0
spark.kubernetes.authenticate.caCertFilespark.kubernetes.authenticate.caCertFile (none) In client mode, path to the CA cert file for connecting to the Kubernetes API server over TLS when @@ -862,7 +869,7 @@ See the [configuration page](configuration.html) for information on Spark config 2.4.0
spark.kubernetes.authenticate.clientKeyFilespark.kubernetes.authenticate.clientKeyFile (none) In client mode, path to the client key file for authenticating against the Kubernetes API server @@ -871,7 +878,7 @@ See the [configuration page](configuration.html) for information on Spark config 2.4.0
spark.kubernetes.authenticate.clientCertFilespark.kubernetes.authenticate.clientCertFile (none) In client mode, path to the client cert file for authenticating against the Kubernetes API server @@ -880,7 +887,7 @@ See the [configuration page](configuration.html) for information on Spark config 2.4.0
spark.kubernetes.authenticate.oauthTokenspark.kubernetes.authenticate.oauthToken (none) In client mode, the OAuth token to use when authenticating against the Kubernetes API server when @@ -890,7 +897,7 @@ See the [configuration page](configuration.html) for information on Spark config 2.4.0
spark.kubernetes.authenticate.oauthTokenFilespark.kubernetes.authenticate.oauthTokenFile (none) In client mode, path to the file containing the OAuth token to use when authenticating against the Kubernetes API @@ -899,67 +906,67 @@ See the [configuration page](configuration.html) for information on Spark config 2.4.0
spark.kubernetes.driver.label.[LabelName]spark.kubernetes.driver.label.[LabelName] (none) - Add the label specified by LabelName to the driver pod. - For example, spark.kubernetes.driver.label.something=true. + Add the label specified by LabelName to the driver pod. + For example, spark.kubernetes.driver.label.something=true. Note that Spark also adds its own labels to the driver pod for bookkeeping purposes. 2.3.0
spark.kubernetes.driver.annotation.[AnnotationName]spark.kubernetes.driver.annotation.[AnnotationName] (none) - Add the Kubernetes annotation specified by AnnotationName to the driver pod. - For example, spark.kubernetes.driver.annotation.something=true. + Add the Kubernetes annotation specified by AnnotationName to the driver pod. + For example, spark.kubernetes.driver.annotation.something=true. 2.3.0
spark.kubernetes.driver.service.label.[LabelName]spark.kubernetes.driver.service.label.[LabelName] (none) - Add the Kubernetes label specified by LabelName to the driver service. - For example, spark.kubernetes.driver.service.label.something=true. + Add the Kubernetes label specified by LabelName to the driver service. + For example, spark.kubernetes.driver.service.label.something=true. Note that Spark also adds its own labels to the driver service for bookkeeping purposes. 3.4.0
spark.kubernetes.driver.service.annotation.[AnnotationName]spark.kubernetes.driver.service.annotation.[AnnotationName] (none) - Add the Kubernetes annotation specified by AnnotationName to the driver service. - For example, spark.kubernetes.driver.service.annotation.something=true. + Add the Kubernetes annotation specified by AnnotationName to the driver service. + For example, spark.kubernetes.driver.service.annotation.something=true. 3.0.0
spark.kubernetes.executor.label.[LabelName]spark.kubernetes.executor.label.[LabelName] (none) - Add the label specified by LabelName to the executor pods. - For example, spark.kubernetes.executor.label.something=true. + Add the label specified by LabelName to the executor pods. + For example, spark.kubernetes.executor.label.something=true. Note that Spark also adds its own labels to the executor pod for bookkeeping purposes. 2.3.0
spark.kubernetes.executor.annotation.[AnnotationName]spark.kubernetes.executor.annotation.[AnnotationName] (none) - Add the Kubernetes annotation specified by AnnotationName to the executor pods. - For example, spark.kubernetes.executor.annotation.something=true. + Add the Kubernetes annotation specified by AnnotationName to the executor pods. + For example, spark.kubernetes.executor.annotation.something=true. 2.3.0
spark.kubernetes.driver.pod.namespark.kubernetes.driver.pod.name (none) Name of the driver pod. In cluster mode, if this is not set, the driver pod name is set to "spark.app.name" @@ -971,19 +978,19 @@ See the [configuration page](configuration.html) for information on Spark config 2.3.0
spark.kubernetes.executor.podNamePrefixspark.kubernetes.executor.podNamePrefix (none) Prefix to use in front of the executor pod names. It must conform the rules defined by the Kubernetes DNS Label Names. - The prefix will be used to generate executor pod names in the form of $podNamePrefix-exec-$id, where the `id` is + The prefix will be used to generate executor pod names in the form of $podNamePrefix-exec-$id, where the `id` is a positive int value, so the length of the `podNamePrefix` needs to be less than or equal to 47(= 63 - 10 - 6). 2.3.0
spark.kubernetes.submission.waitAppCompletiontruespark.kubernetes.submission.waitAppCompletiontrue In cluster mode, whether to wait for the application to finish before exiting the launcher process. When changed to false, the launcher has a "fire-and-forget" behavior when launching the Spark job. @@ -991,33 +998,33 @@ See the [configuration page](configuration.html) for information on Spark config 2.3.0
spark.kubernetes.report.interval1sspark.kubernetes.report.interval1s Interval between reports of the current Spark job status in cluster mode. 2.3.0
spark.kubernetes.executor.apiPollingInterval30sspark.kubernetes.executor.apiPollingInterval30s Interval between polls against the Kubernetes API server to inspect the state of executors. 2.4.0
spark.kubernetes.driver.request.coresspark.kubernetes.driver.request.cores (none) Specify the cpu request for the driver pod. Values conform to the Kubernetes convention. Example values include 0.1, 500m, 1.5, 5, etc., with the definition of cpu units documented in CPU units. - This takes precedence over spark.driver.cores for specifying the driver pod cpu request if set. + This takes precedence over spark.driver.cores for specifying the driver pod cpu request if set. 3.0.0
spark.kubernetes.driver.limit.coresspark.kubernetes.driver.limit.cores (none) Specify a hard cpu limit for the driver pod. @@ -1025,18 +1032,18 @@ See the [configuration page](configuration.html) for information on Spark config 2.3.0
spark.kubernetes.executor.request.coresspark.kubernetes.executor.request.cores (none) Specify the cpu request for each executor pod. Values conform to the Kubernetes convention. Example values include 0.1, 500m, 1.5, 5, etc., with the definition of cpu units documented in CPU units. - This is distinct from spark.executor.cores: it is only used and takes precedence over spark.executor.cores for specifying the executor pod cpu request if set. Task + This is distinct from spark.executor.cores: it is only used and takes precedence over spark.executor.cores for specifying the executor pod cpu request if set. Task parallelism, e.g., number of tasks an executor can run concurrently is not affected by this. 2.4.0
spark.kubernetes.executor.limit.coresspark.kubernetes.executor.limit.cores (none) Specify a hard cpu limit for each executor pod launched for the Spark Application. @@ -1044,177 +1051,177 @@ See the [configuration page](configuration.html) for information on Spark config 2.3.0
spark.kubernetes.node.selector.[labelKey]spark.kubernetes.node.selector.[labelKey] (none) - Adds to the node selector of the driver pod and executor pods, with key labelKey and the value as the - configuration's value. For example, setting spark.kubernetes.node.selector.identifier to myIdentifier - will result in the driver pod and executors having a node selector with key identifier and value - myIdentifier. Multiple node selector keys can be added by setting multiple configurations with this prefix. + Adds to the node selector of the driver pod and executor pods, with key labelKey and the value as the + configuration's value. For example, setting spark.kubernetes.node.selector.identifier to myIdentifier + will result in the driver pod and executors having a node selector with key identifier and value + myIdentifier. Multiple node selector keys can be added by setting multiple configurations with this prefix. 2.3.0
spark.kubernetes.driver.node.selector.[labelKey]spark.kubernetes.driver.node.selector.[labelKey] (none) - Adds to the driver node selector of the driver pod, with key labelKey and the value as the - configuration's value. For example, setting spark.kubernetes.driver.node.selector.identifier to myIdentifier - will result in the driver pod having a node selector with key identifier and value - myIdentifier. Multiple driver node selector keys can be added by setting multiple configurations with this prefix. + Adds to the driver node selector of the driver pod, with key labelKey and the value as the + configuration's value. For example, setting spark.kubernetes.driver.node.selector.identifier to myIdentifier + will result in the driver pod having a node selector with key identifier and value + myIdentifier. Multiple driver node selector keys can be added by setting multiple configurations with this prefix. 3.3.0
spark.kubernetes.executor.node.selector.[labelKey]spark.kubernetes.executor.node.selector.[labelKey] (none) - Adds to the executor node selector of the executor pods, with key labelKey and the value as the - configuration's value. For example, setting spark.kubernetes.executor.node.selector.identifier to myIdentifier - will result in the executors having a node selector with key identifier and value - myIdentifier. Multiple executor node selector keys can be added by setting multiple configurations with this prefix. + Adds to the executor node selector of the executor pods, with key labelKey and the value as the + configuration's value. For example, setting spark.kubernetes.executor.node.selector.identifier to myIdentifier + will result in the executors having a node selector with key identifier and value + myIdentifier. Multiple executor node selector keys can be added by setting multiple configurations with this prefix. 3.3.0
spark.kubernetes.driverEnv.[EnvironmentVariableName]spark.kubernetes.driverEnv.[EnvironmentVariableName] (none) - Add the environment variable specified by EnvironmentVariableName to + Add the environment variable specified by EnvironmentVariableName to the Driver process. The user can specify multiple of these to set multiple environment variables. 2.3.0
spark.kubernetes.driver.secrets.[SecretName]spark.kubernetes.driver.secrets.[SecretName] (none) - Add the Kubernetes Secret named SecretName to the driver pod on the path specified in the value. For example, - spark.kubernetes.driver.secrets.spark-secret=/etc/secrets. + Add the Kubernetes Secret named SecretName to the driver pod on the path specified in the value. For example, + spark.kubernetes.driver.secrets.spark-secret=/etc/secrets. 2.3.0
spark.kubernetes.executor.secrets.[SecretName]spark.kubernetes.executor.secrets.[SecretName] (none) - Add the Kubernetes Secret named SecretName to the executor pod on the path specified in the value. For example, - spark.kubernetes.executor.secrets.spark-secret=/etc/secrets. + Add the Kubernetes Secret named SecretName to the executor pod on the path specified in the value. For example, + spark.kubernetes.executor.secrets.spark-secret=/etc/secrets. 2.3.0
spark.kubernetes.driver.secretKeyRef.[EnvName]spark.kubernetes.driver.secretKeyRef.[EnvName] (none) - Add as an environment variable to the driver container with name EnvName (case sensitive), the value referenced by key key in the data of the referenced Kubernetes Secret. For example, - spark.kubernetes.driver.secretKeyRef.ENV_VAR=spark-secret:key. + Add as an environment variable to the driver container with name EnvName (case sensitive), the value referenced by key in the data of the referenced Kubernetes Secret. For example, + spark.kubernetes.driver.secretKeyRef.ENV_VAR=spark-secret:key. 2.4.0
spark.kubernetes.executor.secretKeyRef.[EnvName]spark.kubernetes.executor.secretKeyRef.[EnvName] (none) - Add as an environment variable to the executor container with name EnvName (case sensitive), the value referenced by key key in the data of the referenced Kubernetes Secret. For example, - spark.kubernetes.executor.secrets.ENV_VAR=spark-secret:key. + Add as an environment variable to the executor container with name EnvName (case sensitive), the value referenced by key in the data of the referenced Kubernetes Secret. For example, + spark.kubernetes.executor.secrets.ENV_VAR=spark-secret:key. 2.4.0
spark.kubernetes.driver.volumes.[VolumeType].[VolumeName].mount.pathspark.kubernetes.driver.volumes.[VolumeType].[VolumeName].mount.path (none) - Add the Kubernetes Volume named VolumeName of the VolumeType type to the driver pod on the path specified in the value. For example, - spark.kubernetes.driver.volumes.persistentVolumeClaim.checkpointpvc.mount.path=/checkpoint. + Add the Kubernetes Volume named VolumeName of the VolumeType type to the driver pod on the path specified in the value. For example, + spark.kubernetes.driver.volumes.persistentVolumeClaim.checkpointpvc.mount.path=/checkpoint. 2.4.0
spark.kubernetes.driver.volumes.[VolumeType].[VolumeName].mount.subPathspark.kubernetes.driver.volumes.[VolumeType].[VolumeName].mount.subPath (none) Specifies a subpath to be mounted from the volume into the driver pod. - spark.kubernetes.driver.volumes.persistentVolumeClaim.checkpointpvc.mount.subPath=checkpoint. + spark.kubernetes.driver.volumes.persistentVolumeClaim.checkpointpvc.mount.subPath=checkpoint. 3.0.0
spark.kubernetes.driver.volumes.[VolumeType].[VolumeName].mount.readOnlyspark.kubernetes.driver.volumes.[VolumeType].[VolumeName].mount.readOnly (none) Specify if the mounted volume is read only or not. For example, - spark.kubernetes.driver.volumes.persistentVolumeClaim.checkpointpvc.mount.readOnly=false. + spark.kubernetes.driver.volumes.persistentVolumeClaim.checkpointpvc.mount.readOnly=false. 2.4.0
spark.kubernetes.driver.volumes.[VolumeType].[VolumeName].options.[OptionName]spark.kubernetes.driver.volumes.[VolumeType].[VolumeName].options.[OptionName] (none) - Configure Kubernetes Volume options passed to the Kubernetes with OptionName as key having specified value, must conform with Kubernetes option format. For example, - spark.kubernetes.driver.volumes.persistentVolumeClaim.checkpointpvc.options.claimName=spark-pvc-claim. + Configure Kubernetes Volume options passed to the Kubernetes with OptionName as key having specified value, must conform with Kubernetes option format. For example, + spark.kubernetes.driver.volumes.persistentVolumeClaim.checkpointpvc.options.claimName=spark-pvc-claim. 2.4.0
spark.kubernetes.executor.volumes.[VolumeType].[VolumeName].mount.pathspark.kubernetes.executor.volumes.[VolumeType].[VolumeName].mount.path (none) - Add the Kubernetes Volume named VolumeName of the VolumeType type to the executor pod on the path specified in the value. For example, - spark.kubernetes.executor.volumes.persistentVolumeClaim.checkpointpvc.mount.path=/checkpoint. + Add the Kubernetes Volume named VolumeName of the VolumeType type to the executor pod on the path specified in the value. For example, + spark.kubernetes.executor.volumes.persistentVolumeClaim.checkpointpvc.mount.path=/checkpoint. 2.4.0
spark.kubernetes.executor.volumes.[VolumeType].[VolumeName].mount.subPathspark.kubernetes.executor.volumes.[VolumeType].[VolumeName].mount.subPath (none) Specifies a subpath to be mounted from the volume into the executor pod. - spark.kubernetes.executor.volumes.persistentVolumeClaim.checkpointpvc.mount.subPath=checkpoint. + spark.kubernetes.executor.volumes.persistentVolumeClaim.checkpointpvc.mount.subPath=checkpoint. 3.0.0
spark.kubernetes.executor.volumes.[VolumeType].[VolumeName].mount.readOnlyspark.kubernetes.executor.volumes.[VolumeType].[VolumeName].mount.readOnly false Specify if the mounted volume is read only or not. For example, - spark.kubernetes.executor.volumes.persistentVolumeClaim.checkpointpvc.mount.readOnly=false. + spark.kubernetes.executor.volumes.persistentVolumeClaim.checkpointpvc.mount.readOnly=false. 2.4.0
spark.kubernetes.executor.volumes.[VolumeType].[VolumeName].options.[OptionName]spark.kubernetes.executor.volumes.[VolumeType].[VolumeName].options.[OptionName] (none) - Configure Kubernetes Volume options passed to the Kubernetes with OptionName as key having specified value. For example, - spark.kubernetes.executor.volumes.persistentVolumeClaim.checkpointpvc.options.claimName=spark-pvc-claim. + Configure Kubernetes Volume options passed to the Kubernetes with OptionName as key having specified value. For example, + spark.kubernetes.executor.volumes.persistentVolumeClaim.checkpointpvc.options.claimName=spark-pvc-claim. 2.4.0
spark.kubernetes.local.dirs.tmpfsfalsespark.kubernetes.local.dirs.tmpfsfalse - Configure the emptyDir volumes used to back SPARK_LOCAL_DIRS within the Spark driver and executor pods to use tmpfs backing i.e. RAM. See Local Storage earlier on this page + Configure the emptyDir volumes used to back SPARK_LOCAL_DIRS within the Spark driver and executor pods to use tmpfs backing i.e. RAM. See Local Storage earlier on this page for more discussion of this. 3.0.0
spark.kubernetes.memoryOverheadFactor0.1spark.kubernetes.memoryOverheadFactor0.1 - This sets the Memory Overhead Factor that will allocate memory to non-JVM memory, which includes off-heap memory allocations, non-JVM tasks, various systems processes, and tmpfs-based local directories when spark.kubernetes.local.dirs.tmpfs is true. For JVM-based jobs this value will default to 0.10 and 0.40 for non-JVM jobs. + This sets the Memory Overhead Factor that will allocate memory to non-JVM memory, which includes off-heap memory allocations, non-JVM tasks, various systems processes, and tmpfs-based local directories when spark.kubernetes.local.dirs.tmpfs is true. For JVM-based jobs this value will default to 0.10 and 0.40 for non-JVM jobs. This is done as non-JVM tasks need more non-JVM heap space and such tasks commonly fail with "Memory Overhead Exceeded" errors. This preempts this error with a higher default. - This will be overridden by the value set by spark.driver.memoryOverheadFactor and spark.executor.memoryOverheadFactor explicitly. + This will be overridden by the value set by spark.driver.memoryOverheadFactor and spark.executor.memoryOverheadFactor explicitly. 2.4.0
spark.kubernetes.pyspark.pythonVersion"3"spark.kubernetes.pyspark.pythonVersion"3" This sets the major Python version of the docker image used to run the driver and executor containers. It can be only "3". This configuration was deprecated from Spark 3.1.0, and is effectively no-op. @@ -1224,8 +1231,8 @@ See the [configuration page](configuration.html) for information on Spark config 2.4.0
spark.kubernetes.kerberos.krb5.path(none)spark.kubernetes.kerberos.krb5.path(none) Specify the local location of the krb5.conf file to be mounted on the driver and executors for Kerberos interaction. It is important to note that the KDC defined needs to be visible from inside the containers. @@ -1233,8 +1240,8 @@ See the [configuration page](configuration.html) for information on Spark config 3.0.0
spark.kubernetes.kerberos.krb5.configMapName(none)spark.kubernetes.kerberos.krb5.configMapName(none) Specify the name of the ConfigMap, containing the krb5.conf file, to be mounted on the driver and executors for Kerberos interaction. The KDC defined needs to be visible from inside the containers. The ConfigMap must also @@ -1243,8 +1250,8 @@ See the [configuration page](configuration.html) for information on Spark config 3.0.0
spark.kubernetes.hadoop.configMapName(none)spark.kubernetes.hadoop.configMapName(none) Specify the name of the ConfigMap, containing the HADOOP_CONF_DIR files, to be mounted on the driver and executors for custom Hadoop configuration. @@ -1252,8 +1259,8 @@ See the [configuration page](configuration.html) for information on Spark config 3.0.0
spark.kubernetes.kerberos.tokenSecret.name(none)spark.kubernetes.kerberos.tokenSecret.name(none) Specify the name of the secret where your existing delegation tokens are stored. This removes the need for the job user to provide any kerberos credentials for launching a job. @@ -1261,8 +1268,8 @@ See the [configuration page](configuration.html) for information on Spark config 3.0.0
spark.kubernetes.kerberos.tokenSecret.itemKey(none)spark.kubernetes.kerberos.tokenSecret.itemKey(none) Specify the item key of the data where your existing delegation tokens are stored. This removes the need for the job user to provide any kerberos credentials for launching a job. @@ -1270,43 +1277,43 @@ See the [configuration page](configuration.html) for information on Spark config 3.0.0
spark.kubernetes.driver.podTemplateFilespark.kubernetes.driver.podTemplateFile (none) Specify the local file that contains the driver pod template. For example - spark.kubernetes.driver.podTemplateFile=/path/to/driver-pod-template.yaml + spark.kubernetes.driver.podTemplateFile=/path/to/driver-pod-template.yaml 3.0.0
spark.kubernetes.driver.podTemplateContainerNamespark.kubernetes.driver.podTemplateContainerName (none) Specify the container name to be used as a basis for the driver in the given pod template. - For example spark.kubernetes.driver.podTemplateContainerName=spark-driver + For example spark.kubernetes.driver.podTemplateContainerName=spark-driver 3.0.0
spark.kubernetes.executor.podTemplateFilespark.kubernetes.executor.podTemplateFile (none) Specify the local file that contains the executor pod template. For example - spark.kubernetes.executor.podTemplateFile=/path/to/executor-pod-template.yaml + spark.kubernetes.executor.podTemplateFile=/path/to/executor-pod-template.yaml 3.0.0
spark.kubernetes.executor.podTemplateContainerNamespark.kubernetes.executor.podTemplateContainerName (none) Specify the container name to be used as a basis for the executor in the given pod template. - For example spark.kubernetes.executor.podTemplateContainerName=spark-executor + For example spark.kubernetes.executor.podTemplateContainerName=spark-executor 3.0.0
spark.kubernetes.executor.deleteOnTerminationspark.kubernetes.executor.deleteOnTermination true Specify whether executor pods should be deleted in case of failure or normal termination. @@ -1314,55 +1321,55 @@ See the [configuration page](configuration.html) for information on Spark config 3.0.0
spark.kubernetes.executor.checkAllContainersfalsespark.kubernetes.executor.checkAllContainersfalse Specify whether executor pods should be check all containers (including sidecars) or only the executor container when determining the pod status. 3.1.0
spark.kubernetes.submission.connectionTimeout10000spark.kubernetes.submission.connectionTimeout10000 Connection timeout in milliseconds for the kubernetes client to use for starting the driver. 3.0.0
spark.kubernetes.submission.requestTimeout10000spark.kubernetes.submission.requestTimeout10000 Request timeout in milliseconds for the kubernetes client to use for starting the driver. 3.0.0
spark.kubernetes.trust.certificatesfalsespark.kubernetes.trust.certificatesfalse If set to true then client can submit to kubernetes cluster only with token. 3.2.0
spark.kubernetes.driver.connectionTimeout10000spark.kubernetes.driver.connectionTimeout10000 Connection timeout in milliseconds for the kubernetes client in driver to use when requesting executors. 3.0.0
spark.kubernetes.driver.requestTimeout10000spark.kubernetes.driver.requestTimeout10000 Request timeout in milliseconds for the kubernetes client in driver to use when requesting executors. 3.0.0
spark.kubernetes.appKillPodDeletionGracePeriodspark.kubernetes.appKillPodDeletionGracePeriod (none) Specify the grace period in seconds when deleting a Spark application using spark-submit. @@ -1370,25 +1377,25 @@ See the [configuration page](configuration.html) for information on Spark config 3.0.0
spark.kubernetes.dynamicAllocation.deleteGracePeriod5sspark.kubernetes.dynamicAllocation.deleteGracePeriod5s How long to wait for executors to shut down gracefully before a forceful kill. 3.0.0
spark.kubernetes.file.upload.pathspark.kubernetes.file.upload.path (none) Path to store files at the spark submit side in cluster mode. For example: - spark.kubernetes.file.upload.path=s3a://<s3-bucket>/path - File should specified as file://path/to/file or absolute path. + spark.kubernetes.file.upload.path=s3a://<s3-bucket>/path + File should specified as file://path/to/file or absolute path. 3.0.0
spark.kubernetes.executor.decommissionLabelspark.kubernetes.executor.decommissionLabel (none) Label to be applied to pods which are exiting or being decommissioned. Intended for use @@ -1397,16 +1404,16 @@ See the [configuration page](configuration.html) for information on Spark config 3.3.0
spark.kubernetes.executor.decommissionLabelValuespark.kubernetes.executor.decommissionLabelValue (none) Value to be applied with the label when - spark.kubernetes.executor.decommissionLabel is enabled. + spark.kubernetes.executor.decommissionLabel is enabled. 3.3.0
spark.kubernetes.executor.scheduler.namespark.kubernetes.executor.scheduler.name (none) Specify the scheduler name for each executor pod. @@ -1414,7 +1421,7 @@ See the [configuration page](configuration.html) for information on Spark config 3.0.0
spark.kubernetes.driver.scheduler.namespark.kubernetes.driver.scheduler.name (none) Specify the scheduler name for driver pod. @@ -1422,7 +1429,7 @@ See the [configuration page](configuration.html) for information on Spark config 3.3.0
spark.kubernetes.scheduler.namespark.kubernetes.scheduler.name (none) Specify the scheduler name for driver and executor pods. If `spark.kubernetes.driver.scheduler.name` or @@ -1431,8 +1438,8 @@ See the [configuration page](configuration.html) for information on Spark config 3.3.0
spark.kubernetes.configMap.maxSize1572864spark.kubernetes.configMap.maxSize1572864 Max size limit for a config map. This is configurable as per limit on k8s server end. @@ -1440,8 +1447,8 @@ See the [configuration page](configuration.html) for information on Spark config 3.1.0
spark.kubernetes.executor.missingPodDetectDelta30sspark.kubernetes.executor.missingPodDetectDelta30s When a registered executor's POD is missing from the Kubernetes API server's polled list of PODs then this delta time is taken as the accepted time difference between the @@ -1451,50 +1458,50 @@ See the [configuration page](configuration.html) for information on Spark config 3.1.1
spark.kubernetes.decommission.script/opt/decom.shspark.kubernetes.decommission.script/opt/decom.sh The location of the script to use for graceful decommissioning. 3.2.0
spark.kubernetes.driver.service.deleteOnTerminationtruespark.kubernetes.driver.service.deleteOnTerminationtrue If true, driver service will be deleted on Spark application termination. If false, it will be cleaned up when the driver pod is deletion. 3.2.0
spark.kubernetes.driver.service.ipFamilyPolicySingleStackspark.kubernetes.driver.service.ipFamilyPolicySingleStack K8s IP Family Policy for Driver Service. Valid values are - SingleStack, PreferDualStack, and RequireDualStack. + SingleStack, PreferDualStack, and RequireDualStack. 3.4.0
spark.kubernetes.driver.service.ipFamiliesIPv4spark.kubernetes.driver.service.ipFamiliesIPv4 A list of IP families for K8s Driver Service. Valid values are - IPv4 and IPv6. + IPv4 and IPv6. 3.4.0
spark.kubernetes.driver.ownPersistentVolumeClaimtruespark.kubernetes.driver.ownPersistentVolumeClaimtrue If true, driver pod becomes the owner of on-demand persistent volume claims instead of the executor pods 3.2.0
spark.kubernetes.driver.reusePersistentVolumeClaimtruespark.kubernetes.driver.reusePersistentVolumeClaimtrue If true, driver pod tries to reuse driver-owned on-demand persistent volume claims of the deleted executor pods if exists. This can be useful to reduce executor pod @@ -1503,32 +1510,32 @@ See the [configuration page](configuration.html) for information on Spark config including persistent volume claims are not reusable yet. Spark will create new persistent volume claims when there exists no reusable one. In other words, the total number of persistent volume claims can be larger than the number of running executors - sometimes. This config requires spark.kubernetes.driver.ownPersistentVolumeClaim=true. + sometimes. This config requires spark.kubernetes.driver.ownPersistentVolumeClaim=true. 3.2.0
spark.kubernetes.driver.waitToReusePersistentVolumeClaimfalsespark.kubernetes.driver.waitToReusePersistentVolumeClaimfalse If true, driver pod counts the number of created on-demand persistent volume claims and wait if the number is greater than or equal to the total number of volumes which the Spark job is able to have. This config requires both - spark.kubernetes.driver.ownPersistentVolumeClaim=true and - spark.kubernetes.driver.reusePersistentVolumeClaim=true. + spark.kubernetes.driver.ownPersistentVolumeClaim=true and + spark.kubernetes.driver.reusePersistentVolumeClaim=true. 3.4.0
spark.kubernetes.executor.disableConfigMapfalsespark.kubernetes.executor.disableConfigMapfalse If true, disable ConfigMap creation for executors. 3.2.0
spark.kubernetes.driver.pod.featureStepsspark.kubernetes.driver.pod.featureSteps (none) Class names of an extra driver pod feature step implementing @@ -1540,7 +1547,7 @@ See the [configuration page](configuration.html) for information on Spark config 3.2.0
spark.kubernetes.executor.pod.featureStepsspark.kubernetes.executor.pod.featureSteps (none) Class names of an extra executor pod feature step implementing @@ -1552,8 +1559,8 @@ See the [configuration page](configuration.html) for information on Spark config 3.2.0
spark.kubernetes.allocation.maxPendingPodsInt.MaxValuespark.kubernetes.allocation.maxPendingPodsInt.MaxValue Maximum number of pending PODs allowed during executor allocation for this application. Those newly requested executors which are unknown by Kubernetes yet are @@ -1564,19 +1571,19 @@ See the [configuration page](configuration.html) for information on Spark config 3.2.0
spark.kubernetes.allocation.pods.allocatordirectspark.kubernetes.allocation.pods.allocatordirect - Allocator to use for pods. Possible values are direct (the default) - and statefulset, or a full class name of a class implementing `AbstractPodsAllocator`. + Allocator to use for pods. Possible values are direct (the default) + and statefulset, or a full class name of a class implementing `AbstractPodsAllocator`. Future version may add Job or replicaset. This is a developer API and may change or be removed at anytime. 3.3.0
spark.kubernetes.allocation.executor.timeout600sspark.kubernetes.allocation.executor.timeou600s Time to wait before a newly created executor POD request, which does not reached the POD pending state yet, considered timedout and will be deleted. @@ -1584,8 +1591,8 @@ See the [configuration page](configuration.html) for information on Spark config 3.1.0
spark.kubernetes.allocation.driver.readinessTimeout1sspark.kubernetes.allocation.driver.readinessTimeout1s Time to wait for driver pod to get ready before creating executor pods. This wait only happens on application start. If timeout happens, executor pods will still be @@ -1594,8 +1601,8 @@ See the [configuration page](configuration.html) for information on Spark config 3.1.3
spark.kubernetes.executor.enablePollingWithResourceVersionfalsespark.kubernetes.executor.enablePollingWithResourceVersionfalse If true, `resourceVersion` is set with `0` during invoking pod listing APIs in order to allow API Server-side caching. This should be used carefully. @@ -1603,24 +1610,24 @@ See the [configuration page](configuration.html) for information on Spark config 3.3.0
spark.kubernetes.executor.eventProcessingInterval1sspark.kubernetes.executor.eventProcessingInterval1s Interval between successive inspection of executor events sent from the Kubernetes API. 2.4.0
spark.kubernetes.executor.rollInterval0sspark.kubernetes.executor.rollInterval0s Interval between executor roll operations. It's disabled by default with `0s`. 3.3.0
spark.kubernetes.executor.minTasksPerExecutorBeforeRolling0spark.kubernetes.executor.minTasksPerExecutorBeforeRolling0 The minimum number of tasks per executor before rolling. Spark will not roll executors whose total number of tasks is smaller @@ -1629,8 +1636,8 @@ See the [configuration page](configuration.html) for information on Spark config 3.3.0
spark.kubernetes.executor.rollPolicyOUTLIERspark.kubernetes.executor.rollPolicyOUTLIER Executor roll policy: Valid values are ID, ADD_TIME, TOTAL_GC_TIME, TOTAL_DURATION, FAILED_TASKS, and OUTLIER (default). @@ -1658,19 +1665,25 @@ See the below table for the full list of pod specifications that will be overwri ### Pod Metadata - - +
Pod metadata keyModified valueDescription
+ + + + + + + - + - + - + - + @@ -1694,48 +1707,54 @@ See the below table for the full list of pod specifications that will be overwri ### Pod Spec -
Pod metadata keyModified valueDescription
nameValue of spark.kubernetes.driver.pod.nameValue of spark.kubernetes.driver.pod.name The driver pod name will be overwritten with either the configured or default value of - spark.kubernetes.driver.pod.name. The executor pod names will be unaffected. + spark.kubernetes.driver.pod.name. The executor pod names will be unaffected.
namespaceValue of spark.kubernetes.namespaceValue of spark.kubernetes.namespace Spark makes strong assumptions about the driver and executor namespaces. Both driver and executor namespaces will be replaced by either the configured or default spark conf value. @@ -1678,14 +1691,14 @@ See the below table for the full list of pod specifications that will be overwri
labelsAdds the labels from spark.kubernetes.{driver,executor}.label.*Adds the labels from spark.kubernetes.{driver,executor}.label.* Spark will add additional labels specified by the spark configuration.
annotationsAdds the annotations from spark.kubernetes.{driver,executor}.annotation.*Adds the annotations from spark.kubernetes.{driver,executor}.annotation.* Spark will add additional annotations specified by the spark configuration.
- +
Pod spec keyModified valueDescription
+ + + + + + + - + - + - + - + - + - +
Pod spec keyModified valueDescription
imagePullSecretsAdds image pull secrets from spark.kubernetes.container.image.pullSecretsAdds image pull secrets from spark.kubernetes.container.image.pullSecrets Additional pull secrets will be added from the spark configuration to both executor pods.
nodeSelectorAdds node selectors from spark.kubernetes.node.selector.*Adds node selectors from spark.kubernetes.node.selector.* Additional node selectors will be added from the spark configuration to both executor pods.
restartPolicy"never""never" Spark assumes that both drivers and executors never restart.
serviceAccountValue of spark.kubernetes.authenticate.driver.serviceAccountNameValue of spark.kubernetes.authenticate.driver.serviceAccountName - Spark will override serviceAccount with the value of the spark configuration for only + Spark will override serviceAccount with the value of the spark configuration for only driver pods, and only if the spark configuration is specified. Executor pods will remain unaffected.
serviceAccountNameValue of spark.kubernetes.authenticate.driver.serviceAccountNameValue of spark.kubernetes.authenticate.driver.serviceAccountName - Spark will override serviceAccountName with the value of the spark configuration for only + Spark will override serviceAccountName with the value of the spark configuration for only driver pods, and only if the spark configuration is specified. Executor pods will remain unaffected.
volumesAdds volumes from spark.kubernetes.{driver,executor}.volumes.[VolumeType].[VolumeName].mount.pathAdds volumes from spark.kubernetes.{driver,executor}.volumes.[VolumeType].[VolumeName].mount.path Spark will add volumes as specified by the spark conf, as well as additional volumes necessary for passing spark conf and pod template files. @@ -1747,26 +1766,32 @@ See the below table for the full list of pod specifications that will be overwri The following affect the driver and executor containers. All other containers in the pod spec will be unaffected. - - +
Container spec keyModified valueDescription
+ + + + + + + - + - + - + @@ -1784,15 +1809,15 @@ The following affect the driver and executor containers. All other containers in - +
Container spec keyModified valueDescription
envAdds env variables from spark.kubernetes.driverEnv.[EnvironmentVariableName]Adds env variables from spark.kubernetes.driverEnv.[EnvironmentVariableName] - Spark will add driver env variables from spark.kubernetes.driverEnv.[EnvironmentVariableName], and - executor env variables from spark.executorEnv.[EnvironmentVariableName]. + Spark will add driver env variables from spark.kubernetes.driverEnv.[EnvironmentVariableName], and + executor env variables from spark.executorEnv.[EnvironmentVariableName].
imageValue of spark.kubernetes.{driver,executor}.container.imageValue of spark.kubernetes.{driver,executor}.container.image The image will be defined by the spark configurations.
imagePullPolicyValue of spark.kubernetes.container.image.pullPolicyValue of spark.kubernetes.container.image.pullPolicy Spark will override the pull policy for both driver and executors. resources See description - The cpu limits are set by spark.kubernetes.{driver,executor}.limit.cores. The cpu is set by - spark.{driver,executor}.cores. The memory request and limit are set by summing the values of - spark.{driver,executor}.memory and spark.{driver,executor}.memoryOverhead. - Other resource limits are set by spark.{driver,executor}.resources.{resourceName}.* configs. + The cpu limits are set by spark.kubernetes.{driver,executor}.limit.cores. The cpu is set by + spark.{driver,executor}.cores. The memory request and limit are set by summing the values of + spark.{driver,executor}.memory and spark.{driver,executor}.memoryOverhead. + Other resource limits are set by spark.{driver,executor}.resources.{resourceName}.* configs.
volumeMountsAdd volumes from spark.kubernetes.driver.volumes.[VolumeType].[VolumeName].mount.{path,readOnly}Add volumes from spark.kubernetes.driver.volumes.[VolumeType].[VolumeName].mount.{path,readOnly} Spark will add volumes as specified by the spark conf, as well as additional volumes necessary for passing spark conf and pod template files. diff --git a/docs/running-on-yarn.md b/docs/running-on-yarn.md index 3dfa63e1cb2e3..de39a6129b2c5 100644 --- a/docs/running-on-yarn.md +++ b/docs/running-on-yarn.md @@ -143,118 +143,124 @@ To use a custom metrics.properties for the application master and executors, upd #### Spark Properties - - +
Property NameDefaultMeaningSince Version
+ + + + + + + + - - + + - - + + - - + + - - + + - - + + - - + + - - + + - - + + - - + + - - + + - + - - + + - - + + - - + + - + - + - + - + - - + + - - + + - + - - + + - + - + - + - - + + - + - + - + - - + + - + - + - + - + - - + + - + - + - + - + - + - + - + - + - + - + - + - - + + - - + + - - + + - - + + - - + + - +
Property NameDefaultMeaningSince Version
spark.yarn.am.memory512mspark.yarn.am.memory512m - Amount of memory to use for the YARN Application Master in client mode, in the same format as JVM memory strings (e.g. 512m, 2g). - In cluster mode, use spark.driver.memory instead. -

- Use lower-case suffixes, e.g. k, m, g, t, and p, for kibi-, mebi-, gibi-, tebi-, and pebibytes, respectively. + Amount of memory to use for the YARN Application Master in client mode, in the same format as JVM memory strings (e.g. 512m, 2g). + In cluster mode, use spark.driver.memory instead.
+ Use lower-case suffixes, e.g. k, m, g, t, and p, for kibi-, mebi-, gibi-, tebi-, and pebibytes, respectively.

1.3.0
spark.yarn.am.resource.{resource-type}.amount(none)spark.yarn.am.resource.{resource-type}.amount(none) Amount of resource to use for the YARN Application Master in client mode. - In cluster mode, use spark.yarn.driver.resource.<resource-type>.amount instead. + In cluster mode, use spark.yarn.driver.resource.<resource-type>.amount instead. Please note that this feature can be used only with YARN 3.0+ For reference, see YARN Resource Model documentation -

+
Example: - To request GPU resources from YARN, use: spark.yarn.am.resource.yarn.io/gpu.amount + To request GPU resources from YARN, use: spark.yarn.am.resource.yarn.io/gpu.amount

3.0.0
spark.yarn.applicationTypeSPARKspark.yarn.applicationTypeSPARK - Defines more specific application types, e.g. SPARK, SPARK-SQL, SPARK-STREAMING, - SPARK-MLLIB and SPARK-GRAPH. Please be careful not to exceed 20 characters. + Defines more specific application types, e.g. SPARK, SPARK-SQL, SPARK-STREAMING, + SPARK-MLLIB and SPARK-GRAPH. Please be careful not to exceed 20 characters. 3.1.0
spark.yarn.driver.resource.{resource-type}.amount(none)spark.yarn.driver.resource.{resource-type}.amount(none) Amount of resource to use for the YARN Application Master in cluster mode. Please note that this feature can be used only with YARN 3.0+ For reference, see YARN Resource Model documentation

Example: - To request GPU resources from YARN, use: spark.yarn.driver.resource.yarn.io/gpu.amount + To request GPU resources from YARN, use: spark.yarn.driver.resource.yarn.io/gpu.amount

3.0.0
spark.yarn.executor.resource.{resource-type}.amount(none)spark.yarn.executor.resource.{resource-type}.amount(none) Amount of resource to use per executor process. Please note that this feature can be used only with YARN 3.0+ For reference, see YARN Resource Model documentation -

+
Example: - To request GPU resources from YARN, use: spark.yarn.executor.resource.yarn.io/gpu.amount + To request GPU resources from YARN, use: spark.yarn.executor.resource.yarn.io/gpu.amount

3.0.0
spark.yarn.resourceGpuDeviceNameyarn.io/gpuspark.yarn.resourceGpuDeviceNameyarn.io/gpu - Specify the mapping of the Spark resource type of gpu to the YARN resource - representing a GPU. By default YARN uses yarn.io/gpu but if YARN has been + Specify the mapping of the Spark resource type of gpu to the YARN resource + representing a GPU. By default YARN uses yarn.io/gpu but if YARN has been configured with a custom resource type, this allows remapping it. - Applies when using the spark.{driver/executor}.resource.gpu.* configs. + Applies when using the spark.{driver/executor}.resource.gpu.* configs. 3.2.1
spark.yarn.resourceFpgaDeviceNameyarn.io/fpgaspark.yarn.resourceFpgaDeviceNameyarn.io/fpga - Specify the mapping of the Spark resource type of fpga to the YARN resource - representing a FPGA. By default YARN uses yarn.io/fpga but if YARN has been + Specify the mapping of the Spark resource type of fpga to the YARN resource + representing a FPGA. By default YARN uses yarn.io/fpga but if YARN has been configured with a custom resource type, this allows remapping it. - Applies when using the spark.{driver/executor}.resource.fpga.* configs. + Applies when using the spark.{driver/executor}.resource.fpga.* configs. 3.2.1
spark.yarn.am.cores1spark.yarn.am.cores1 Number of cores to use for the YARN Application Master in client mode. - In cluster mode, use spark.driver.cores instead. + In cluster mode, use spark.driver.cores instead. 1.3.0
spark.yarn.am.waitTime100sspark.yarn.am.waitTime100s - Only used in cluster mode. Time for the YARN Application Master to wait for the + Only used in cluster mode. Time for the YARN Application Master to wait for the SparkContext to be initialized. 1.3.0
spark.yarn.submit.file.replicationThe default HDFS replication (usually 3)spark.yarn.submit.file.replicationThe default HDFS replication (usually 3) HDFS replication level for the files uploaded into HDFS for the application. These include things like the Spark jar, the app jar, and any distributed cache files/archives. 0.8.1
spark.yarn.stagingDirspark.yarn.stagingDir Current user's home directory in the filesystem Staging directory used while submitting applications. @@ -262,46 +268,46 @@ To use a custom metrics.properties for the application master and executors, upd 2.0.0
spark.yarn.preserve.staging.filesfalsespark.yarn.preserve.staging.filesfalse - Set to true to preserve the staged files (Spark jar, app jar, distributed cache files) at the end of the job rather than delete them. + Set to true to preserve the staged files (Spark jar, app jar, distributed cache files) at the end of the job rather than delete them. 1.1.0
spark.yarn.scheduler.heartbeat.interval-ms3000spark.yarn.scheduler.heartbeat.interval-ms3000 The interval in ms in which the Spark application master heartbeats into the YARN ResourceManager. The value is capped at half the value of YARN's configuration for the expiry interval, i.e. - yarn.am.liveness-monitor.expiry-interval-ms. + yarn.am.liveness-monitor.expiry-interval-ms. 0.8.1
spark.yarn.scheduler.initial-allocation.interval200msspark.yarn.scheduler.initial-allocation.interval200ms The initial interval in which the Spark application master eagerly heartbeats to the YARN ResourceManager when there are pending container allocation requests. It should be no larger than - spark.yarn.scheduler.heartbeat.interval-ms. The allocation interval will doubled on + spark.yarn.scheduler.heartbeat.interval-ms. The allocation interval will doubled on successive eager heartbeats if pending containers still exist, until - spark.yarn.scheduler.heartbeat.interval-ms is reached. + spark.yarn.scheduler.heartbeat.interval-ms is reached. 1.4.0
spark.yarn.historyServer.addressspark.yarn.historyServer.address (none) - The address of the Spark history server, e.g. host.com:18080. The address should not contain a scheme (http://). Defaults to not being set since the history server is an optional service. This address is given to the YARN ResourceManager when the Spark application finishes to link the application from the ResourceManager UI to the Spark history server UI. - For this property, YARN properties can be used as variables, and these are substituted by Spark at runtime. For example, if the Spark history server runs on the same node as the YARN ResourceManager, it can be set to ${hadoopconf-yarn.resourcemanager.hostname}:18080. + The address of the Spark history server, e.g. host.com:18080. The address should not contain a scheme (http://). Defaults to not being set since the history server is an optional service. This address is given to the YARN ResourceManager when the Spark application finishes to link the application from the ResourceManager UI to the Spark history server UI. + For this property, YARN properties can be used as variables, and these are substituted by Spark at runtime. For example, if the Spark history server runs on the same node as the YARN ResourceManager, it can be set to ${hadoopconf-yarn.resourcemanager.hostname}:18080. 1.0.0
spark.yarn.dist.archivesspark.yarn.dist.archives (none) Comma separated list of archives to be extracted into the working directory of each executor. @@ -309,7 +315,7 @@ To use a custom metrics.properties for the application master and executors, upd 1.0.0
spark.yarn.dist.filesspark.yarn.dist.files (none) Comma-separated list of files to be placed in the working directory of each executor. @@ -317,7 +323,7 @@ To use a custom metrics.properties for the application master and executors, upd 1.0.0
spark.yarn.dist.jarsspark.yarn.dist.jars (none) Comma-separated list of jars to be placed in the working directory of each executor. @@ -325,8 +331,8 @@ To use a custom metrics.properties for the application master and executors, upd 2.0.0
spark.yarn.dist.forceDownloadSchemes(none)spark.yarn.dist.forceDownloadSchemes(none) Comma-separated list of schemes for which resources will be downloaded to the local disk prior to being added to YARN's distributed cache. For use in cases where the YARN service does not @@ -336,47 +342,47 @@ To use a custom metrics.properties for the application master and executors, upd 2.3.0
spark.executor.instances2spark.executor.instances2 - The number of executors for static allocation. With spark.dynamicAllocation.enabled, the initial set of executors will be at least this large. + The number of executors for static allocation. With spark.dynamicAllocation.enabled, the initial set of executors will be at least this large. 1.0.0
spark.yarn.am.memoryOverheadspark.yarn.am.memoryOverhead AM memory * 0.10, with minimum of 384 - Same as spark.driver.memoryOverhead, but for the YARN Application Master in client mode. + Same as spark.driver.memoryOverhead, but for the YARN Application Master in client mode. 1.3.0
spark.yarn.queuedefaultspark.yarn.queuedefault The name of the YARN queue to which the application is submitted. 1.0.0
spark.yarn.jarsspark.yarn.jars (none) List of libraries containing Spark code to distribute to YARN containers. By default, Spark on YARN will use Spark jars installed locally, but the Spark jars can also be in a world-readable location on HDFS. This allows YARN to cache it on nodes so that it doesn't need to be distributed each time an application runs. To point to jars on HDFS, for example, - set this configuration to hdfs:///some/path. Globs are allowed. + set this configuration to hdfs:///some/path. Globs are allowed. 2.0.0
spark.yarn.archivespark.yarn.archive (none) An archive containing needed Spark jars for distribution to the YARN cache. If set, this - configuration replaces spark.yarn.jars and the archive is used in all the + configuration replaces spark.yarn.jars and the archive is used in all the application's containers. The archive should contain jar files in its root directory. Like with the previous option, the archive can also be hosted on HDFS to speed up file distribution. @@ -384,38 +390,38 @@ To use a custom metrics.properties for the application master and executors, upd 2.0.0
spark.yarn.appMasterEnv.[EnvironmentVariableName]spark.yarn.appMasterEnv.[EnvironmentVariableName] (none) - Add the environment variable specified by EnvironmentVariableName to the + Add the environment variable specified by EnvironmentVariableName to the Application Master process launched on YARN. The user can specify multiple of - these and to set multiple environment variables. In cluster mode this controls - the environment of the Spark driver and in client mode it only controls + these and to set multiple environment variables. In cluster mode this controls + the environment of the Spark driver and in client mode it only controls the environment of the executor launcher. 1.1.0
spark.yarn.containerLauncherMaxThreads25spark.yarn.containerLauncherMaxThreads25 The maximum number of threads to use in the YARN Application Master for launching executor containers. 1.2.0
spark.yarn.am.extraJavaOptionsspark.yarn.am.extraJavaOptions (none) A string of extra JVM options to pass to the YARN Application Master in client mode. - In cluster mode, use spark.driver.extraJavaOptions instead. Note that it is illegal + In cluster mode, use spark.driver.extraJavaOptions instead. Note that it is illegal to set maximum heap size (-Xmx) settings with this option. Maximum heap size settings can be set - with spark.yarn.am.memory + with spark.yarn.am.memory 1.3.0
spark.yarn.am.extraLibraryPathspark.yarn.am.extraLibraryPath (none) Set a special library path to use when launching the YARN Application Master in client mode. @@ -423,22 +429,22 @@ To use a custom metrics.properties for the application master and executors, upd 1.4.0
spark.yarn.populateHadoopClasspathspark.yarn.populateHadoopClasspath - For with-hadoop Spark distribution, this is set to false; - for no-hadoop distribution, this is set to true. + For with-hadoop Spark distribution, this is set to false; + for no-hadoop distribution, this is set to true. - Whether to populate Hadoop classpath from yarn.application.classpath and - mapreduce.application.classpath Note that if this is set to false, - it requires a with-Hadoop Spark distribution that bundles Hadoop runtime or + Whether to populate Hadoop classpath from yarn.application.classpath and + mapreduce.application.classpath Note that if this is set to false, + it requires a with-Hadoop Spark distribution that bundles Hadoop runtime or user has to provide a Hadoop installation separately. 2.4.6
spark.yarn.maxAppAttemptsyarn.resourcemanager.am.max-attempts in YARNspark.yarn.maxAppAttemptsyarn.resourcemanager.am.max-attempts in YARN The maximum number of attempts that will be made to submit the application. It should be no larger than the global number of max attempts in the YARN configuration. @@ -446,7 +452,7 @@ To use a custom metrics.properties for the application master and executors, upd 1.3.0
spark.yarn.am.attemptFailuresValidityIntervalspark.yarn.am.attemptFailuresValidityInterval (none) Defines the validity interval for AM failure tracking. @@ -456,7 +462,7 @@ To use a custom metrics.properties for the application master and executors, upd 1.6.0
spark.yarn.am.clientModeTreatDisconnectAsFailedspark.yarn.am.clientModeTreatDisconnectAsFailed false Treat yarn-client unclean disconnects as failures. In yarn-client mode, normally the application will always finish @@ -469,7 +475,7 @@ To use a custom metrics.properties for the application master and executors, upd 3.3.0
spark.yarn.am.clientModeExitOnErrorspark.yarn.am.clientModeExitOnError false In yarn-client mode, when this is true, if driver got application report with final status of KILLED or FAILED, @@ -479,30 +485,30 @@ To use a custom metrics.properties for the application master and executors, upd 3.3.0
spark.yarn.am.tokenConfRegexspark.yarn.am.tokenConfRegex (none) The value of this config is a regex expression used to grep a list of config entries from the job's configuration file (e.g., hdfs-site.xml) and send to RM, which uses them when renewing delegation tokens. A typical use case of this feature is to support delegation tokens in an environment where a YARN cluster needs to talk to multiple downstream HDFS clusters, where the YARN RM may not have configs (e.g., dfs.nameservices, dfs.ha.namenodes.*, dfs.namenode.rpc-address.*) to connect to these clusters. - In this scenario, Spark users can specify the config value to be ^dfs.nameservices$|^dfs.namenode.rpc-address.*$|^dfs.ha.namenodes.*$ to parse - these HDFS configs from the job's local configuration files. This config is very similar to mapreduce.job.send-token-conf. Please check YARN-5910 for more details. + In this scenario, Spark users can specify the config value to be ^dfs.nameservices$|^dfs.namenode.rpc-address.*$|^dfs.ha.namenodes.*$ to parse + these HDFS configs from the job's local configuration files. This config is very similar to mapreduce.job.send-token-conf. Please check YARN-5910 for more details. 3.3.0
spark.yarn.submit.waitAppCompletiontruespark.yarn.submit.waitAppCompletiontrue In YARN cluster mode, controls whether the client waits to exit until the application completes. - If set to true, the client process will stay alive reporting the application's status. + If set to true, the client process will stay alive reporting the application's status. Otherwise, the client process will exit after submission. 1.4.0
spark.yarn.am.nodeLabelExpressionspark.yarn.am.nodeLabelExpression (none) A YARN node label expression that restricts the set of nodes AM will be scheduled on. @@ -512,7 +518,7 @@ To use a custom metrics.properties for the application master and executors, upd 1.6.0
spark.yarn.executor.nodeLabelExpressionspark.yarn.executor.nodeLabelExpression (none) A YARN node label expression that restricts the set of nodes executors will be scheduled on. @@ -522,7 +528,7 @@ To use a custom metrics.properties for the application master and executors, upd 1.4.0
spark.yarn.tagsspark.yarn.tags (none) Comma-separated list of strings to pass through as YARN application tags appearing @@ -531,7 +537,7 @@ To use a custom metrics.properties for the application master and executors, upd 1.5.0
spark.yarn.priorityspark.yarn.priority (none) Application priority for YARN to define pending applications ordering policy, those with higher @@ -541,41 +547,41 @@ To use a custom metrics.properties for the application master and executors, upd 3.0.0
spark.yarn.config.gatewayPathspark.yarn.config.gatewayPath (none) A path that is valid on the gateway host (the host where a Spark application is started) but may differ for paths for the same resource in other nodes in the cluster. Coupled with - spark.yarn.config.replacementPath, this is used to support clusters with + spark.yarn.config.replacementPath, this is used to support clusters with heterogeneous configurations, so that Spark can correctly launch remote processes. -

+
The replacement path normally will contain a reference to some environment variable exported by YARN (and, thus, visible to Spark containers). -

- For example, if the gateway node has Hadoop libraries installed on /disk1/hadoop, and - the location of the Hadoop install is exported by YARN as the HADOOP_HOME - environment variable, setting this value to /disk1/hadoop and the replacement path to - $HADOOP_HOME will make sure that paths used to launch remote processes properly +
+ For example, if the gateway node has Hadoop libraries installed on /disk1/hadoop, and + the location of the Hadoop install is exported by YARN as the HADOOP_HOME + environment variable, setting this value to /disk1/hadoop and the replacement path to + $HADOOP_HOME will make sure that paths used to launch remote processes properly reference the local YARN configuration.

1.5.0
spark.yarn.config.replacementPathspark.yarn.config.replacementPath (none) - See spark.yarn.config.gatewayPath. + See spark.yarn.config.gatewayPath. 1.5.0
spark.yarn.rolledLog.includePatternspark.yarn.rolledLog.includePattern (none) Java Regex to filter the log files which match the defined include pattern and those log files will be aggregated in a rolling fashion. This will be used with YARN's rolling log aggregation, to enable this feature in YARN side - yarn.nodemanager.log-aggregation.roll-monitoring-interval-seconds should be + yarn.nodemanager.log-aggregation.roll-monitoring-interval-seconds should be configured in yarn-site.xml. The Spark log4j appender needs be changed to use FileAppender or another appender that can handle the files being removed while it is running. Based on the file name configured in the log4j configuration (like spark.log), the user should set the @@ -584,7 +590,7 @@ To use a custom metrics.properties for the application master and executors, upd 2.0.0
spark.yarn.rolledLog.excludePatternspark.yarn.rolledLog.excludePattern (none) Java Regex to filter the log files which match the defined exclude pattern @@ -594,17 +600,17 @@ To use a custom metrics.properties for the application master and executors, upd 2.0.0
spark.yarn.executor.launch.excludeOnFailure.enabledspark.yarn.executor.launch.excludeOnFailure.enabled false Flag to enable exclusion of nodes having YARN resource allocation problems. The error limit for excluding can be configured by - spark.excludeOnFailure.application.maxFailedExecutorsPerNode. + spark.excludeOnFailure.application.maxFailedExecutorsPerNode. 2.4.0
spark.yarn.exclude.nodesspark.yarn.exclude.nodes (none) Comma-separated list of YARN node names which are excluded from resource allocation. @@ -612,7 +618,7 @@ To use a custom metrics.properties for the application master and executors, upd 3.0.0
spark.yarn.metrics.namespacespark.yarn.metrics.namespace (none) The root namespace for AM metrics reporting. @@ -621,16 +627,16 @@ To use a custom metrics.properties for the application master and executors, upd 2.4.0
spark.yarn.report.interval1sspark.yarn.report.interval1s Interval between reports of the current Spark job status in cluster mode. 0.9.0
spark.yarn.report.loggingFrequency30spark.yarn.report.loggingFrequency30 Maximum number of application reports processed until the next application status is logged. If there is a change of state, the application status will be logged regardless @@ -639,16 +645,16 @@ To use a custom metrics.properties for the application master and executors, upd 3.5.0
spark.yarn.clientLaunchMonitorInterval1sspark.yarn.clientLaunchMonitorInterval1s Interval between requests for status the client mode AM when starting the app. 2.3.0
spark.yarn.includeDriverLogsLinkfalsespark.yarn.includeDriverLogsLinkfalse In cluster mode, whether the client application report includes links to the driver container's logs. This requires polling the ResourceManager's REST API, so it @@ -657,8 +663,8 @@ To use a custom metrics.properties for the application master and executors, upd 3.1.0
spark.yarn.unmanagedAM.enabledfalsespark.yarn.unmanagedAM.enabledfalse In client mode, whether to launch the Application Master service as part of the client using unmanaged am. @@ -666,7 +672,7 @@ To use a custom metrics.properties for the application master and executors, upd 3.0.0
spark.yarn.shuffle.server.recovery.disabledspark.yarn.shuffle.server.recovery.disabled false Set to true for applications that have higher security requirements and prefer that their @@ -721,7 +727,7 @@ To use a custom metrics.properties for the application master and executors, upd For example, suppose you would like to point log url link to Job History Server directly instead of let NodeManager http server redirects it, you can configure `spark.history.custom.executor.log.url` as below: -{{HTTP_SCHEME}}<JHS_HOST>:<JHS_PORT>/jobhistory/logs/{{NM_HOST}}:{{NM_PORT}}/{{CONTAINER_ID}}/{{CONTAINER_ID}}/{{USER}}/{{FILE_NAME}}?start=-4096 +{{HTTP_SCHEME}}<JHS_HOST>:<JHS_PORT>/jobhistory/logs/{{NM_HOST}}:{{NM_PORT}}/{{CONTAINER_ID}}/{{CONTAINER_ID}}/{{USER}}/{{FILE_NAME}}?start=-4096 NOTE: you need to replace `` and `` with actual value. @@ -766,34 +772,39 @@ staging directory of the Spark application. ## YARN-specific Kerberos Configuration - - +
Property NameDefaultMeaningSince Version
+ - + + + + + + - + - + - + @@ -865,38 +876,48 @@ to avoid garbage collection issues during shuffle. The following extra configuration options are available when the shuffle service is running on YARN: -
spark.kerberos.keytabProperty NameDefaultMeaningSince Version
spark.kerberos.keytab (none) The full path to the file that contains the keytab for the principal specified above. This keytab will be copied to the node running the YARN Application Master via the YARN Distributed Cache, and will be used for renewing the login tickets and the delegation tokens periodically. Equivalent to - the --keytab command line argument. + the --keytab command line argument.
(Works also with the "local" master.) 		3.0.0
spark.kerberos.principalspark.kerberos.principal (none) Principal to be used to login to KDC, while running on secure clusters. Equivalent to the - --principal command line argument. + --principal command line argument.
(Works also with the "local" master.) 		3.0.0
spark.yarn.kerberos.relogin.periodspark.yarn.kerberos.relogin.period 1m How often to check whether the kerberos TGT should be renewed. This should be set to a value @@ -803,12 +814,12 @@ staging directory of the Spark application. 2.3.0
spark.yarn.kerberos.renewal.excludeHadoopFileSystemsspark.yarn.kerberos.renewal.excludeHadoopFileSystems (none) A comma-separated list of Hadoop filesystems for whose hosts will be excluded from delegation - token renewal at resource scheduler. For example, spark.yarn.kerberos.renewal.excludeHadoopFileSystems=hdfs://nn1.com:8032, - hdfs://nn2.com:8032. This is known to work under YARN for now, so YARN Resource Manager won't renew tokens for the application. + token renewal at resource scheduler. For example, spark.yarn.kerberos.renewal.excludeHadoopFileSystems=hdfs://nn1.com:8032, + hdfs://nn2.com:8032. This is known to work under YARN for now, so YARN Resource Manager won't renew tokens for the application. Note that as resource scheduler does not renew token, so any application running longer than the original token expiration that tries to use that token will likely fail.
- +
Property NameDefaultMeaning
+ + + + + + + + - - + + + - - + + + - - + + + - +
Property NameDefaultMeaningSince Version
spark.yarn.shuffle.stopOnFailurefalsespark.yarn.shuffle.stopOnFailurefalse Whether to stop the NodeManager when there's a failure in the Spark Shuffle Service's initialization. This prevents application failures caused by running containers on NodeManagers where the Spark Shuffle Service is not running.
spark.yarn.shuffle.service.metrics.namespacesparkShuffleServicespark.yarn.shuffle.service.metrics.namespacesparkShuffleService The namespace to use when emitting shuffle service metrics into Hadoop metrics2 system of the NodeManager.
spark.yarn.shuffle.service.logs.namespace(not set)spark.yarn.shuffle.service.logs.namespace(not set) A namespace which will be appended to the class name when forming the logger name to use for emitting logs from the YARN shuffle service, like - org.apache.spark.network.yarn.YarnShuffleService.logsNamespaceValue. Since some logging frameworks + org.apache.spark.network.yarn.YarnShuffleService.logsNamespaceValue. Since some logging frameworks may expect the logger name to look like a class name, it's generally recommended to provide a value which would be a valid Java package or class name and not include spaces.
spark.shuffle.service.db.backendspark.shuffle.service.db.backend ROCKSDB When work-preserving restart is enabled in YARN, this is used to specify the disk-base store used From d8c54609c9a5d2fc666566bde68cb8f9a1407e76 Mon Sep 17 00:00:00 2001 From: panbingkun Date: Wed, 3 Jan 2024 16:22:59 +0800 Subject: [PATCH 2/7] Make the table fixed width --- docs/css/custom.css | 6 ++++++ docs/running-on-kubernetes.md | 8 ++++---- docs/running-on-yarn.md | 6 +++--- 3 files changed, 13 insertions(+), 7 deletions(-) diff --git a/docs/css/custom.css b/docs/css/custom.css index e80ca506a74c3..f55434df6c369 100644 --- a/docs/css/custom.css +++ b/docs/css/custom.css @@ -1126,3 +1126,9 @@ table tr { table tr:nth-child(2n) { background-color: #F1F4F5; /* Sets a different background color for even rows */ } + +.property-table { + width: 100%; + table-layout: fixed; + word-wrap: break-word; +} diff --git a/docs/running-on-kubernetes.md b/docs/running-on-kubernetes.md index 4f3894f28b8ca..c9dac95c80422 100644 --- a/docs/running-on-kubernetes.md +++ b/docs/running-on-kubernetes.md @@ -592,7 +592,7 @@ See the [configuration page](configuration.html) for information on Spark config #### Spark Properties - +
@@ -1665,7 +1665,7 @@ See the below table for the full list of pod specifications that will be overwri ### Pod Metadata -
Property Name
+
@@ -1707,7 +1707,7 @@ See the below table for the full list of pod specifications that will be overwri ### Pod Spec -
Pod metadata key
+
@@ -1766,7 +1766,7 @@ See the below table for the full list of pod specifications that will be overwri The following affect the driver and executor containers. All other containers in the pod spec will be unaffected. -
Pod spec key
+
diff --git a/docs/running-on-yarn.md b/docs/running-on-yarn.md index de39a6129b2c5..cde656c54ac7d 100644 --- a/docs/running-on-yarn.md +++ b/docs/running-on-yarn.md @@ -143,7 +143,7 @@ To use a custom metrics.properties for the application master and executors, upd #### Spark Properties -
Container spec key
+
@@ -772,7 +772,7 @@ staging directory of the Spark application. ## YARN-specific Kerberos Configuration -
Property Name
+
@@ -876,7 +876,7 @@ to avoid garbage collection issues during shuffle. The following extra configuration options are available when the shuffle service is running on YARN: -
Property Name
+
From 36355121284589a42ecf1c11d56459d4c537729a Mon Sep 17 00:00:00 2001 From: panbingkun Date: Wed, 3 Jan 2024 16:51:33 +0800 Subject: [PATCH 3/7] Revert "Make the table fixed width" This reverts commit d8c54609c9a5d2fc666566bde68cb8f9a1407e76. --- docs/css/custom.css | 6 ------ docs/running-on-kubernetes.md | 8 ++++---- docs/running-on-yarn.md | 6 +++--- 3 files changed, 7 insertions(+), 13 deletions(-) diff --git a/docs/css/custom.css b/docs/css/custom.css index f55434df6c369..e80ca506a74c3 100644 --- a/docs/css/custom.css +++ b/docs/css/custom.css @@ -1126,9 +1126,3 @@ table tr { table tr:nth-child(2n) { background-color: #F1F4F5; /* Sets a different background color for even rows */ } - -.property-table { - width: 100%; - table-layout: fixed; - word-wrap: break-word; -} diff --git a/docs/running-on-kubernetes.md b/docs/running-on-kubernetes.md index c9dac95c80422..4f3894f28b8ca 100644 --- a/docs/running-on-kubernetes.md +++ b/docs/running-on-kubernetes.md @@ -592,7 +592,7 @@ See the [configuration page](configuration.html) for information on Spark config #### Spark Properties -
Property Name
+
@@ -1665,7 +1665,7 @@ See the below table for the full list of pod specifications that will be overwri ### Pod Metadata -
Property Name
+
@@ -1707,7 +1707,7 @@ See the below table for the full list of pod specifications that will be overwri ### Pod Spec -
Pod metadata key
+
@@ -1766,7 +1766,7 @@ See the below table for the full list of pod specifications that will be overwri The following affect the driver and executor containers. All other containers in the pod spec will be unaffected. -
Pod spec key
+
diff --git a/docs/running-on-yarn.md b/docs/running-on-yarn.md index cde656c54ac7d..de39a6129b2c5 100644 --- a/docs/running-on-yarn.md +++ b/docs/running-on-yarn.md @@ -143,7 +143,7 @@ To use a custom metrics.properties for the application master and executors, upd #### Spark Properties -
Container spec key
+
@@ -772,7 +772,7 @@ staging directory of the Spark application. ## YARN-specific Kerberos Configuration -
Property Name
+
@@ -876,7 +876,7 @@ to avoid garbage collection issues during shuffle. The following extra configuration options are available when the shuffle service is running on YARN: -
Property Name
+
From cc110578a2833dac053bde04acdfa015d34e2b88 Mon Sep 17 00:00:00 2001 From: panbingkun Date: Wed, 3 Jan 2024 16:51:47 +0800 Subject: [PATCH 4/7] Revert "[SPARK-46546][DOCS] Fix the formatting of tables in `running-on-kubernetes` and `running-on-yarn` pages" This reverts commit 674496de9b1d9ae69057c8e019a5789a34f0c2dd. --- docs/running-on-kubernetes.md | 573 ++++++++++++++++------------------ docs/running-on-yarn.md | 325 +++++++++---------- 2 files changed, 426 insertions(+), 472 deletions(-) diff --git a/docs/running-on-kubernetes.md b/docs/running-on-kubernetes.md index 4f3894f28b8ca..4b4dc9d304fbd 100644 --- a/docs/running-on-kubernetes.md +++ b/docs/running-on-kubernetes.md @@ -592,30 +592,23 @@ See the [configuration page](configuration.html) for information on Spark config #### Spark Properties -
Property Name
- - - - - - - - - - - +
Property NameDefaultMeaningSince Version
spark.kubernetes.context(none)
+ + + + - - + + - - + + - - + + - - + + - - + + - - + + - - + + - - + + - - + + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - - + + - - + + - + - + - + - + - + - + - + - + - + - + - + - + - + - - + + - - + + - - + + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - - + + - - + + - - + + - - + + - - + + - - + + - - + + - - + + - + - + - + - + - + - - + + - - + + - - + + - - + + - - + + - - + + - + - - + + - + - + - + - + - + - + - - + + - - + + - - + + - - + + - - + + - - + + - - + + - - + + - - + + - - + + - + - + - - + + - - + + - - + + - - + + - - + + - - + + - - + + - - + + - - + +
Property NameDefaultMeaningSince Version
spark.kubernetes.context(none) The context from the user Kubernetes configuration file used for the initial auto-configuration of the Kubernetes client library. When not specified then the users current context is used. NB: Many of the auto-configured settings can be overridden by the use of other Spark - configuration properties e.g. spark.kubernetes.namespace. + configuration properties e.g. spark.kubernetes.namespace. 3.0.0
spark.kubernetes.driver.masterhttps://kubernetes.default.svcspark.kubernetes.driver.masterhttps://kubernetes.default.svc The internal Kubernetes master (API server) address to be used for driver to request executors or 'local[*]' for driver-pod-only mode. @@ -623,68 +616,68 @@ See the [configuration page](configuration.html) for information on Spark config 3.0.0
spark.kubernetes.namespacedefaultspark.kubernetes.namespacedefault The namespace that will be used for running the driver and executor pods. 2.3.0
spark.kubernetes.container.image(none)spark.kubernetes.container.image(none) Container image to use for the Spark application. - This is usually of the form example.com/repo/spark:v1.0.0. + This is usually of the form example.com/repo/spark:v1.0.0. This configuration is required and must be provided by the user, unless explicit images are provided for each different container type. 2.3.0
spark.kubernetes.driver.container.image(value of spark.kubernetes.container.image)spark.kubernetes.driver.container.image(value of spark.kubernetes.container.image) Custom container image to use for the driver. 2.3.0
spark.kubernetes.executor.container.image(value of spark.kubernetes.container.image)spark.kubernetes.executor.container.image(value of spark.kubernetes.container.image) Custom container image to use for executors. 2.3.0
spark.kubernetes.container.image.pullPolicyIfNotPresentspark.kubernetes.container.image.pullPolicyIfNotPresent Container image pull policy used when pulling images within Kubernetes. - Valid values are Always, Never, and IfNotPresent. + Valid values are Always, Never, and IfNotPresent. 2.3.0
spark.kubernetes.container.image.pullSecretsspark.kubernetes.container.image.pullSecrets Comma separated list of Kubernetes secrets used to pull images from private image registries. 2.4.0
spark.kubernetes.allocation.batch.size5spark.kubernetes.allocation.batch.size5 Number of pods to launch at once in each round of executor pod allocation. 2.3.0
spark.kubernetes.allocation.batch.delay1sspark.kubernetes.allocation.batch.delay1s Time to wait between each round of executor pod allocation. Specifying values less than 1 second may lead to excessive CPU usage on the spark driver. @@ -692,167 +685,167 @@ See the [configuration page](configuration.html) for information on Spark config 2.3.0
spark.kubernetes.authenticate.submission.caCertFilespark.kubernetes.authenticate.submission.caCertFile (none) Path to the CA cert file for connecting to the Kubernetes API server over TLS when starting the driver. This file must be located on the submitting machine's disk. Specify this as a path as opposed to a URI (i.e. do not provide - a scheme). In client mode, use spark.kubernetes.authenticate.caCertFile instead. + a scheme). In client mode, use spark.kubernetes.authenticate.caCertFile instead. 2.3.0
spark.kubernetes.authenticate.submission.clientKeyFilespark.kubernetes.authenticate.submission.clientKeyFile (none) Path to the client key file for authenticating against the Kubernetes API server when starting the driver. This file must be located on the submitting machine's disk. Specify this as a path as opposed to a URI (i.e. do not provide - a scheme). In client mode, use spark.kubernetes.authenticate.clientKeyFile instead. + a scheme). In client mode, use spark.kubernetes.authenticate.clientKeyFile instead. 2.3.0
spark.kubernetes.authenticate.submission.clientCertFilespark.kubernetes.authenticate.submission.clientCertFile (none) Path to the client cert file for authenticating against the Kubernetes API server when starting the driver. This file must be located on the submitting machine's disk. Specify this as a path as opposed to a URI (i.e. do not - provide a scheme). In client mode, use spark.kubernetes.authenticate.clientCertFile instead. + provide a scheme). In client mode, use spark.kubernetes.authenticate.clientCertFile instead. 2.3.0
spark.kubernetes.authenticate.submission.oauthTokenspark.kubernetes.authenticate.submission.oauthToken (none) OAuth token to use when authenticating against the Kubernetes API server when starting the driver. Note that unlike the other authentication options, this is expected to be the exact string value of the token to use for - the authentication. In client mode, use spark.kubernetes.authenticate.oauthToken instead. + the authentication. In client mode, use spark.kubernetes.authenticate.oauthToken instead. 2.3.0
spark.kubernetes.authenticate.submission.oauthTokenFilespark.kubernetes.authenticate.submission.oauthTokenFile (none) Path to the OAuth token file containing the token to use when authenticating against the Kubernetes API server when starting the driver. This file must be located on the submitting machine's disk. Specify this as a path as opposed to a URI (i.e. do not - provide a scheme). In client mode, use spark.kubernetes.authenticate.oauthTokenFile instead. + provide a scheme). In client mode, use spark.kubernetes.authenticate.oauthTokenFile instead. 2.3.0
spark.kubernetes.authenticate.driver.caCertFilespark.kubernetes.authenticate.driver.caCertFile (none) Path to the CA cert file for connecting to the Kubernetes API server over TLS from the driver pod when requesting executors. This file must be located on the submitting machine's disk, and will be uploaded to the driver pod. Specify this as a path as opposed to a URI (i.e. do not provide a scheme). In client mode, use - spark.kubernetes.authenticate.caCertFile instead. + spark.kubernetes.authenticate.caCertFile instead. 2.3.0
spark.kubernetes.authenticate.driver.clientKeyFilespark.kubernetes.authenticate.driver.clientKeyFile (none) Path to the client key file for authenticating against the Kubernetes API server from the driver pod when requesting executors. This file must be located on the submitting machine's disk, and will be uploaded to the driver pod as a Kubernetes secret. Specify this as a path as opposed to a URI (i.e. do not provide a scheme). - In client mode, use spark.kubernetes.authenticate.clientKeyFile instead. + In client mode, use spark.kubernetes.authenticate.clientKeyFile instead. 2.3.0
spark.kubernetes.authenticate.driver.clientCertFilespark.kubernetes.authenticate.driver.clientCertFile (none) Path to the client cert file for authenticating against the Kubernetes API server from the driver pod when requesting executors. This file must be located on the submitting machine's disk, and will be uploaded to the driver pod as a Kubernetes secret. Specify this as a path as opposed to a URI (i.e. do not provide a scheme). - In client mode, use spark.kubernetes.authenticate.clientCertFile instead. + In client mode, use spark.kubernetes.authenticate.clientCertFile instead. 2.3.0
spark.kubernetes.authenticate.driver.oauthTokenspark.kubernetes.authenticate.driver.oauthToken (none) OAuth token to use when authenticating against the Kubernetes API server from the driver pod when requesting executors. Note that unlike the other authentication options, this must be the exact string value of the token to use for the authentication. This token value is uploaded to the driver pod as a Kubernetes secret. - In client mode, use spark.kubernetes.authenticate.oauthToken instead. + In client mode, use spark.kubernetes.authenticate.oauthToken instead. 2.3.0
spark.kubernetes.authenticate.driver.oauthTokenFilespark.kubernetes.authenticate.driver.oauthTokenFile (none) Path to the OAuth token file containing the token to use when authenticating against the Kubernetes API server from the driver pod when requesting executors. Note that unlike the other authentication options, this file must contain the exact string value of the token to use for the authentication. This token value is uploaded to the driver pod as a secret. In client mode, use - spark.kubernetes.authenticate.oauthTokenFile instead. + spark.kubernetes.authenticate.oauthTokenFile instead. 2.3.0
spark.kubernetes.authenticate.driver.mounted.caCertFilespark.kubernetes.authenticate.driver.mounted.caCertFile (none) Path to the CA cert file for connecting to the Kubernetes API server over TLS from the driver pod when requesting executors. This path must be accessible from the driver pod. Specify this as a path as opposed to a URI (i.e. do not provide a scheme). In client mode, use - spark.kubernetes.authenticate.caCertFile instead. + spark.kubernetes.authenticate.caCertFile instead. 2.3.0
spark.kubernetes.authenticate.driver.mounted.clientKeyFilespark.kubernetes.authenticate.driver.mounted.clientKeyFile (none) Path to the client key file for authenticating against the Kubernetes API server from the driver pod when requesting executors. This path must be accessible from the driver pod. Specify this as a path as opposed to a URI (i.e. do not provide a scheme). In client mode, use - spark.kubernetes.authenticate.clientKeyFile instead. + spark.kubernetes.authenticate.clientKeyFile instead. 2.3.0
spark.kubernetes.authenticate.driver.mounted.clientCertFilespark.kubernetes.authenticate.driver.mounted.clientCertFile (none) Path to the client cert file for authenticating against the Kubernetes API server from the driver pod when requesting executors. This path must be accessible from the driver pod. Specify this as a path as opposed to a URI (i.e. do not provide a scheme). In client mode, use - spark.kubernetes.authenticate.clientCertFile instead. + spark.kubernetes.authenticate.clientCertFile instead. 2.3.0
spark.kubernetes.authenticate.driver.mounted.oauthTokenFilespark.kubernetes.authenticate.driver.mounted.oauthTokenFile (none) Path to the file containing the OAuth token to use when authenticating against the Kubernetes API server from the driver pod when requesting executors. This path must be accessible from the driver pod. Note that unlike the other authentication options, this file must contain the exact string value of the token to use - for the authentication. In client mode, use spark.kubernetes.authenticate.oauthTokenFile instead. + for the authentication. In client mode, use spark.kubernetes.authenticate.oauthTokenFile instead. 2.3.0
spark.kubernetes.authenticate.driver.serviceAccountNamedefaultspark.kubernetes.authenticate.driver.serviceAccountNamedefault Service account that is used when running the driver pod. The driver pod uses this service account when requesting executor pods from the API server. Note that this cannot be specified alongside a CA cert file, client key file, - client cert file, and/or OAuth token. In client mode, use spark.kubernetes.authenticate.serviceAccountName instead. + client cert file, and/or OAuth token. In client mode, use spark.kubernetes.authenticate.serviceAccountName instead. 2.3.0
spark.kubernetes.authenticate.executor.serviceAccountName(value of spark.kubernetes.authenticate.driver.serviceAccountName)spark.kubernetes.authenticate.executor.serviceAccountName(value of spark.kubernetes.authenticate.driver.serviceAccountName) Service account that is used when running the executor pod. If this parameter is not setup, the fallback logic will use the driver's service account. @@ -860,7 +853,7 @@ See the [configuration page](configuration.html) for information on Spark config 3.1.0
spark.kubernetes.authenticate.caCertFilespark.kubernetes.authenticate.caCertFile (none) In client mode, path to the CA cert file for connecting to the Kubernetes API server over TLS when @@ -869,7 +862,7 @@ See the [configuration page](configuration.html) for information on Spark config 2.4.0
spark.kubernetes.authenticate.clientKeyFilespark.kubernetes.authenticate.clientKeyFile (none) In client mode, path to the client key file for authenticating against the Kubernetes API server @@ -878,7 +871,7 @@ See the [configuration page](configuration.html) for information on Spark config 2.4.0
spark.kubernetes.authenticate.clientCertFilespark.kubernetes.authenticate.clientCertFile (none) In client mode, path to the client cert file for authenticating against the Kubernetes API server @@ -887,7 +880,7 @@ See the [configuration page](configuration.html) for information on Spark config 2.4.0
spark.kubernetes.authenticate.oauthTokenspark.kubernetes.authenticate.oauthToken (none) In client mode, the OAuth token to use when authenticating against the Kubernetes API server when @@ -897,7 +890,7 @@ See the [configuration page](configuration.html) for information on Spark config 2.4.0
spark.kubernetes.authenticate.oauthTokenFilespark.kubernetes.authenticate.oauthTokenFile (none) In client mode, path to the file containing the OAuth token to use when authenticating against the Kubernetes API @@ -906,67 +899,67 @@ See the [configuration page](configuration.html) for information on Spark config 2.4.0
spark.kubernetes.driver.label.[LabelName]spark.kubernetes.driver.label.[LabelName] (none) - Add the label specified by LabelName to the driver pod. - For example, spark.kubernetes.driver.label.something=true. + Add the label specified by LabelName to the driver pod. + For example, spark.kubernetes.driver.label.something=true. Note that Spark also adds its own labels to the driver pod for bookkeeping purposes. 2.3.0
spark.kubernetes.driver.annotation.[AnnotationName]spark.kubernetes.driver.annotation.[AnnotationName] (none) - Add the Kubernetes annotation specified by AnnotationName to the driver pod. - For example, spark.kubernetes.driver.annotation.something=true. + Add the Kubernetes annotation specified by AnnotationName to the driver pod. + For example, spark.kubernetes.driver.annotation.something=true. 2.3.0
spark.kubernetes.driver.service.label.[LabelName]spark.kubernetes.driver.service.label.[LabelName] (none) - Add the Kubernetes label specified by LabelName to the driver service. - For example, spark.kubernetes.driver.service.label.something=true. + Add the Kubernetes label specified by LabelName to the driver service. + For example, spark.kubernetes.driver.service.label.something=true. Note that Spark also adds its own labels to the driver service for bookkeeping purposes. 3.4.0
spark.kubernetes.driver.service.annotation.[AnnotationName]spark.kubernetes.driver.service.annotation.[AnnotationName] (none) - Add the Kubernetes annotation specified by AnnotationName to the driver service. - For example, spark.kubernetes.driver.service.annotation.something=true. + Add the Kubernetes annotation specified by AnnotationName to the driver service. + For example, spark.kubernetes.driver.service.annotation.something=true. 3.0.0
spark.kubernetes.executor.label.[LabelName]spark.kubernetes.executor.label.[LabelName] (none) - Add the label specified by LabelName to the executor pods. - For example, spark.kubernetes.executor.label.something=true. + Add the label specified by LabelName to the executor pods. + For example, spark.kubernetes.executor.label.something=true. Note that Spark also adds its own labels to the executor pod for bookkeeping purposes. 2.3.0
spark.kubernetes.executor.annotation.[AnnotationName]spark.kubernetes.executor.annotation.[AnnotationName] (none) - Add the Kubernetes annotation specified by AnnotationName to the executor pods. - For example, spark.kubernetes.executor.annotation.something=true. + Add the Kubernetes annotation specified by AnnotationName to the executor pods. + For example, spark.kubernetes.executor.annotation.something=true. 2.3.0
spark.kubernetes.driver.pod.namespark.kubernetes.driver.pod.name (none) Name of the driver pod. In cluster mode, if this is not set, the driver pod name is set to "spark.app.name" @@ -978,19 +971,19 @@ See the [configuration page](configuration.html) for information on Spark config 2.3.0
spark.kubernetes.executor.podNamePrefixspark.kubernetes.executor.podNamePrefix (none) Prefix to use in front of the executor pod names. It must conform the rules defined by the Kubernetes DNS Label Names. - The prefix will be used to generate executor pod names in the form of $podNamePrefix-exec-$id, where the `id` is + The prefix will be used to generate executor pod names in the form of $podNamePrefix-exec-$id, where the `id` is a positive int value, so the length of the `podNamePrefix` needs to be less than or equal to 47(= 63 - 10 - 6). 2.3.0
spark.kubernetes.submission.waitAppCompletiontruespark.kubernetes.submission.waitAppCompletiontrue In cluster mode, whether to wait for the application to finish before exiting the launcher process. When changed to false, the launcher has a "fire-and-forget" behavior when launching the Spark job. @@ -998,33 +991,33 @@ See the [configuration page](configuration.html) for information on Spark config 2.3.0
spark.kubernetes.report.interval1sspark.kubernetes.report.interval1s Interval between reports of the current Spark job status in cluster mode. 2.3.0
spark.kubernetes.executor.apiPollingInterval30sspark.kubernetes.executor.apiPollingInterval30s Interval between polls against the Kubernetes API server to inspect the state of executors. 2.4.0
spark.kubernetes.driver.request.coresspark.kubernetes.driver.request.cores (none) Specify the cpu request for the driver pod. Values conform to the Kubernetes convention. Example values include 0.1, 500m, 1.5, 5, etc., with the definition of cpu units documented in CPU units. - This takes precedence over spark.driver.cores for specifying the driver pod cpu request if set. + This takes precedence over spark.driver.cores for specifying the driver pod cpu request if set. 3.0.0
spark.kubernetes.driver.limit.coresspark.kubernetes.driver.limit.cores (none) Specify a hard cpu limit for the driver pod. @@ -1032,18 +1025,18 @@ See the [configuration page](configuration.html) for information on Spark config 2.3.0
spark.kubernetes.executor.request.coresspark.kubernetes.executor.request.cores (none) Specify the cpu request for each executor pod. Values conform to the Kubernetes convention. Example values include 0.1, 500m, 1.5, 5, etc., with the definition of cpu units documented in CPU units. - This is distinct from spark.executor.cores: it is only used and takes precedence over spark.executor.cores for specifying the executor pod cpu request if set. Task + This is distinct from spark.executor.cores: it is only used and takes precedence over spark.executor.cores for specifying the executor pod cpu request if set. Task parallelism, e.g., number of tasks an executor can run concurrently is not affected by this. 2.4.0
spark.kubernetes.executor.limit.coresspark.kubernetes.executor.limit.cores (none) Specify a hard cpu limit for each executor pod launched for the Spark Application. @@ -1051,177 +1044,177 @@ See the [configuration page](configuration.html) for information on Spark config 2.3.0
spark.kubernetes.node.selector.[labelKey]spark.kubernetes.node.selector.[labelKey] (none) - Adds to the node selector of the driver pod and executor pods, with key labelKey and the value as the - configuration's value. For example, setting spark.kubernetes.node.selector.identifier to myIdentifier - will result in the driver pod and executors having a node selector with key identifier and value - myIdentifier. Multiple node selector keys can be added by setting multiple configurations with this prefix. + Adds to the node selector of the driver pod and executor pods, with key labelKey and the value as the + configuration's value. For example, setting spark.kubernetes.node.selector.identifier to myIdentifier + will result in the driver pod and executors having a node selector with key identifier and value + myIdentifier. Multiple node selector keys can be added by setting multiple configurations with this prefix. 2.3.0
spark.kubernetes.driver.node.selector.[labelKey]spark.kubernetes.driver.node.selector.[labelKey] (none) - Adds to the driver node selector of the driver pod, with key labelKey and the value as the - configuration's value. For example, setting spark.kubernetes.driver.node.selector.identifier to myIdentifier - will result in the driver pod having a node selector with key identifier and value - myIdentifier. Multiple driver node selector keys can be added by setting multiple configurations with this prefix. + Adds to the driver node selector of the driver pod, with key labelKey and the value as the + configuration's value. For example, setting spark.kubernetes.driver.node.selector.identifier to myIdentifier + will result in the driver pod having a node selector with key identifier and value + myIdentifier. Multiple driver node selector keys can be added by setting multiple configurations with this prefix. 3.3.0
spark.kubernetes.executor.node.selector.[labelKey]spark.kubernetes.executor.node.selector.[labelKey] (none) - Adds to the executor node selector of the executor pods, with key labelKey and the value as the - configuration's value. For example, setting spark.kubernetes.executor.node.selector.identifier to myIdentifier - will result in the executors having a node selector with key identifier and value - myIdentifier. Multiple executor node selector keys can be added by setting multiple configurations with this prefix. + Adds to the executor node selector of the executor pods, with key labelKey and the value as the + configuration's value. For example, setting spark.kubernetes.executor.node.selector.identifier to myIdentifier + will result in the executors having a node selector with key identifier and value + myIdentifier. Multiple executor node selector keys can be added by setting multiple configurations with this prefix. 3.3.0
spark.kubernetes.driverEnv.[EnvironmentVariableName]spark.kubernetes.driverEnv.[EnvironmentVariableName] (none) - Add the environment variable specified by EnvironmentVariableName to + Add the environment variable specified by EnvironmentVariableName to the Driver process. The user can specify multiple of these to set multiple environment variables. 2.3.0
spark.kubernetes.driver.secrets.[SecretName]spark.kubernetes.driver.secrets.[SecretName] (none) - Add the Kubernetes Secret named SecretName to the driver pod on the path specified in the value. For example, - spark.kubernetes.driver.secrets.spark-secret=/etc/secrets. + Add the Kubernetes Secret named SecretName to the driver pod on the path specified in the value. For example, + spark.kubernetes.driver.secrets.spark-secret=/etc/secrets. 2.3.0
spark.kubernetes.executor.secrets.[SecretName]spark.kubernetes.executor.secrets.[SecretName] (none) - Add the Kubernetes Secret named SecretName to the executor pod on the path specified in the value. For example, - spark.kubernetes.executor.secrets.spark-secret=/etc/secrets. + Add the Kubernetes Secret named SecretName to the executor pod on the path specified in the value. For example, + spark.kubernetes.executor.secrets.spark-secret=/etc/secrets. 2.3.0
spark.kubernetes.driver.secretKeyRef.[EnvName]spark.kubernetes.driver.secretKeyRef.[EnvName] (none) - Add as an environment variable to the driver container with name EnvName (case sensitive), the value referenced by key in the data of the referenced Kubernetes Secret. For example, - spark.kubernetes.driver.secretKeyRef.ENV_VAR=spark-secret:key. + Add as an environment variable to the driver container with name EnvName (case sensitive), the value referenced by key key in the data of the referenced Kubernetes Secret. For example, + spark.kubernetes.driver.secretKeyRef.ENV_VAR=spark-secret:key. 2.4.0
spark.kubernetes.executor.secretKeyRef.[EnvName]spark.kubernetes.executor.secretKeyRef.[EnvName] (none) - Add as an environment variable to the executor container with name EnvName (case sensitive), the value referenced by key in the data of the referenced Kubernetes Secret. For example, - spark.kubernetes.executor.secrets.ENV_VAR=spark-secret:key. + Add as an environment variable to the executor container with name EnvName (case sensitive), the value referenced by key key in the data of the referenced Kubernetes Secret. For example, + spark.kubernetes.executor.secrets.ENV_VAR=spark-secret:key. 2.4.0
spark.kubernetes.driver.volumes.[VolumeType].[VolumeName].mount.pathspark.kubernetes.driver.volumes.[VolumeType].[VolumeName].mount.path (none) - Add the Kubernetes Volume named VolumeName of the VolumeType type to the driver pod on the path specified in the value. For example, - spark.kubernetes.driver.volumes.persistentVolumeClaim.checkpointpvc.mount.path=/checkpoint. + Add the Kubernetes Volume named VolumeName of the VolumeType type to the driver pod on the path specified in the value. For example, + spark.kubernetes.driver.volumes.persistentVolumeClaim.checkpointpvc.mount.path=/checkpoint. 2.4.0
spark.kubernetes.driver.volumes.[VolumeType].[VolumeName].mount.subPathspark.kubernetes.driver.volumes.[VolumeType].[VolumeName].mount.subPath (none) Specifies a subpath to be mounted from the volume into the driver pod. - spark.kubernetes.driver.volumes.persistentVolumeClaim.checkpointpvc.mount.subPath=checkpoint. + spark.kubernetes.driver.volumes.persistentVolumeClaim.checkpointpvc.mount.subPath=checkpoint. 3.0.0
spark.kubernetes.driver.volumes.[VolumeType].[VolumeName].mount.readOnlyspark.kubernetes.driver.volumes.[VolumeType].[VolumeName].mount.readOnly (none) Specify if the mounted volume is read only or not. For example, - spark.kubernetes.driver.volumes.persistentVolumeClaim.checkpointpvc.mount.readOnly=false. + spark.kubernetes.driver.volumes.persistentVolumeClaim.checkpointpvc.mount.readOnly=false. 2.4.0
spark.kubernetes.driver.volumes.[VolumeType].[VolumeName].options.[OptionName]spark.kubernetes.driver.volumes.[VolumeType].[VolumeName].options.[OptionName] (none) - Configure Kubernetes Volume options passed to the Kubernetes with OptionName as key having specified value, must conform with Kubernetes option format. For example, - spark.kubernetes.driver.volumes.persistentVolumeClaim.checkpointpvc.options.claimName=spark-pvc-claim. + Configure Kubernetes Volume options passed to the Kubernetes with OptionName as key having specified value, must conform with Kubernetes option format. For example, + spark.kubernetes.driver.volumes.persistentVolumeClaim.checkpointpvc.options.claimName=spark-pvc-claim. 2.4.0
spark.kubernetes.executor.volumes.[VolumeType].[VolumeName].mount.pathspark.kubernetes.executor.volumes.[VolumeType].[VolumeName].mount.path (none) - Add the Kubernetes Volume named VolumeName of the VolumeType type to the executor pod on the path specified in the value. For example, - spark.kubernetes.executor.volumes.persistentVolumeClaim.checkpointpvc.mount.path=/checkpoint. + Add the Kubernetes Volume named VolumeName of the VolumeType type to the executor pod on the path specified in the value. For example, + spark.kubernetes.executor.volumes.persistentVolumeClaim.checkpointpvc.mount.path=/checkpoint. 2.4.0
spark.kubernetes.executor.volumes.[VolumeType].[VolumeName].mount.subPathspark.kubernetes.executor.volumes.[VolumeType].[VolumeName].mount.subPath (none) Specifies a subpath to be mounted from the volume into the executor pod. - spark.kubernetes.executor.volumes.persistentVolumeClaim.checkpointpvc.mount.subPath=checkpoint. + spark.kubernetes.executor.volumes.persistentVolumeClaim.checkpointpvc.mount.subPath=checkpoint. 3.0.0
spark.kubernetes.executor.volumes.[VolumeType].[VolumeName].mount.readOnlyspark.kubernetes.executor.volumes.[VolumeType].[VolumeName].mount.readOnly false Specify if the mounted volume is read only or not. For example, - spark.kubernetes.executor.volumes.persistentVolumeClaim.checkpointpvc.mount.readOnly=false. + spark.kubernetes.executor.volumes.persistentVolumeClaim.checkpointpvc.mount.readOnly=false. 2.4.0
spark.kubernetes.executor.volumes.[VolumeType].[VolumeName].options.[OptionName]spark.kubernetes.executor.volumes.[VolumeType].[VolumeName].options.[OptionName] (none) - Configure Kubernetes Volume options passed to the Kubernetes with OptionName as key having specified value. For example, - spark.kubernetes.executor.volumes.persistentVolumeClaim.checkpointpvc.options.claimName=spark-pvc-claim. + Configure Kubernetes Volume options passed to the Kubernetes with OptionName as key having specified value. For example, + spark.kubernetes.executor.volumes.persistentVolumeClaim.checkpointpvc.options.claimName=spark-pvc-claim. 2.4.0
spark.kubernetes.local.dirs.tmpfsfalsespark.kubernetes.local.dirs.tmpfsfalse - Configure the emptyDir volumes used to back SPARK_LOCAL_DIRS within the Spark driver and executor pods to use tmpfs backing i.e. RAM. See Local Storage earlier on this page + Configure the emptyDir volumes used to back SPARK_LOCAL_DIRS within the Spark driver and executor pods to use tmpfs backing i.e. RAM. See Local Storage earlier on this page for more discussion of this. 3.0.0
spark.kubernetes.memoryOverheadFactor0.1spark.kubernetes.memoryOverheadFactor0.1 - This sets the Memory Overhead Factor that will allocate memory to non-JVM memory, which includes off-heap memory allocations, non-JVM tasks, various systems processes, and tmpfs-based local directories when spark.kubernetes.local.dirs.tmpfs is true. For JVM-based jobs this value will default to 0.10 and 0.40 for non-JVM jobs. + This sets the Memory Overhead Factor that will allocate memory to non-JVM memory, which includes off-heap memory allocations, non-JVM tasks, various systems processes, and tmpfs-based local directories when spark.kubernetes.local.dirs.tmpfs is true. For JVM-based jobs this value will default to 0.10 and 0.40 for non-JVM jobs. This is done as non-JVM tasks need more non-JVM heap space and such tasks commonly fail with "Memory Overhead Exceeded" errors. This preempts this error with a higher default. - This will be overridden by the value set by spark.driver.memoryOverheadFactor and spark.executor.memoryOverheadFactor explicitly. + This will be overridden by the value set by spark.driver.memoryOverheadFactor and spark.executor.memoryOverheadFactor explicitly. 2.4.0
spark.kubernetes.pyspark.pythonVersion"3"spark.kubernetes.pyspark.pythonVersion"3" This sets the major Python version of the docker image used to run the driver and executor containers. It can be only "3". This configuration was deprecated from Spark 3.1.0, and is effectively no-op. @@ -1231,8 +1224,8 @@ See the [configuration page](configuration.html) for information on Spark config 2.4.0
spark.kubernetes.kerberos.krb5.path(none)spark.kubernetes.kerberos.krb5.path(none) Specify the local location of the krb5.conf file to be mounted on the driver and executors for Kerberos interaction. It is important to note that the KDC defined needs to be visible from inside the containers. @@ -1240,8 +1233,8 @@ See the [configuration page](configuration.html) for information on Spark config 3.0.0
spark.kubernetes.kerberos.krb5.configMapName(none)spark.kubernetes.kerberos.krb5.configMapName(none) Specify the name of the ConfigMap, containing the krb5.conf file, to be mounted on the driver and executors for Kerberos interaction. The KDC defined needs to be visible from inside the containers. The ConfigMap must also @@ -1250,8 +1243,8 @@ See the [configuration page](configuration.html) for information on Spark config 3.0.0
spark.kubernetes.hadoop.configMapName(none)spark.kubernetes.hadoop.configMapName(none) Specify the name of the ConfigMap, containing the HADOOP_CONF_DIR files, to be mounted on the driver and executors for custom Hadoop configuration. @@ -1259,8 +1252,8 @@ See the [configuration page](configuration.html) for information on Spark config 3.0.0
spark.kubernetes.kerberos.tokenSecret.name(none)spark.kubernetes.kerberos.tokenSecret.name(none) Specify the name of the secret where your existing delegation tokens are stored. This removes the need for the job user to provide any kerberos credentials for launching a job. @@ -1268,8 +1261,8 @@ See the [configuration page](configuration.html) for information on Spark config 3.0.0
spark.kubernetes.kerberos.tokenSecret.itemKey(none)spark.kubernetes.kerberos.tokenSecret.itemKey(none) Specify the item key of the data where your existing delegation tokens are stored. This removes the need for the job user to provide any kerberos credentials for launching a job. @@ -1277,43 +1270,43 @@ See the [configuration page](configuration.html) for information on Spark config 3.0.0
spark.kubernetes.driver.podTemplateFilespark.kubernetes.driver.podTemplateFile (none) Specify the local file that contains the driver pod template. For example - spark.kubernetes.driver.podTemplateFile=/path/to/driver-pod-template.yaml + spark.kubernetes.driver.podTemplateFile=/path/to/driver-pod-template.yaml 3.0.0
spark.kubernetes.driver.podTemplateContainerNamespark.kubernetes.driver.podTemplateContainerName (none) Specify the container name to be used as a basis for the driver in the given pod template. - For example spark.kubernetes.driver.podTemplateContainerName=spark-driver + For example spark.kubernetes.driver.podTemplateContainerName=spark-driver 3.0.0
spark.kubernetes.executor.podTemplateFilespark.kubernetes.executor.podTemplateFile (none) Specify the local file that contains the executor pod template. For example - spark.kubernetes.executor.podTemplateFile=/path/to/executor-pod-template.yaml + spark.kubernetes.executor.podTemplateFile=/path/to/executor-pod-template.yaml 3.0.0
spark.kubernetes.executor.podTemplateContainerNamespark.kubernetes.executor.podTemplateContainerName (none) Specify the container name to be used as a basis for the executor in the given pod template. - For example spark.kubernetes.executor.podTemplateContainerName=spark-executor + For example spark.kubernetes.executor.podTemplateContainerName=spark-executor 3.0.0
spark.kubernetes.executor.deleteOnTerminationspark.kubernetes.executor.deleteOnTermination true Specify whether executor pods should be deleted in case of failure or normal termination. @@ -1321,55 +1314,55 @@ See the [configuration page](configuration.html) for information on Spark config 3.0.0
spark.kubernetes.executor.checkAllContainersfalsespark.kubernetes.executor.checkAllContainersfalse Specify whether executor pods should be check all containers (including sidecars) or only the executor container when determining the pod status. 3.1.0
spark.kubernetes.submission.connectionTimeout10000spark.kubernetes.submission.connectionTimeout10000 Connection timeout in milliseconds for the kubernetes client to use for starting the driver. 3.0.0
spark.kubernetes.submission.requestTimeout10000spark.kubernetes.submission.requestTimeout10000 Request timeout in milliseconds for the kubernetes client to use for starting the driver. 3.0.0
spark.kubernetes.trust.certificatesfalsespark.kubernetes.trust.certificatesfalse If set to true then client can submit to kubernetes cluster only with token. 3.2.0
spark.kubernetes.driver.connectionTimeout10000spark.kubernetes.driver.connectionTimeout10000 Connection timeout in milliseconds for the kubernetes client in driver to use when requesting executors. 3.0.0
spark.kubernetes.driver.requestTimeout10000spark.kubernetes.driver.requestTimeout10000 Request timeout in milliseconds for the kubernetes client in driver to use when requesting executors. 3.0.0
spark.kubernetes.appKillPodDeletionGracePeriodspark.kubernetes.appKillPodDeletionGracePeriod (none) Specify the grace period in seconds when deleting a Spark application using spark-submit. @@ -1377,25 +1370,25 @@ See the [configuration page](configuration.html) for information on Spark config 3.0.0
spark.kubernetes.dynamicAllocation.deleteGracePeriod5sspark.kubernetes.dynamicAllocation.deleteGracePeriod5s How long to wait for executors to shut down gracefully before a forceful kill. 3.0.0
spark.kubernetes.file.upload.pathspark.kubernetes.file.upload.path (none) Path to store files at the spark submit side in cluster mode. For example: - spark.kubernetes.file.upload.path=s3a://<s3-bucket>/path - File should specified as file://path/to/file or absolute path. + spark.kubernetes.file.upload.path=s3a://<s3-bucket>/path + File should specified as file://path/to/file or absolute path. 3.0.0
spark.kubernetes.executor.decommissionLabelspark.kubernetes.executor.decommissionLabel (none) Label to be applied to pods which are exiting or being decommissioned. Intended for use @@ -1404,16 +1397,16 @@ See the [configuration page](configuration.html) for information on Spark config 3.3.0
spark.kubernetes.executor.decommissionLabelValuespark.kubernetes.executor.decommissionLabelValue (none) Value to be applied with the label when - spark.kubernetes.executor.decommissionLabel is enabled. + spark.kubernetes.executor.decommissionLabel is enabled. 3.3.0
spark.kubernetes.executor.scheduler.namespark.kubernetes.executor.scheduler.name (none) Specify the scheduler name for each executor pod. @@ -1421,7 +1414,7 @@ See the [configuration page](configuration.html) for information on Spark config 3.0.0
spark.kubernetes.driver.scheduler.namespark.kubernetes.driver.scheduler.name (none) Specify the scheduler name for driver pod. @@ -1429,7 +1422,7 @@ See the [configuration page](configuration.html) for information on Spark config 3.3.0
spark.kubernetes.scheduler.namespark.kubernetes.scheduler.name (none) Specify the scheduler name for driver and executor pods. If `spark.kubernetes.driver.scheduler.name` or @@ -1438,8 +1431,8 @@ See the [configuration page](configuration.html) for information on Spark config 3.3.0
spark.kubernetes.configMap.maxSize1572864spark.kubernetes.configMap.maxSize1572864 Max size limit for a config map. This is configurable as per limit on k8s server end. @@ -1447,8 +1440,8 @@ See the [configuration page](configuration.html) for information on Spark config 3.1.0
spark.kubernetes.executor.missingPodDetectDelta30sspark.kubernetes.executor.missingPodDetectDelta30s When a registered executor's POD is missing from the Kubernetes API server's polled list of PODs then this delta time is taken as the accepted time difference between the @@ -1458,50 +1451,50 @@ See the [configuration page](configuration.html) for information on Spark config 3.1.1
spark.kubernetes.decommission.script/opt/decom.shspark.kubernetes.decommission.script/opt/decom.sh The location of the script to use for graceful decommissioning. 3.2.0
spark.kubernetes.driver.service.deleteOnTerminationtruespark.kubernetes.driver.service.deleteOnTerminationtrue If true, driver service will be deleted on Spark application termination. If false, it will be cleaned up when the driver pod is deletion. 3.2.0
spark.kubernetes.driver.service.ipFamilyPolicySingleStackspark.kubernetes.driver.service.ipFamilyPolicySingleStack K8s IP Family Policy for Driver Service. Valid values are - SingleStack, PreferDualStack, and RequireDualStack. + SingleStack, PreferDualStack, and RequireDualStack. 3.4.0
spark.kubernetes.driver.service.ipFamiliesIPv4spark.kubernetes.driver.service.ipFamiliesIPv4 A list of IP families for K8s Driver Service. Valid values are - IPv4 and IPv6. + IPv4 and IPv6. 3.4.0
spark.kubernetes.driver.ownPersistentVolumeClaimtruespark.kubernetes.driver.ownPersistentVolumeClaimtrue If true, driver pod becomes the owner of on-demand persistent volume claims instead of the executor pods 3.2.0
spark.kubernetes.driver.reusePersistentVolumeClaimtruespark.kubernetes.driver.reusePersistentVolumeClaimtrue If true, driver pod tries to reuse driver-owned on-demand persistent volume claims of the deleted executor pods if exists. This can be useful to reduce executor pod @@ -1510,32 +1503,32 @@ See the [configuration page](configuration.html) for information on Spark config including persistent volume claims are not reusable yet. Spark will create new persistent volume claims when there exists no reusable one. In other words, the total number of persistent volume claims can be larger than the number of running executors - sometimes. This config requires spark.kubernetes.driver.ownPersistentVolumeClaim=true. + sometimes. This config requires spark.kubernetes.driver.ownPersistentVolumeClaim=true. 3.2.0
spark.kubernetes.driver.waitToReusePersistentVolumeClaimfalsespark.kubernetes.driver.waitToReusePersistentVolumeClaimfalse If true, driver pod counts the number of created on-demand persistent volume claims and wait if the number is greater than or equal to the total number of volumes which the Spark job is able to have. This config requires both - spark.kubernetes.driver.ownPersistentVolumeClaim=true and - spark.kubernetes.driver.reusePersistentVolumeClaim=true. + spark.kubernetes.driver.ownPersistentVolumeClaim=true and + spark.kubernetes.driver.reusePersistentVolumeClaim=true. 3.4.0
spark.kubernetes.executor.disableConfigMapfalsespark.kubernetes.executor.disableConfigMapfalse If true, disable ConfigMap creation for executors. 3.2.0
spark.kubernetes.driver.pod.featureStepsspark.kubernetes.driver.pod.featureSteps (none) Class names of an extra driver pod feature step implementing @@ -1547,7 +1540,7 @@ See the [configuration page](configuration.html) for information on Spark config 3.2.0
spark.kubernetes.executor.pod.featureStepsspark.kubernetes.executor.pod.featureSteps (none) Class names of an extra executor pod feature step implementing @@ -1559,8 +1552,8 @@ See the [configuration page](configuration.html) for information on Spark config 3.2.0
spark.kubernetes.allocation.maxPendingPodsInt.MaxValuespark.kubernetes.allocation.maxPendingPodsInt.MaxValue Maximum number of pending PODs allowed during executor allocation for this application. Those newly requested executors which are unknown by Kubernetes yet are @@ -1571,19 +1564,19 @@ See the [configuration page](configuration.html) for information on Spark config 3.2.0
spark.kubernetes.allocation.pods.allocatordirectspark.kubernetes.allocation.pods.allocatordirect - Allocator to use for pods. Possible values are direct (the default) - and statefulset, or a full class name of a class implementing `AbstractPodsAllocator`. + Allocator to use for pods. Possible values are direct (the default) + and statefulset, or a full class name of a class implementing `AbstractPodsAllocator`. Future version may add Job or replicaset. This is a developer API and may change or be removed at anytime. 3.3.0
spark.kubernetes.allocation.executor.timeou600sspark.kubernetes.allocation.executor.timeout600s Time to wait before a newly created executor POD request, which does not reached the POD pending state yet, considered timedout and will be deleted. @@ -1591,8 +1584,8 @@ See the [configuration page](configuration.html) for information on Spark config 3.1.0
spark.kubernetes.allocation.driver.readinessTimeout1sspark.kubernetes.allocation.driver.readinessTimeout1s Time to wait for driver pod to get ready before creating executor pods. This wait only happens on application start. If timeout happens, executor pods will still be @@ -1601,8 +1594,8 @@ See the [configuration page](configuration.html) for information on Spark config 3.1.3
spark.kubernetes.executor.enablePollingWithResourceVersionfalsespark.kubernetes.executor.enablePollingWithResourceVersionfalse If true, `resourceVersion` is set with `0` during invoking pod listing APIs in order to allow API Server-side caching. This should be used carefully. @@ -1610,24 +1603,24 @@ See the [configuration page](configuration.html) for information on Spark config 3.3.0
spark.kubernetes.executor.eventProcessingInterval1sspark.kubernetes.executor.eventProcessingInterval1s Interval between successive inspection of executor events sent from the Kubernetes API. 2.4.0
spark.kubernetes.executor.rollInterval0sspark.kubernetes.executor.rollInterval0s Interval between executor roll operations. It's disabled by default with `0s`. 3.3.0
spark.kubernetes.executor.minTasksPerExecutorBeforeRolling0spark.kubernetes.executor.minTasksPerExecutorBeforeRolling0 The minimum number of tasks per executor before rolling. Spark will not roll executors whose total number of tasks is smaller @@ -1636,8 +1629,8 @@ See the [configuration page](configuration.html) for information on Spark config 3.3.0
spark.kubernetes.executor.rollPolicyOUTLIERspark.kubernetes.executor.rollPolicyOUTLIER Executor roll policy: Valid values are ID, ADD_TIME, TOTAL_GC_TIME, TOTAL_DURATION, FAILED_TASKS, and OUTLIER (default). @@ -1665,25 +1658,19 @@ See the below table for the full list of pod specifications that will be overwri ### Pod Metadata - - - - - - - - +
Pod metadata keyModified valueDescription
+ - + - + - + - + @@ -1707,54 +1694,48 @@ See the below table for the full list of pod specifications that will be overwri ### Pod Spec -
Pod metadata keyModified valueDescription
nameValue of spark.kubernetes.driver.pod.nameValue of spark.kubernetes.driver.pod.name The driver pod name will be overwritten with either the configured or default value of - spark.kubernetes.driver.pod.name. The executor pod names will be unaffected. + spark.kubernetes.driver.pod.name. The executor pod names will be unaffected.
namespaceValue of spark.kubernetes.namespaceValue of spark.kubernetes.namespace Spark makes strong assumptions about the driver and executor namespaces. Both driver and executor namespaces will be replaced by either the configured or default spark conf value. @@ -1691,14 +1678,14 @@ See the below table for the full list of pod specifications that will be overwri
labelsAdds the labels from spark.kubernetes.{driver,executor}.label.*Adds the labels from spark.kubernetes.{driver,executor}.label.* Spark will add additional labels specified by the spark configuration.
annotationsAdds the annotations from spark.kubernetes.{driver,executor}.annotation.*Adds the annotations from spark.kubernetes.{driver,executor}.annotation.* Spark will add additional annotations specified by the spark configuration.
- - - - - - - +
Pod spec keyModified valueDescription
+ - + - + - + - + - + - +
Pod spec keyModified valueDescription
imagePullSecretsAdds image pull secrets from spark.kubernetes.container.image.pullSecretsAdds image pull secrets from spark.kubernetes.container.image.pullSecrets Additional pull secrets will be added from the spark configuration to both executor pods.
nodeSelectorAdds node selectors from spark.kubernetes.node.selector.*Adds node selectors from spark.kubernetes.node.selector.* Additional node selectors will be added from the spark configuration to both executor pods.
restartPolicy"never""never" Spark assumes that both drivers and executors never restart.
serviceAccountValue of spark.kubernetes.authenticate.driver.serviceAccountNameValue of spark.kubernetes.authenticate.driver.serviceAccountName - Spark will override serviceAccount with the value of the spark configuration for only + Spark will override serviceAccount with the value of the spark configuration for only driver pods, and only if the spark configuration is specified. Executor pods will remain unaffected.
serviceAccountNameValue of spark.kubernetes.authenticate.driver.serviceAccountNameValue of spark.kubernetes.authenticate.driver.serviceAccountName - Spark will override serviceAccountName with the value of the spark configuration for only + Spark will override serviceAccountName with the value of the spark configuration for only driver pods, and only if the spark configuration is specified. Executor pods will remain unaffected.
volumesAdds volumes from spark.kubernetes.{driver,executor}.volumes.[VolumeType].[VolumeName].mount.pathAdds volumes from spark.kubernetes.{driver,executor}.volumes.[VolumeType].[VolumeName].mount.path Spark will add volumes as specified by the spark conf, as well as additional volumes necessary for passing spark conf and pod template files. @@ -1766,32 +1747,26 @@ See the below table for the full list of pod specifications that will be overwri The following affect the driver and executor containers. All other containers in the pod spec will be unaffected. - - - - - - - - +
Container spec keyModified valueDescription
+ - + - + - + @@ -1809,15 +1784,15 @@ The following affect the driver and executor containers. All other containers in - +
Container spec keyModified valueDescription
envAdds env variables from spark.kubernetes.driverEnv.[EnvironmentVariableName]Adds env variables from spark.kubernetes.driverEnv.[EnvironmentVariableName] - Spark will add driver env variables from spark.kubernetes.driverEnv.[EnvironmentVariableName], and - executor env variables from spark.executorEnv.[EnvironmentVariableName]. + Spark will add driver env variables from spark.kubernetes.driverEnv.[EnvironmentVariableName], and + executor env variables from spark.executorEnv.[EnvironmentVariableName].
imageValue of spark.kubernetes.{driver,executor}.container.imageValue of spark.kubernetes.{driver,executor}.container.image The image will be defined by the spark configurations.
imagePullPolicyValue of spark.kubernetes.container.image.pullPolicyValue of spark.kubernetes.container.image.pullPolicy Spark will override the pull policy for both driver and executors. resources See description - The cpu limits are set by spark.kubernetes.{driver,executor}.limit.cores. The cpu is set by - spark.{driver,executor}.cores. The memory request and limit are set by summing the values of - spark.{driver,executor}.memory and spark.{driver,executor}.memoryOverhead. - Other resource limits are set by spark.{driver,executor}.resources.{resourceName}.* configs. + The cpu limits are set by spark.kubernetes.{driver,executor}.limit.cores. The cpu is set by + spark.{driver,executor}.cores. The memory request and limit are set by summing the values of + spark.{driver,executor}.memory and spark.{driver,executor}.memoryOverhead. + Other resource limits are set by spark.{driver,executor}.resources.{resourceName}.* configs.
volumeMountsAdd volumes from spark.kubernetes.driver.volumes.[VolumeType].[VolumeName].mount.{path,readOnly}Add volumes from spark.kubernetes.driver.volumes.[VolumeType].[VolumeName].mount.{path,readOnly} Spark will add volumes as specified by the spark conf, as well as additional volumes necessary for passing spark conf and pod template files. diff --git a/docs/running-on-yarn.md b/docs/running-on-yarn.md index de39a6129b2c5..3dfa63e1cb2e3 100644 --- a/docs/running-on-yarn.md +++ b/docs/running-on-yarn.md @@ -143,124 +143,118 @@ To use a custom metrics.properties for the application master and executors, upd #### Spark Properties - - - - - - - - - +
Property NameDefaultMeaningSince Version
+ - - + + - - + + - - + + - - + + - - + + - - + + - - + + - - + + - - + + - - + + - + - - + + - - + + - - + + - + - + - + - + - - + + - - + + - + - - + + - + - + - + - - + + - + - + - + - - + + - + - + - + - + - - + + - + - + - + - + - + - + - + - + - + - + - + - - + + - - + + - - + + - - + + - - + + - +
Property NameDefaultMeaningSince Version
spark.yarn.am.memory512mspark.yarn.am.memory512m - Amount of memory to use for the YARN Application Master in client mode, in the same format as JVM memory strings (e.g. 512m, 2g). - In cluster mode, use spark.driver.memory instead.
- Use lower-case suffixes, e.g. k, m, g, t, and p, for kibi-, mebi-, gibi-, tebi-, and pebibytes, respectively. + Amount of memory to use for the YARN Application Master in client mode, in the same format as JVM memory strings (e.g. 512m, 2g). + In cluster mode, use spark.driver.memory instead. +

+ Use lower-case suffixes, e.g. k, m, g, t, and p, for kibi-, mebi-, gibi-, tebi-, and pebibytes, respectively.

1.3.0
spark.yarn.am.resource.{resource-type}.amount(none)spark.yarn.am.resource.{resource-type}.amount(none) Amount of resource to use for the YARN Application Master in client mode. - In cluster mode, use spark.yarn.driver.resource.<resource-type>.amount instead. + In cluster mode, use spark.yarn.driver.resource.<resource-type>.amount instead. Please note that this feature can be used only with YARN 3.0+ For reference, see YARN Resource Model documentation -
+

Example: - To request GPU resources from YARN, use: spark.yarn.am.resource.yarn.io/gpu.amount + To request GPU resources from YARN, use: spark.yarn.am.resource.yarn.io/gpu.amount

3.0.0
spark.yarn.applicationTypeSPARKspark.yarn.applicationTypeSPARK - Defines more specific application types, e.g. SPARK, SPARK-SQL, SPARK-STREAMING, - SPARK-MLLIB and SPARK-GRAPH. Please be careful not to exceed 20 characters. + Defines more specific application types, e.g. SPARK, SPARK-SQL, SPARK-STREAMING, + SPARK-MLLIB and SPARK-GRAPH. Please be careful not to exceed 20 characters. 3.1.0
spark.yarn.driver.resource.{resource-type}.amount(none)spark.yarn.driver.resource.{resource-type}.amount(none) Amount of resource to use for the YARN Application Master in cluster mode. Please note that this feature can be used only with YARN 3.0+ For reference, see YARN Resource Model documentation

Example: - To request GPU resources from YARN, use: spark.yarn.driver.resource.yarn.io/gpu.amount + To request GPU resources from YARN, use: spark.yarn.driver.resource.yarn.io/gpu.amount

3.0.0
spark.yarn.executor.resource.{resource-type}.amount(none)spark.yarn.executor.resource.{resource-type}.amount(none) Amount of resource to use per executor process. Please note that this feature can be used only with YARN 3.0+ For reference, see YARN Resource Model documentation -
+

Example: - To request GPU resources from YARN, use: spark.yarn.executor.resource.yarn.io/gpu.amount + To request GPU resources from YARN, use: spark.yarn.executor.resource.yarn.io/gpu.amount

3.0.0
spark.yarn.resourceGpuDeviceNameyarn.io/gpuspark.yarn.resourceGpuDeviceNameyarn.io/gpu - Specify the mapping of the Spark resource type of gpu to the YARN resource - representing a GPU. By default YARN uses yarn.io/gpu but if YARN has been + Specify the mapping of the Spark resource type of gpu to the YARN resource + representing a GPU. By default YARN uses yarn.io/gpu but if YARN has been configured with a custom resource type, this allows remapping it. - Applies when using the spark.{driver/executor}.resource.gpu.* configs. + Applies when using the spark.{driver/executor}.resource.gpu.* configs. 3.2.1
spark.yarn.resourceFpgaDeviceNameyarn.io/fpgaspark.yarn.resourceFpgaDeviceNameyarn.io/fpga - Specify the mapping of the Spark resource type of fpga to the YARN resource - representing a FPGA. By default YARN uses yarn.io/fpga but if YARN has been + Specify the mapping of the Spark resource type of fpga to the YARN resource + representing a FPGA. By default YARN uses yarn.io/fpga but if YARN has been configured with a custom resource type, this allows remapping it. - Applies when using the spark.{driver/executor}.resource.fpga.* configs. + Applies when using the spark.{driver/executor}.resource.fpga.* configs. 3.2.1
spark.yarn.am.cores1spark.yarn.am.cores1 Number of cores to use for the YARN Application Master in client mode. - In cluster mode, use spark.driver.cores instead. + In cluster mode, use spark.driver.cores instead. 1.3.0
spark.yarn.am.waitTime100sspark.yarn.am.waitTime100s - Only used in cluster mode. Time for the YARN Application Master to wait for the + Only used in cluster mode. Time for the YARN Application Master to wait for the SparkContext to be initialized. 1.3.0
spark.yarn.submit.file.replicationThe default HDFS replication (usually 3)spark.yarn.submit.file.replicationThe default HDFS replication (usually 3) HDFS replication level for the files uploaded into HDFS for the application. These include things like the Spark jar, the app jar, and any distributed cache files/archives. 0.8.1
spark.yarn.stagingDirspark.yarn.stagingDir Current user's home directory in the filesystem Staging directory used while submitting applications. @@ -268,46 +262,46 @@ To use a custom metrics.properties for the application master and executors, upd 2.0.0
spark.yarn.preserve.staging.filesfalsespark.yarn.preserve.staging.filesfalse - Set to true to preserve the staged files (Spark jar, app jar, distributed cache files) at the end of the job rather than delete them. + Set to true to preserve the staged files (Spark jar, app jar, distributed cache files) at the end of the job rather than delete them. 1.1.0
spark.yarn.scheduler.heartbeat.interval-ms3000spark.yarn.scheduler.heartbeat.interval-ms3000 The interval in ms in which the Spark application master heartbeats into the YARN ResourceManager. The value is capped at half the value of YARN's configuration for the expiry interval, i.e. - yarn.am.liveness-monitor.expiry-interval-ms. + yarn.am.liveness-monitor.expiry-interval-ms. 0.8.1
spark.yarn.scheduler.initial-allocation.interval200msspark.yarn.scheduler.initial-allocation.interval200ms The initial interval in which the Spark application master eagerly heartbeats to the YARN ResourceManager when there are pending container allocation requests. It should be no larger than - spark.yarn.scheduler.heartbeat.interval-ms. The allocation interval will doubled on + spark.yarn.scheduler.heartbeat.interval-ms. The allocation interval will doubled on successive eager heartbeats if pending containers still exist, until - spark.yarn.scheduler.heartbeat.interval-ms is reached. + spark.yarn.scheduler.heartbeat.interval-ms is reached. 1.4.0
spark.yarn.historyServer.addressspark.yarn.historyServer.address (none) - The address of the Spark history server, e.g. host.com:18080. The address should not contain a scheme (http://). Defaults to not being set since the history server is an optional service. This address is given to the YARN ResourceManager when the Spark application finishes to link the application from the ResourceManager UI to the Spark history server UI. - For this property, YARN properties can be used as variables, and these are substituted by Spark at runtime. For example, if the Spark history server runs on the same node as the YARN ResourceManager, it can be set to ${hadoopconf-yarn.resourcemanager.hostname}:18080. + The address of the Spark history server, e.g. host.com:18080. The address should not contain a scheme (http://). Defaults to not being set since the history server is an optional service. This address is given to the YARN ResourceManager when the Spark application finishes to link the application from the ResourceManager UI to the Spark history server UI. + For this property, YARN properties can be used as variables, and these are substituted by Spark at runtime. For example, if the Spark history server runs on the same node as the YARN ResourceManager, it can be set to ${hadoopconf-yarn.resourcemanager.hostname}:18080. 1.0.0
spark.yarn.dist.archivesspark.yarn.dist.archives (none) Comma separated list of archives to be extracted into the working directory of each executor. @@ -315,7 +309,7 @@ To use a custom metrics.properties for the application master and executors, upd 1.0.0
spark.yarn.dist.filesspark.yarn.dist.files (none) Comma-separated list of files to be placed in the working directory of each executor. @@ -323,7 +317,7 @@ To use a custom metrics.properties for the application master and executors, upd 1.0.0
spark.yarn.dist.jarsspark.yarn.dist.jars (none) Comma-separated list of jars to be placed in the working directory of each executor. @@ -331,8 +325,8 @@ To use a custom metrics.properties for the application master and executors, upd 2.0.0
spark.yarn.dist.forceDownloadSchemes(none)spark.yarn.dist.forceDownloadSchemes(none) Comma-separated list of schemes for which resources will be downloaded to the local disk prior to being added to YARN's distributed cache. For use in cases where the YARN service does not @@ -342,47 +336,47 @@ To use a custom metrics.properties for the application master and executors, upd 2.3.0
spark.executor.instances2spark.executor.instances2 - The number of executors for static allocation. With spark.dynamicAllocation.enabled, the initial set of executors will be at least this large. + The number of executors for static allocation. With spark.dynamicAllocation.enabled, the initial set of executors will be at least this large. 1.0.0
spark.yarn.am.memoryOverheadspark.yarn.am.memoryOverhead AM memory * 0.10, with minimum of 384 - Same as spark.driver.memoryOverhead, but for the YARN Application Master in client mode. + Same as spark.driver.memoryOverhead, but for the YARN Application Master in client mode. 1.3.0
spark.yarn.queuedefaultspark.yarn.queuedefault The name of the YARN queue to which the application is submitted. 1.0.0
spark.yarn.jarsspark.yarn.jars (none) List of libraries containing Spark code to distribute to YARN containers. By default, Spark on YARN will use Spark jars installed locally, but the Spark jars can also be in a world-readable location on HDFS. This allows YARN to cache it on nodes so that it doesn't need to be distributed each time an application runs. To point to jars on HDFS, for example, - set this configuration to hdfs:///some/path. Globs are allowed. + set this configuration to hdfs:///some/path. Globs are allowed. 2.0.0
spark.yarn.archivespark.yarn.archive (none) An archive containing needed Spark jars for distribution to the YARN cache. If set, this - configuration replaces spark.yarn.jars and the archive is used in all the + configuration replaces spark.yarn.jars and the archive is used in all the application's containers. The archive should contain jar files in its root directory. Like with the previous option, the archive can also be hosted on HDFS to speed up file distribution. @@ -390,38 +384,38 @@ To use a custom metrics.properties for the application master and executors, upd 2.0.0
spark.yarn.appMasterEnv.[EnvironmentVariableName]spark.yarn.appMasterEnv.[EnvironmentVariableName] (none) - Add the environment variable specified by EnvironmentVariableName to the + Add the environment variable specified by EnvironmentVariableName to the Application Master process launched on YARN. The user can specify multiple of - these and to set multiple environment variables. In cluster mode this controls - the environment of the Spark driver and in client mode it only controls + these and to set multiple environment variables. In cluster mode this controls + the environment of the Spark driver and in client mode it only controls the environment of the executor launcher. 1.1.0
spark.yarn.containerLauncherMaxThreads25spark.yarn.containerLauncherMaxThreads25 The maximum number of threads to use in the YARN Application Master for launching executor containers. 1.2.0
spark.yarn.am.extraJavaOptionsspark.yarn.am.extraJavaOptions (none) A string of extra JVM options to pass to the YARN Application Master in client mode. - In cluster mode, use spark.driver.extraJavaOptions instead. Note that it is illegal + In cluster mode, use spark.driver.extraJavaOptions instead. Note that it is illegal to set maximum heap size (-Xmx) settings with this option. Maximum heap size settings can be set - with spark.yarn.am.memory + with spark.yarn.am.memory 1.3.0
spark.yarn.am.extraLibraryPathspark.yarn.am.extraLibraryPath (none) Set a special library path to use when launching the YARN Application Master in client mode. @@ -429,22 +423,22 @@ To use a custom metrics.properties for the application master and executors, upd 1.4.0
spark.yarn.populateHadoopClasspathspark.yarn.populateHadoopClasspath - For with-hadoop Spark distribution, this is set to false; - for no-hadoop distribution, this is set to true. + For with-hadoop Spark distribution, this is set to false; + for no-hadoop distribution, this is set to true. - Whether to populate Hadoop classpath from yarn.application.classpath and - mapreduce.application.classpath Note that if this is set to false, - it requires a with-Hadoop Spark distribution that bundles Hadoop runtime or + Whether to populate Hadoop classpath from yarn.application.classpath and + mapreduce.application.classpath Note that if this is set to false, + it requires a with-Hadoop Spark distribution that bundles Hadoop runtime or user has to provide a Hadoop installation separately. 2.4.6
spark.yarn.maxAppAttemptsyarn.resourcemanager.am.max-attempts in YARNspark.yarn.maxAppAttemptsyarn.resourcemanager.am.max-attempts in YARN The maximum number of attempts that will be made to submit the application. It should be no larger than the global number of max attempts in the YARN configuration. @@ -452,7 +446,7 @@ To use a custom metrics.properties for the application master and executors, upd 1.3.0
spark.yarn.am.attemptFailuresValidityIntervalspark.yarn.am.attemptFailuresValidityInterval (none) Defines the validity interval for AM failure tracking. @@ -462,7 +456,7 @@ To use a custom metrics.properties for the application master and executors, upd 1.6.0
spark.yarn.am.clientModeTreatDisconnectAsFailedspark.yarn.am.clientModeTreatDisconnectAsFailed false Treat yarn-client unclean disconnects as failures. In yarn-client mode, normally the application will always finish @@ -475,7 +469,7 @@ To use a custom metrics.properties for the application master and executors, upd 3.3.0
spark.yarn.am.clientModeExitOnErrorspark.yarn.am.clientModeExitOnError false In yarn-client mode, when this is true, if driver got application report with final status of KILLED or FAILED, @@ -485,30 +479,30 @@ To use a custom metrics.properties for the application master and executors, upd 3.3.0
spark.yarn.am.tokenConfRegexspark.yarn.am.tokenConfRegex (none) The value of this config is a regex expression used to grep a list of config entries from the job's configuration file (e.g., hdfs-site.xml) and send to RM, which uses them when renewing delegation tokens. A typical use case of this feature is to support delegation tokens in an environment where a YARN cluster needs to talk to multiple downstream HDFS clusters, where the YARN RM may not have configs (e.g., dfs.nameservices, dfs.ha.namenodes.*, dfs.namenode.rpc-address.*) to connect to these clusters. - In this scenario, Spark users can specify the config value to be ^dfs.nameservices$|^dfs.namenode.rpc-address.*$|^dfs.ha.namenodes.*$ to parse - these HDFS configs from the job's local configuration files. This config is very similar to mapreduce.job.send-token-conf. Please check YARN-5910 for more details. + In this scenario, Spark users can specify the config value to be ^dfs.nameservices$|^dfs.namenode.rpc-address.*$|^dfs.ha.namenodes.*$ to parse + these HDFS configs from the job's local configuration files. This config is very similar to mapreduce.job.send-token-conf. Please check YARN-5910 for more details. 3.3.0
spark.yarn.submit.waitAppCompletiontruespark.yarn.submit.waitAppCompletiontrue In YARN cluster mode, controls whether the client waits to exit until the application completes. - If set to true, the client process will stay alive reporting the application's status. + If set to true, the client process will stay alive reporting the application's status. Otherwise, the client process will exit after submission. 1.4.0
spark.yarn.am.nodeLabelExpressionspark.yarn.am.nodeLabelExpression (none) A YARN node label expression that restricts the set of nodes AM will be scheduled on. @@ -518,7 +512,7 @@ To use a custom metrics.properties for the application master and executors, upd 1.6.0
spark.yarn.executor.nodeLabelExpressionspark.yarn.executor.nodeLabelExpression (none) A YARN node label expression that restricts the set of nodes executors will be scheduled on. @@ -528,7 +522,7 @@ To use a custom metrics.properties for the application master and executors, upd 1.4.0
spark.yarn.tagsspark.yarn.tags (none) Comma-separated list of strings to pass through as YARN application tags appearing @@ -537,7 +531,7 @@ To use a custom metrics.properties for the application master and executors, upd 1.5.0
spark.yarn.priorityspark.yarn.priority (none) Application priority for YARN to define pending applications ordering policy, those with higher @@ -547,41 +541,41 @@ To use a custom metrics.properties for the application master and executors, upd 3.0.0
spark.yarn.config.gatewayPathspark.yarn.config.gatewayPath (none) A path that is valid on the gateway host (the host where a Spark application is started) but may differ for paths for the same resource in other nodes in the cluster. Coupled with - spark.yarn.config.replacementPath, this is used to support clusters with + spark.yarn.config.replacementPath, this is used to support clusters with heterogeneous configurations, so that Spark can correctly launch remote processes. -
+

The replacement path normally will contain a reference to some environment variable exported by YARN (and, thus, visible to Spark containers). -
- For example, if the gateway node has Hadoop libraries installed on /disk1/hadoop, and - the location of the Hadoop install is exported by YARN as the HADOOP_HOME - environment variable, setting this value to /disk1/hadoop and the replacement path to - $HADOOP_HOME will make sure that paths used to launch remote processes properly +

+ For example, if the gateway node has Hadoop libraries installed on /disk1/hadoop, and + the location of the Hadoop install is exported by YARN as the HADOOP_HOME + environment variable, setting this value to /disk1/hadoop and the replacement path to + $HADOOP_HOME will make sure that paths used to launch remote processes properly reference the local YARN configuration.

1.5.0
spark.yarn.config.replacementPathspark.yarn.config.replacementPath (none) - See spark.yarn.config.gatewayPath. + See spark.yarn.config.gatewayPath. 1.5.0
spark.yarn.rolledLog.includePatternspark.yarn.rolledLog.includePattern (none) Java Regex to filter the log files which match the defined include pattern and those log files will be aggregated in a rolling fashion. This will be used with YARN's rolling log aggregation, to enable this feature in YARN side - yarn.nodemanager.log-aggregation.roll-monitoring-interval-seconds should be + yarn.nodemanager.log-aggregation.roll-monitoring-interval-seconds should be configured in yarn-site.xml. The Spark log4j appender needs be changed to use FileAppender or another appender that can handle the files being removed while it is running. Based on the file name configured in the log4j configuration (like spark.log), the user should set the @@ -590,7 +584,7 @@ To use a custom metrics.properties for the application master and executors, upd 2.0.0
spark.yarn.rolledLog.excludePatternspark.yarn.rolledLog.excludePattern (none) Java Regex to filter the log files which match the defined exclude pattern @@ -600,17 +594,17 @@ To use a custom metrics.properties for the application master and executors, upd 2.0.0
spark.yarn.executor.launch.excludeOnFailure.enabledspark.yarn.executor.launch.excludeOnFailure.enabled false Flag to enable exclusion of nodes having YARN resource allocation problems. The error limit for excluding can be configured by - spark.excludeOnFailure.application.maxFailedExecutorsPerNode. + spark.excludeOnFailure.application.maxFailedExecutorsPerNode. 2.4.0
spark.yarn.exclude.nodesspark.yarn.exclude.nodes (none) Comma-separated list of YARN node names which are excluded from resource allocation. @@ -618,7 +612,7 @@ To use a custom metrics.properties for the application master and executors, upd 3.0.0
spark.yarn.metrics.namespacespark.yarn.metrics.namespace (none) The root namespace for AM metrics reporting. @@ -627,16 +621,16 @@ To use a custom metrics.properties for the application master and executors, upd 2.4.0
spark.yarn.report.interval1sspark.yarn.report.interval1s Interval between reports of the current Spark job status in cluster mode. 0.9.0
spark.yarn.report.loggingFrequency30spark.yarn.report.loggingFrequency30 Maximum number of application reports processed until the next application status is logged. If there is a change of state, the application status will be logged regardless @@ -645,16 +639,16 @@ To use a custom metrics.properties for the application master and executors, upd 3.5.0
spark.yarn.clientLaunchMonitorInterval1sspark.yarn.clientLaunchMonitorInterval1s Interval between requests for status the client mode AM when starting the app. 2.3.0
spark.yarn.includeDriverLogsLinkfalsespark.yarn.includeDriverLogsLinkfalse In cluster mode, whether the client application report includes links to the driver container's logs. This requires polling the ResourceManager's REST API, so it @@ -663,8 +657,8 @@ To use a custom metrics.properties for the application master and executors, upd 3.1.0
spark.yarn.unmanagedAM.enabledfalsespark.yarn.unmanagedAM.enabledfalse In client mode, whether to launch the Application Master service as part of the client using unmanaged am. @@ -672,7 +666,7 @@ To use a custom metrics.properties for the application master and executors, upd 3.0.0
spark.yarn.shuffle.server.recovery.disabledspark.yarn.shuffle.server.recovery.disabled false Set to true for applications that have higher security requirements and prefer that their @@ -727,7 +721,7 @@ To use a custom metrics.properties for the application master and executors, upd For example, suppose you would like to point log url link to Job History Server directly instead of let NodeManager http server redirects it, you can configure `spark.history.custom.executor.log.url` as below: -{{HTTP_SCHEME}}<JHS_HOST>:<JHS_PORT>/jobhistory/logs/{{NM_HOST}}:{{NM_PORT}}/{{CONTAINER_ID}}/{{CONTAINER_ID}}/{{USER}}/{{FILE_NAME}}?start=-4096 +{{HTTP_SCHEME}}<JHS_HOST>:<JHS_PORT>/jobhistory/logs/{{NM_HOST}}:{{NM_PORT}}/{{CONTAINER_ID}}/{{CONTAINER_ID}}/{{USER}}/{{FILE_NAME}}?start=-4096 NOTE: you need to replace `` and `` with actual value. @@ -772,39 +766,34 @@ staging directory of the Spark application. ## YARN-specific Kerberos Configuration - - - - - - - +
Property NameDefaultMeaningSince Version
+ - + - + - + - + @@ -876,48 +865,38 @@ to avoid garbage collection issues during shuffle. The following extra configuration options are available when the shuffle service is running on YARN: -
Property NameDefaultMeaningSince Version
spark.kerberos.keytabspark.kerberos.keytab (none) The full path to the file that contains the keytab for the principal specified above. This keytab will be copied to the node running the YARN Application Master via the YARN Distributed Cache, and will be used for renewing the login tickets and the delegation tokens periodically. Equivalent to - the --keytab command line argument. + the --keytab command line argument.
(Works also with the "local" master.) 		3.0.0
spark.kerberos.principalspark.kerberos.principal (none) Principal to be used to login to KDC, while running on secure clusters. Equivalent to the - --principal command line argument. + --principal command line argument.
(Works also with the "local" master.) 		3.0.0
spark.yarn.kerberos.relogin.periodspark.yarn.kerberos.relogin.period 1m How often to check whether the kerberos TGT should be renewed. This should be set to a value @@ -814,12 +803,12 @@ staging directory of the Spark application. 2.3.0
spark.yarn.kerberos.renewal.excludeHadoopFileSystemsspark.yarn.kerberos.renewal.excludeHadoopFileSystems (none) A comma-separated list of Hadoop filesystems for whose hosts will be excluded from delegation - token renewal at resource scheduler. For example, spark.yarn.kerberos.renewal.excludeHadoopFileSystems=hdfs://nn1.com:8032, - hdfs://nn2.com:8032. This is known to work under YARN for now, so YARN Resource Manager won't renew tokens for the application. + token renewal at resource scheduler. For example, spark.yarn.kerberos.renewal.excludeHadoopFileSystems=hdfs://nn1.com:8032, + hdfs://nn2.com:8032. This is known to work under YARN for now, so YARN Resource Manager won't renew tokens for the application. Note that as resource scheduler does not renew token, so any application running longer than the original token expiration that tries to use that token will likely fail.
- - - - - - - - +
Property NameDefaultMeaningSince Version
+ - - + + - - - + + - - - + + - - +
Property NameDefaultMeaning
spark.yarn.shuffle.stopOnFailurefalsespark.yarn.shuffle.stopOnFailurefalse Whether to stop the NodeManager when there's a failure in the Spark Shuffle Service's initialization. This prevents application failures caused by running containers on NodeManagers where the Spark Shuffle Service is not running.
spark.yarn.shuffle.service.metrics.namespacesparkShuffleServicespark.yarn.shuffle.service.metrics.namespacesparkShuffleService The namespace to use when emitting shuffle service metrics into Hadoop metrics2 system of the NodeManager.
spark.yarn.shuffle.service.logs.namespace(not set)spark.yarn.shuffle.service.logs.namespace(not set) A namespace which will be appended to the class name when forming the logger name to use for emitting logs from the YARN shuffle service, like - org.apache.spark.network.yarn.YarnShuffleService.logsNamespaceValue. Since some logging frameworks + org.apache.spark.network.yarn.YarnShuffleService.logsNamespaceValue. Since some logging frameworks may expect the logger name to look like a class name, it's generally recommended to provide a value which would be a valid Java package or class name and not include spaces.
spark.shuffle.service.db.backendspark.shuffle.service.db.backend ROCKSDB When work-preserving restart is enabled in YARN, this is used to specify the disk-base store used From 39a112d16b76f2935121ac32ae79020ff997d486 Mon Sep 17 00:00:00 2001 From: panbingkun Date: Wed, 3 Jan 2024 16:56:53 +0800 Subject: [PATCH 5/7] [SPARK-46546][DOCS] Fix the formatting of tables in running-on-yarn pages --- docs/running-on-yarn.md | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-) diff --git a/docs/running-on-yarn.md b/docs/running-on-yarn.md index 3dfa63e1cb2e3..9553ca4423ad4 100644 --- a/docs/running-on-yarn.md +++ b/docs/running-on-yarn.md @@ -866,7 +866,7 @@ to avoid garbage collection issues during shuffle. The following extra configuration options are available when the shuffle service is running on YARN: - + @@ -875,6 +875,7 @@ The following extra configuration options are available when the shuffle service initialization. This prevents application failures caused by running containers on NodeManagers where the Spark Shuffle Service is not running. + @@ -883,6 +884,7 @@ The following extra configuration options are available when the shuffle service The namespace to use when emitting shuffle service metrics into Hadoop metrics2 system of the NodeManager. + @@ -894,6 +896,7 @@ The following extra configuration options are available when the shuffle service may expect the logger name to look like a class name, it's generally recommended to provide a value which would be a valid Java package or class name and not include spaces. + From ca304b619d8b10c67368f8090645dde38f05aa3d Mon Sep 17 00:00:00 2001 From: panbingkun Date: Wed, 3 Jan 2024 17:25:09 +0800 Subject: [PATCH 6/7] update --- docs/running-on-yarn.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/running-on-yarn.md b/docs/running-on-yarn.md index 9553ca4423ad4..4c3152baaccd6 100644 --- a/docs/running-on-yarn.md +++ b/docs/running-on-yarn.md @@ -875,7 +875,7 @@ The following extra configuration options are available when the shuffle service initialization. This prevents application failures caused by running containers on NodeManagers where the Spark Shuffle Service is not running. - + From f87a26078d2c0f273e13392e749d9afe9924a678 Mon Sep 17 00:00:00 2001 From: panbingkun Date: Wed, 3 Jan 2024 19:00:55 +0800 Subject: [PATCH 7/7] [SPARK-46546][DOCS] Fix the formatting of tables in running-on-yarn pages --- docs/running-on-yarn.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/running-on-yarn.md b/docs/running-on-yarn.md index 4c3152baaccd6..02547b30d2e50 100644 --- a/docs/running-on-yarn.md +++ b/docs/running-on-yarn.md @@ -884,7 +884,7 @@ The following extra configuration options are available when the shuffle service The namespace to use when emitting shuffle service metrics into Hadoop metrics2 system of the NodeManager. - + @@ -896,7 +896,7 @@ The following extra configuration options are available when the shuffle service may expect the logger name to look like a class name, it's generally recommended to provide a value which would be a valid Java package or class name and not include spaces. - +
Property NameDefaultMeaning
Property NameDefaultMeaningSince Version
spark.yarn.shuffle.stopOnFailure false
spark.yarn.shuffle.service.metrics.namespace
spark.yarn.shuffle.service.logs.namespace
spark.shuffle.service.db.backend2.1.0
spark.yarn.shuffle.service.metrics.namespace3.2.0
spark.yarn.shuffle.service.logs.namespace3.3.0
spark.shuffle.service.db.backend