diff --git a/modules/concepts/nav.adoc b/modules/concepts/nav.adoc index 9b38e9ccb..d828bc1ff 100644 --- a/modules/concepts/nav.adoc +++ b/modules/concepts/nav.adoc @@ -1,7 +1,7 @@ * xref:concepts:index.adoc[] -** xref:overview.adoc[Overview] +** xref:overview.adoc[Platform overview] +** xref:stacklet.adoc[] ** Common configuration mechanisms -*** xref:roles-and-role-groups.adoc[] *** xref:product_image_selection.adoc[] *** xref:overrides.adoc[Advanced: overrides] ** Resources diff --git a/modules/concepts/pages/authentication.adoc b/modules/concepts/pages/authentication.adoc index 08ff36c68..864d12034 100644 --- a/modules/concepts/pages/authentication.adoc +++ b/modules/concepts/pages/authentication.adoc @@ -1,7 +1,8 @@ = Authentication +:keycloak: https://www.keycloak.org/ The Stackable Platform uses the AuthenticationClass as a central mechanism to handle user authentication across supported products. -The authentication mechanism needs to be configured only in the AuthenticationClass which is then referenced in the product. +The authentication mechanism needs to be configured only in the AuthenticationClass which is then referenced in the xref:stacklet.adoc[Stacklet] definition. Multiple different authentication providers are supported. [#authenticationclass] @@ -40,7 +41,7 @@ NOTE: Learn more in the xref:tutorials:authentication_with_openldap.adoc[OpenLDA [#OIDC] === OpenID Connect -An OIDC provider like https://www.keycloak.org/[Keycloak {external-link-icon}^] could be configured as follows: +An OIDC provider like {keycloak}[Keycloak] could be configured as follows: [source,yaml] ---- @@ -60,10 +61,10 @@ NOTE: Get a full overview of all the properties in the {crd-docs}/authentication === TLS The `TLS` provider configures a product to authenticate users using TLS certificates. When establishing a connection the client will first validate the certificate of the server. -This step is not influenced by this `AuthenticationClass`, it only affects the next step: +This step is not influenced by this AuthenticationClass, it only affects the next step: Afterwards the server checks the validity of the certificate the client has provided. This includes the usual checks - such as checking that it hasn't expired and matches the hostname of the client. -Additionally the client certificate needs to be signed with the `ca` certificate, which is provided by the `SecretClass` specified in `clientCertSecretClass`. +Additionally the client certificate needs to be signed with the `ca` certificate, which is provided by the SecretClass specified in `clientCertSecretClass`. A sample TLS provider looks as follows: @@ -77,25 +78,25 @@ include::example$authenticationclass-tls.yaml[] The `static` provider is used to represent a simple - static - set of users. Users are identified by a username and a password. -First, the `AuthenticationClass` needs to be defined as follows: +First, the AuthenticationClass needs to be defined as follows: [source,yaml] ---- include::example$authenticationclass-static-authenticationclass.yaml[] ---- -<1> The name of the `Secret` containing the credentials +<1> The name of the Secret containing the credentials -Afterwards the referenced `Secret` needs to be created: +Afterwards the referenced Secret needs to be created: [source,yaml] ---- include::example$authenticationclass-static-secret.yaml[] ---- -<1> The name of the `Secret`, which needs to match the `Secret` name specified in the `AuthenticationClass` above -<2> The namespace of the `Secret`. The `Secret` needs to be in the same namespace as the product that tries to use the static `AuthenticationClass` +<1> The name of the Secret, which needs to match the Secret name specified in the AuthenticationClass above +<2> The namespace of the Secret. The Secret needs to be in the same namespace as the product that tries to use the static AuthenticationClass [#further-reading] -== Further Reading +== Further reading * xref:tutorials:authentication_with_openldap.adoc[] tutorial * {crd-docs}/authentication.stackable.tech/authenticationclass/v1alpha1/[AuthenticationClass CRD reference] diff --git a/modules/concepts/pages/index.adoc b/modules/concepts/pages/index.adoc index 269017d85..aec67a86f 100644 --- a/modules/concepts/pages/index.adoc +++ b/modules/concepts/pages/index.adoc @@ -6,7 +6,7 @@ The xref:overview.adoc[Platform overview] is a good starting point to understand == General configuration mechanisms -Learn about xref:roles-and-role-groups.adoc[], how product image selection works. +Learn what a xref:stacklet.adoc[Stacklet] is, what roles and role groups are, and how product image selection works. There is also the common xref:overrides.adoc[override] mechanism for configuration settings, although this tool should be used with care! == Resources diff --git a/modules/concepts/pages/labels.adoc b/modules/concepts/pages/labels.adoc index ceb4f931f..0b8391382 100644 --- a/modules/concepts/pages/labels.adoc +++ b/modules/concepts/pages/labels.adoc @@ -1,25 +1,26 @@ = Labels +:common-labels: https://kubernetes.io/docs/concepts/overview/working-with-objects/common-labels/ Labels are key/value pairs in the metadata of Kubernetes objects that add identifying information to the object. They do not have direct semantic implications but can be used to organize and manage resources. -The `stackablectl` tool, the cockpit and the Helm Charts add labels to the resources that are part of a Stacklet, and the operators add labels to the resources they create. +The xref:management:stackablectl:index.adoc[`stackablectl`] tool, the cockpit, and the Helm Charts add labels to the resources that are part of a xref:stacklet.adoc[Stacklet], and the operators add labels to the resources they create. == Resource labels added by the operators -Every resource created by a Stackable operator is labelled with a common set of labels. -Some of these labels are https://kubernetes.io/docs/concepts/overview/working-with-objects/common-labels/[recommended] to use by the Kubernetes documentation. +Every resource created by a Stackable operator has a common set of labels applied. +Some of these labels are {common-labels}[recommended] to use by the Kubernetes documentation. The following labels are added to resources created by our operators: -- `app.kubernetes.io/name` - The name of the product, i.e. `druid` or `zookeeper`. -- `app.kubernetes.io/version` - The version of the product and Stackable version, i.e. `3.3.4-stackable23.11.0` -- `app.kubernetes.io/instance` - The name of the Stacklet. -- `app.kubernetes.io/component` - This is the xref:concepts:roles-and-role-groups.adoc[role] that this resource belongs to. -- `app.kubernetes.io/role-group` - The name of the xref:concepts:roles-and-role-groups.adoc[role group] that a resource belongs to. -- `app.kubernetes.io/managed-by` - Which software manages this resource? This will be the operator, i.e. `airflow-operator`. +* `app.kubernetes.io/name` - The name of the product, i.e. `druid` or `zookeeper`. +* `app.kubernetes.io/version` - The version of the product and Stackable version, i.e. `3.3.4-stackable24.3.0` +* `app.kubernetes.io/instance` - The name of the Stacklet. +* `app.kubernetes.io/component` - This is the xref:stacklet.adoc#roles[role] that this resource belongs to. +* `app.kubernetes.io/role-group` - The name of the xref:stacklet.adoc#role-groups[role group] that a resource belongs to. +* `app.kubernetes.io/managed-by` - Which software manages this resource? This will be the operator, i.e. `kafka-operator`. also this: -- `stackable.tech/vendor` with value `Stackable` +- `stackable.tech/vendor` with value `Stackable`. == Labels added by tools @@ -55,4 +56,4 @@ The default set of labels includes: == Further reading -Have a look at https://docs.stackable.tech/home/nightly/contributor/adr/adr031-resource-labels[] if you want to find out about the design decisions for our labels. +Take a look at xref:contributor:adr/ADR031-resource-labels.adoc[] if you want to find out about the design decisions for our labels. diff --git a/modules/concepts/pages/logging.adoc b/modules/concepts/pages/logging.adoc index 4520b3afd..376cfbb41 100644 --- a/modules/concepts/pages/logging.adoc +++ b/modules/concepts/pages/logging.adoc @@ -1,48 +1,54 @@ = Logging :description: A conceptual explanation of the logging architecture of the Stackable Data Platform, and how it is configured. :keywords: logging, observability, log aggregation, Kubernetes, k8s, Vector, Elasticsearch, OpenSearch +:vector: https://vector.dev/ +:vector-sinks: https://vector.dev/docs/reference/configuration/sinks/# +:vector-sidecar: https://vector.dev/docs/setup/deployment/roles/#sidecar +:vector-aggregator: https://vector.dev/docs/setup/deployment/roles/#aggregator +:vector-agg-install: https://vector.dev/docs/setup/installation/package-managers/helm/#aggregator +:vector-source-vector: https://vector.dev/docs/reference/configuration/sources/vector/ +:vector-topology-centralized: https://vector.dev/docs/setup/deployment/topologies/#centralized -// Abstract Logging is important for observability of the platform. Stackable provides human-readable plaintext logs for each running container, as well as aggregated and persisted logs with identical structure across the whole platform. -Log levels can be set for individual modules and configuration is identical across all products, but custom logging configuration files can be supplied as well. +Log levels can be set for individual modules and configuration is identical in each xref:stacklet.adoc[Stacklet], but custom logging configuration files can be supplied as well. image::logging_overview.drawio.svg[] == Motivation -**Aggregate and persist** - The logging on the platform was designed to aggregate logs from all parts of the platform to make it easy to correlate events from different parts. -For this, logs should share the same structure, and should be viewable in a central location. -Logs should also be persisted in a central location, so if a component crashes, the logs are still there to identify the reason. +**Aggregate and persist** - The logging architecture is designed to aggregate logs from all parts of the platform to simplify correlating events from different Stacklets. +For this, logs from different sources are configured to share the same structure, and are collected and made viewable in a central location. +The collected logs are also be persisted centrally, so if a component crashes its logs are still there to investigate the cause of the crash. -**Easy to read on the fly** - At the same time, logs should still be accessible in an easy to read format on the containers, to allow for easy on the fly inspection of each part of the platform. +**Easy to read on the fly** - At the same time logs are kept accessible in an plain format on the containers, supporting easy on the fly inspection of running Stacklets. The logging configuration also supports setting different thresholds for the logs readable on the container and the aggregated logs. This way you can get a detailed view of the operations of a component while viewing it on the container, but aggregate logs at a coarser granularity when aggregating across the whole platform. **Consistent configuration** - Finally, logging should be always configured the same way, no matter which product and which underlying technology is used to produce the logs. -Logging for each product is configured in the ProductCluster resource. -It is still supported to supply custom logging configuration files, these are then product specific. +Logging for each product is configured in the Stacklet resource. +For advanced log configurations, supplying custom product specific log configuration files is also supported. == Architecture Below you can see the overall architecture using ZooKeeper as an example. -Stackable uses https://vector.dev/[Vector] for log aggregation and any of the supported https://vector.dev/docs/reference/configuration/sinks/[sinks] can be used to persist the logs. +Stackable uses {vector}[Vector] for log aggregation and any of the supported {vector-sinks}[sinks] can be used to persist the logs. -image::logging_architecture.drawio.svg[] +image::logging_architecture.drawio.svg[An architecture diagram showing the Kubernetes resources involved in the logging stack] === Log source -You configure your logging settings in the resource describing the cluster (ZookeeperCluster in this example), seen in the top left in the diagram (see the <> section below). +You configure your logging settings in the Stacklet definition (ZookeeperCluster in this example), seen in the top left in the diagram (see the <> section below). The operator reads this resource and creates the appropriate log configuration files in the ConfigMap which also holds other product configuration. The ConfigMap is then mounted into the containers. -The ZooKeeper Pod has three containers: The `prepare` sidecar container, the `zookeeper` container and the `vector` https://vector.dev/docs/setup/deployment/roles/#sidecar[sidecar container]. -All logs get written into a shared mounted directory, from which the Vector agent reads them and sends them to the Vector https://vector.dev/docs/setup/deployment/roles/#aggregator[aggregator]. +The ZooKeeper Pod has three containers: The `prepare` sidecar container, the `zookeeper` container and the `vector` {vector-sidecar}[sidecar container]. +All logs get written into a shared mounted directory, from which the Vector agent reads them and sends them to the Vector {vector-aggregator}[aggregator]. === Aggregator and sinks -The aggregator is configured to use one or multiple https://vector.dev/docs/reference/configuration/sinks/[sinks] (for example OpenSearch, Elasticsearch), it sends all logs to all sinks. -If a sink is unavailable, it will buffer the log messages. -It is also the single point where the sinks are configured, so the sinks are decoupled from the individual product configurations and only needs to be configured in this single location for the whole platform. +The aggregator is configured to use one or multiple {vector-sinks}[sinks] (for example OpenSearch, Elasticsearch), it sends all logs to all sinks. +If a sink is unavailable, the aggregator buffers the log messages. +It is also the single point where the sinks are configured, so the sinks are decoupled from the Stacklet definitions and only need to be configured in this single location for the whole platform. [#configuration] == Configuration @@ -109,12 +115,12 @@ A typical use case is that the log level for the console is set to a more detail **Per container configuration** - A Pod actually consists of multiple containers and you might want different log levels for each of them. Also the log levels per module are container specific. -Following the Stackable xref::roles-and-role-groups.adoc[roles and role groups] concept, this configuration fragment can be set either at role or role group level. +Following the Stackable xref:stacklet.adoc#roles[roles] and xref::stacklet.adoc#roles[role groups] concept, this configuration fragment can be set either at role or role group level. === Configuring the Aggregator -Follow the https://vector.dev/docs/setup/installation/package-managers/helm/#aggregator[installation instructions] for the aggregator. -Configure a https://vector.dev/docs/reference/configuration/sources/vector/[Vector source] at adress `0.0.0.0:6000` and configure sinks and additional settings according to your needs. +Follow the {vector-agg-install}[installation instructions] for the aggregator. +Configure a {vector-source-vector}[Vector source] at adress `0.0.0.0:6000` and configure sinks and additional settings according to your needs. === Configuring the aggregator location @@ -123,21 +129,22 @@ The field is called `ADDRESS` and the value could be `vector-aggregator:6000` if == Custom overrides -As with many parts of the Stackable platform, custom overrides are supported as well, by supplying your own logging configuration file. This is then product specific. +As with many parts of the Stackable platform, custom overrides are supported as well, by supplying your own logging configuration file. +This is then product specific. -```yaml +[source,yaml] +---- logging: enableVectorAgent: false // <1> containers: my-container: custom: configMap: my-configmap // <2> -``` - -<1> The vector logging agent is not deployed -<2> A custom logging configuration is loaded from a ConfigMap called `my-configmap` +---- +<1> The vector logging agent is not deployed. +<2> A custom logging configuration is loaded from a ConfigMap called `my-configmap`. == Further reading To get some hands on experience and see logging in action, try out the xref:demos:logging.adoc[logging demo] or follow the xref:tutorials:logging-vector-aggregator.adoc[logging tutorial]. -The Vector documentation contains more information about the https://vector.dev/docs/setup/deployment/topologies/#centralized[deployment topology] and https://vector.dev/docs/reference/configuration/sinks/[sinks]. +The Vector documentation contains more information about the {vector-topology-centralized}[deployment topology] and {vector-sinks}[sinks] that can be used. diff --git a/modules/concepts/pages/opa.adoc b/modules/concepts/pages/opa.adoc index 1abc43875..a8e363b4b 100644 --- a/modules/concepts/pages/opa.adoc +++ b/modules/concepts/pages/opa.adoc @@ -1,28 +1,42 @@ = OPA authorization +:opa: https://www.openpolicyagent.org +:rego: https://www.openpolicyagent.org/docs/latest/policy-language/ +:opa-docs: https://www.openpolicyagent.org/docs/latest/#overview -The Stackable Platform offers policy-based access control via the https://www.openpolicyagent.org[OpenPolicyAgent] (OPA) operator. -// -Authorization policies are defined in the Rego language, divided into packages and supplied via ConfigMaps. -// +The Stackable Data Platform offers policy-based access control via the {opa}[OpenPolicyAgent] (OPA) operator. +Authorization policies are defined in the {rego}[Rego] language, divided into packages and supplied via ConfigMaps. Every node is running an OPA instance for fast policy evaluation and products are connected to OPA with the xref:service_discovery.adoc[service discovery] mechanism. == What is OPA? // What's OPA? What are Rego Rules? -OPA is an open source, general purpose policy engine. It supports a high-level declarative language called https://www.openpolicyagent.org/docs/latest/policy-language/[Rego]. Rego enables you to specify complex policies as code and transfer the decision-making processes from your software to OPA. Policies written in Rego are called _Rego rules_. +OPA is an open-source, general-purpose policy engine. +It supports a high-level declarative language called {rego}[Rego]. +Rego enables you to specify complex policies as code and transfer the decision-making processes from your software to OPA. +Policies written in Rego are called _Rego rules_. // policy requests -Policy requests are made to a REST API, which allows easy requests from microservices, Kubernetes or CI/CD pipelines. In the request the requester can supply arbitrary structured input data as JSON to supply context information to the policy decision rules. For example the name of the user, resource and action for which an authorization is requested. In this way policy decision-making and policy enforcement are decoupled. +Policy requests are made to a REST API, which allows easy requests from microservices, Kubernetes or CI/CD pipelines. +In the request the requester can supply arbitrary structured input data as JSON to supply context information to the policy decision rules. +For example the name of the user, resource and action for which an authorization is requested. +In this way policy decision-making and policy enforcement are decoupled. == How it works // How it is deployed -OPA is run by the xref:opa:index.adoc[Stackable OPA operator]. OPA is deployed with the OpaCluster resource, from which the operator creates a DaemonSet to run an OPA instance on every node of the cluster. Because of this, every Pod making policy requests will always make the request locally, minimizing latency and network traffic. +OPA is run by the xref:opa:index.adoc[Stackable OPA operator]. +OPA is deployed with the OpaCluster resource, from which the operator creates a DaemonSet to run an OPA instance on every node of the cluster. +Because of this, every Pod making policy requests will always make the request locally, minimizing latency and network traffic. === Define policies -OPA by itself does not enforce a specific structure of Rego rules and the packages they are in, but the Stackable platform makes some assumptions. Whenever a product is connected to OPA (see <<_connect_a_product>>), a single package needs to be given, and that package needs to contain specific rules depending on the product. For example for Druid, a single `allow` rule needs to be defined. +OPA by itself does not enforce a specific structure of Rego rules and the packages they are in, but the Stackable platform makes some assumptions. +Whenever a product is connected to OPA (see <<_connect_a_product>>), a single package needs to be given, and that package needs to contain specific rules depending on the product. +For example for Druid, a single `allow` rule needs to be defined. // Rego rules in config maps -The Rego rule policies are supplied as ConfigMaps. Multiple ConfigMaps can be used for multiple packages, for example one package for Druid authorization, one package for a Trino development instance and one package for a Trino production instance. ConfigMaps were chosen as an easy to use method of supplying configuration files in a Kubernetes environment. The operator takes care of assembling the ConfigMaps into a policy bundle, every policy ConfigMap must contain a `opa.stackable.tech/bundle` label in order to be processed by the OPA operator. +The Rego rule policies are supplied as ConfigMaps. +Multiple ConfigMaps can be used for multiple packages, for example one package for Druid authorization, one package for a Trino development instance and one package for a Trino production instance. +ConfigMaps were chosen as an easy to use method of supplying configuration files in a Kubernetes environment. +The operator takes care of assembling the ConfigMaps into a policy bundle, every policy ConfigMap must contain a `opa.stackable.tech/bundle` label in order to be processed by the OPA operator. Here's an example of a Rego rule package in a ConfigMap: @@ -57,11 +71,12 @@ data: The combination of arbitrary input data and the Rego rules enables you to specify and enforce almost any kind of policies. You can define powerful policies for e.g. user access for database tables, schemas, columns etc. You can enforce local network traffic, access time periods and many more. -See the https://www.openpolicyagent.org/docs/latest/#overview[OPA documentation] for further examples. +See the {opa-docs}[OPA documentation] for further examples. === Connect a product -To connect a product to an OpaCluster, the name of the OpaCluster is needed as well as the name of the policy package to use with this product instance. An operator that supports OPA for its operated product will have a spec section like this: +To connect a product to an OpaCluster, the name of the OpaCluster is needed as well as the name of the policy package to use with this product instance. +A Stacklet that supports OPA for its operated product will have a `spec` section like this: [source, yaml] ---- @@ -72,14 +87,17 @@ spec: package: my-product // <2> ... ---- -<1> The reference to the OPA cluster -<2> The name of the policy package to use for this product +<1> The reference to the OPA cluster. +<2> The name of the policy package to use for this product. -The automatic connection is facilitated by the xref:service_discovery.adoc[service discovery mechanism] and no further information about OPA is required. The products query their respective rules via the supplied package name. See <> for links to specific product authorization documentation. +The automatic connection is facilitated by the xref:service_discovery.adoc[service discovery mechanism] and no further information about OPA is required. +The products query their respective rules via the supplied package name. +See <> for links to specific product authorization documentation. == Further reading -Read more about the xref:opa:index.adoc[]. Read more about product integration with OPA for these products: +Read more about the xref:opa:index.adoc[]. +Read more about product integration with OPA for these products: * xref:trino:usage-guide/security.adoc#_authorization[Trino] * xref:kafka:usage.adoc[Kafka] diff --git a/modules/concepts/pages/operations/cluster_operations.adoc b/modules/concepts/pages/operations/cluster_operations.adoc index 43d8ad904..2ddf0a6be 100644 --- a/modules/concepts/pages/operations/cluster_operations.adoc +++ b/modules/concepts/pages/operations/cluster_operations.adoc @@ -1,10 +1,11 @@ = Cluster operations :page-aliases: ../cluster_operations.adoc -Stackable operators offer different cluster operations to control the reconciliation process. This is useful when updating operators, debugging or testing of new settings: +Stackable operators offer different cluster operations to control the reconciliation process. +This is useful when updating operators, debugging or testing of new settings: * `reconciliationPaused` - Stop the operator from reconciling the cluster spec. The status will still be updated. -* `stopped` - Stop all running pods but keep updating all deployed resources like `ConfigMaps`, `Services` and the cluster status. +* `stopped` - Stop all running Pods but keep updating all deployed resources like ConfigMaps, Services and the cluster status. If not specified, `clusterOperation.reconciliationPaused` and `clusterOperation.stopped` default to `false`. @@ -15,26 +16,31 @@ If not specified, `clusterOperation.reconciliationPaused` and `clusterOperation. include::example$cluster-operations.yaml[] ---- <1> The `clusterOperation.reconciliationPaused` flag set to `true` stops the operator from reconciling any changes to the cluster spec. The cluster status is still updated. -<2> The `clusterOperation.stopped` flag set to `true` stops all pods in the cluster. This is done by setting all deployed `StatefulSet` replicas to 0. +<2> The `clusterOperation.stopped` flag set to `true` stops all pods in the cluster. This is done by setting all deployed StatefulSet replicas to 0. -IMPORTANT: When setting `clusterOperation.reconciliationPaused` and `clusterOperation.stopped` to true in the same step, `clusterOperation.reconciliationPaused` will take precedence. This means the cluster will stop reconciling immediately and the `stopped` field is ignored. To avoid this, the cluster should first be stopped and then paused. +IMPORTANT: When setting `clusterOperation.reconciliationPaused` and `clusterOperation.stopped` to true in the same step, `clusterOperation.reconciliationPaused` will take precedence. +This means the cluster will stop reconciling immediately and the `stopped` field is ignored. +To avoid this, the cluster should first be stopped and then paused. -== Service Restarts +== Service restarts -=== Manual Restarts +=== Manual restarts -Sometimes it is necessary to restart services deployed in Kubernetes. A service restart should induce as little disruption as possible, ideally none. +Sometimes it is necessary to restart services deployed in Kubernetes. +A service restart should induce as little disruption as possible, ideally none. -Most operators create StatefulSet objects for the products they manage and Kubernetes offers a rollout mechanism to restart them. You can use `kubectl rollout restart statefulset` to restart a StatefulSet previously created by an operator. +Most operators create StatefulSet objects for the products they manage and Kubernetes offers a rollout mechanism to restart them. +You can use `kubectl rollout restart statefulset` to restart a StatefulSet previously created by an operator. -To illustrate how to use the command line to restart one or more Pods, we will assume you used the Stackable HDFS Operator to deploy an HDFS stacklet called `dumbo`. +To illustrate how to use the command line to restart one or more Pods, we will assume you used the Stackable HDFS Operator to deploy an HDFS xref:stacklet.adoc[Stacklet] called `dumbo`. -This stacklet will consist, among other things, of three StatefulSets created for each HDFS role: `namenode`, `datanode` and `journalnode`. Let's list them: +This Stacklet will consist, among other things, of three StatefulSets created for each HDFS role: `namenode`, `datanode` and `journalnode`. +Let's list them: [source,shell] ---- -❯ kubectl get statefulset -l app.kubernetes.io/instance=dumbo +$ kubectl get statefulset -l app.kubernetes.io/instance=dumbo NAME READY AGE dumbo-datanode-default 2/2 4m41s dumbo-journalnode-default 1/1 4m41s @@ -45,7 +51,7 @@ To restart the HDFS DataNode Pods, run: [source,shell] ---- -❯ kubectl rollout restart statefulset dumbo-datanode-default +$ kubectl rollout restart statefulset dumbo-datanode-default statefulset.apps/dumbo-datanode-default restarted ---- @@ -53,18 +59,21 @@ Sometimes you want to restart all Pods of a stacklet and not just individual rol [source,shell] ---- -❯ kubectl rollout restart statefulset --selector app.kubernetes.io/instance=dumbo +$ kubectl rollout restart statefulset --selector app.kubernetes.io/instance=dumbo ---- To wait for all Pods to be running again: [source,shell] ---- -❯ kubectl rollout status statefulset --selector app.kubernetes.io/instance=dumbo +$ kubectl rollout status statefulset --selector app.kubernetes.io/instance=dumbo ---- -Here we used the label `app.kubernetes.io/instance=dumbo` to select all Pods that belong to a specific HDFS stacklet. This label is created by the operator and `dumbo` is the name of the HDFS stacklet as specified in the custom resource. You can add more labels to make finer grained restarts. +Here we used the label `app.kubernetes.io/instance=dumbo` to select all Pods that belong to a specific HDFS Stacklet. +This label is created by the operator and `dumbo` is the name of the HDFS Stacklet as specified in the custom resource. +You can add more labels to make finer grained restarts. -== Automatic Restarts +== Automatic restarts -The Commons Operator of the Stackable Platform may restart Pods automatically, for purposes such as ensuring that TLS certificates are up-to-date. For details, see the xref:commons-operator:index.adoc[Commons Operator documentation]. +The Commons Operator of the Stackable Platform may restart Pods automatically, for purposes such as ensuring that TLS certificates are up-to-date. +For details, see the xref:commons-operator:index.adoc[Commons Operator documentation]. diff --git a/modules/concepts/pages/operations/pod_disruptions.adoc b/modules/concepts/pages/operations/pod_disruptions.adoc index d604c55a3..b56f5fc1e 100644 --- a/modules/concepts/pages/operations/pod_disruptions.adoc +++ b/modules/concepts/pages/operations/pod_disruptions.adoc @@ -1,13 +1,14 @@ = Allowed Pod disruptions +:k8s-pdb: https://kubernetes.io/docs/tasks/run-application/configure-pdb/ Any downtime of our products is generally considered to be bad. Although downtime can't be prevented 100% of the time - especially if the product does not support High Availability - we can try to do our best to reduce it to an absolute minimum. -Kubernetes has mechanisms to ensure minimal *planned* downtime. -Please keep in mind, that this only affects planned (voluntary) downtime of Pods - unplanned Kubernetes node crashes can always occur. +Kubernetes provides mechanisms to ensure minimal *planned* downtime. +Please keep in mind that this only affects planned (voluntary) downtime of Pods - unplanned Kubernetes node crashes can always occur. -Our product operator will always deploy so-called https://kubernetes.io/docs/tasks/run-application/configure-pdb/[PodDisruptionBudget (PDB)] resources alongside the products. -For every role that you specify (e.g. HDFS namenodes or Trino workers) a PDB is created. +Our operators will always deploy so-called {k8s-pdb}[PodDisruptionBudget (PDB)] resources as part of a xref:stacklet.adoc[]. +For every xref:stacklet.adoc#roles[role] that you specify (e.g. HDFS namenodes or Trino workers) a PDB is created. == Default values The defaults depend on the individual product and can be found below the "Operations" usage guide. @@ -27,13 +28,14 @@ Increasing the number of allowed disruptions and increasing the number of replic === Multiple replicas to increase performance -For these roles (e.g. HDFS datanodes, HBase regionservers or Trino workers), more than a single Pod is allowed to be unavailable. Otherwise, rolling re-deployments may take very long. +For these roles (e.g. HDFS datanodes, HBase regionservers or Trino workers), more than a single Pod is allowed to be unavailable. +Otherwise, rolling re-deployments may take very long. -IMPORTANT: The operators calculate the number of Pods for a given role by adding the number of replicas of every rolegroup that is part of that role. +IMPORTANT: The operators calculate the number of Pods for a given role by adding the number of replicas of every role group that is part of that role. -In case there are no replicas defined on a rolegroup, one Pod will be assumed for this rolegroup, as the created Kubernetes objects (StatefulSets or Deployments) will default to a single replica as well. +In case there are no replicas defined on a role group, one Pod will be assumed for this role group, as the created Kubernetes objects (StatefulSets or Deployments) will default to a single replica as well. However, in case there are https://kubernetes.io/docs/tasks/run-application/horizontal-pod-autoscale/[HorizontalPodAutoscaler] in place, the number of replicas of a rolegroup can change dynamically. -In this case the operators might falsely assume that rolegroups have fewer Pods than they actually have. +In this case the operators might falsely assume that role groups have fewer Pods than they actually have. This is a pessimistic approach, as the number of allowed disruptions normally stays the same or even increases when the number of Pods increases. This should be safe, but in some cases more Pods *could* have been allowed to be unavailable which may increase the duration of rolling re-deployments. diff --git a/modules/concepts/pages/operations/pod_placement.adoc b/modules/concepts/pages/operations/pod_placement.adoc index f5aa7c4f6..ea5e90b01 100644 --- a/modules/concepts/pages/operations/pod_placement.adoc +++ b/modules/concepts/pages/operations/pod_placement.adoc @@ -1,11 +1,15 @@ = Pod placement :page-aliases: ../pod_placement.adoc -Several operators of the Stackable Data Platform permit the configuration of pod affinity as described in the Kubernetes https://kubernetes.io/docs/concepts/scheduling-eviction/assign-pod-node/[documentation]. If no affinity is defined in the product's custom resource, the operators apply reasonable defaults that make use of the `preferred_during_scheduling_ignored_during_execution` property. Refer to the operator documentation for details. +Several operators of the Stackable Data Platform permit the configuration of pod affinity as described in the Kubernetes https://kubernetes.io/docs/concepts/scheduling-eviction/assign-pod-node/[documentation]. +If no affinity is defined in the product's custom resource, the operators apply reasonable defaults that make use of the `preferred_during_scheduling_ignored_during_execution` property. +Refer to the operator documentation for details. -Affinities can be defined at role level, group level or both. When defined at both levels, the two affinities are merged with the group affinity taking precedence. The resulting definition is then added to the corresponding pod template. +Affinities can be defined at role level, role group level or both. +When defined at both levels, the two affinities are merged with the role group affinity taking precedence. +The resulting definition is then added to the corresponding Pod template. -Affinities can be configured by adding an `affinity` property to the product's custom resource, as shown below: +Affinities can be configured by adding an `affinity` property to the Stacklet as shown below: [source,yaml] ---- @@ -47,12 +51,9 @@ spec: values: - enabled ---- -Here the pod placement for the broker Pods of an Apache Kafka cluster is configured as follows: +Here the Pod placement for the broker Pods of an Apache Kafka cluster is configured as follows: <1> The pod assignment for all Pods of the `brokers` role. <2> No constraint is defined for clustering Pods. <3> A `preferred` constraint is defined for spreading `brokers` across cluster nodes. The intent here is to increase availability in case of node failures. <4> A `required` constraint for scheduling `brokers` on nodes with the fictional `high-throughput-network` label. Failing to satisfy this constraint will prevent the Kafka cluster from starting up. - - - diff --git a/modules/concepts/pages/overrides.adoc b/modules/concepts/pages/overrides.adoc index 4336137d6..5e12cdeda 100644 --- a/modules/concepts/pages/overrides.adoc +++ b/modules/concepts/pages/overrides.adoc @@ -1,19 +1,21 @@ = Overrides +:k8s-openapi-deepmerge: https://arnavion.github.io/k8s-openapi/v0.22.x/k8s_openapi/trait.DeepMerge.html -The Stackable Operators configure the products they are operating with sensible defaults and required settings to enable connectivity and security. -Other important settings are usually exposed as settings in the resource definition. -In some cases however, you might want to set certain settings that are not exposed, or override settings that the operator has set on the product. +The Stackable operators configure the xref:stacklet.adoc[Stacklets] they are operating with sensible defaults and required settings to enable connectivity and security. +Other important settings are usually exposed in the Stacklet resource definition for you to configure directly. +In some cases however, you might want to configure certain settings that are not exposed or are set by the operator. -The resource definitions of all products support overrides, specifically for the product configuration, environment variables, and the PodSpec the operators generate. +The resource definitions of all Stacklets support overrides, specifically for the <>, <>, and the <> the operators generate. -WARNING: Overriding certain configuration properties can lead to faulty clusters. Overrides should only be used as a last resort! +WARNING: Overriding certain configuration properties can lead to faulty clusters. +Overrides should only be used as a last resort! -The cluster definitions also supports overriding configuration aspects, either per xref:roles-and-role-groups.adoc[role or per role group], where the more specific override (role group) has precedence over the less specific one (role). +The Stacklet definitions support overriding configuration aspects either per xref:stacklet.adoc#roles[role] or xref:stacklet.adoc#role-groups[role group], where the more specific override (role group) has precedence over the less specific one (role). [#config-overrides] == Config overrides -For a xref:roles-and-role-groups.adoc[role or role group], at the same level of `config`, you can specify `configOverrides` for any of the configuration files the product uses. +For a role or role group, at the same level of `config`, you can specify `configOverrides` for any of the configuration files the product uses. An example for an HDFS cluster looks as follows: @@ -24,23 +26,31 @@ kind: HdfsCluster metadata: name: simple-hdfs spec: - nameNodes: # change to your your role - config: # [..] - configOverrides: # on role level - core-site.xml: # change to your desired configuration file - fs.trash.interval: "5" + nameNodes: + config: + ... + configOverrides: <1> + core-site.xml: <2> + fs.trash.interval: "5" <3> roleGroups: default: - config: # [..] - configOverrides: # on rolegroup level, takes precedence over the ones specified at role level + config: + ... + configOverrides: <4> hdfs-site.xml: dfs.namenode.num.checkpoints.retained: "3" replicas: 1 + ... ---- +<1> `configOverride` on role level. +<2> The name of the configuration file that the override should be put into. You can find the available files as keys in the ConfigMaps created by the operator. +<3> All keys must be strings, even if they are integers like in the example here. +<4> `configOverride` on role group level, takes precedence over overrides at role level. -The role, as well as the configuration file and configuration settings available depend on the specific product. +The roles, as well as the configuration file and configuration settings available depend on the specific product. All override property values must be strings. -The properties will be formatted and escaped correctly into the file format used by the product. +The properties will be formatted and escaped correctly into the file format used by the product, which is again product specific. +This can be a `.properities` file, XML, YAML or JSON. You can also set the property to an empty string (`my.property: ""`), which effectively disables the property the operator would write out normally. In case of a `.properties` file, this will show up as `my.property=` in the `.properties` file. @@ -48,7 +58,7 @@ In case of a `.properties` file, this will show up as `my.property=` in the `.pr [#env-overrides] == Environment variable overrides -For a xref:roles-and-role-groups.adoc[role or role group], at the same level of `config`, you can specify `envOverrides` for any env variable +For a role or role group, at the same level of `config`, you can specify `envOverrides` for any environment variable. An example for an HDFS cluster looks as follows: @@ -59,28 +69,32 @@ kind: HdfsCluster metadata: name: simple-hdfs spec: - nameNodes: # change to your your role - config: # [..] - envOverrides: # on role level + nameNodes: + config: + ... + envOverrides: <1> MY_ENV_VAR: "MY_VALUE" - configOverrides: # on role level - core-site.xml: # change to your desired configuration file - fs.trash.interval: "5" roleGroups: default: - config: # [..] - envOverrides: # on rolegroup level, takes precedence over the ones specified at role level + config: + ... + envOverrides: <2> MY_ENV_VAR: "MY_VALUE" replicas: 1 + ... ---- +<1> `envOverrides` at role level. +<2> `envOverrides` on role group level, takes precedence over the overrides specified at role level. -You can set any environment variable, but every specific product does support a different set of environment variables. +You can set any environment variable, but every Stacklet supports a different set of environment variables, these are documented in the product documentation. All override property values must be strings. [#pod-overrides] == Pod overrides -For a xref:roles-and-role-groups.adoc[role or role group], at the same level of `config`, you can specify `podOverrides` for any of the attributes you can configure on a Pod. +For a role or role group, at the same level of `config`, you can specify `podOverrides` for any of the attributes you can configure on a Pod. +Every Stacklet contains one or more StatefulSets, DaemonSets, or Deployments, which in turn contain a Pod template that is used by Kubernetes to create the Pods that make up the Stacklet. +The `podOverrides` allow you to specify a fragment of a Pod template that is then overlayed over the one created by the operator. An example for an HDFS cluster looks as follows: @@ -91,9 +105,10 @@ kind: HdfsCluster metadata: name: simple-hdfs spec: - nameNodes: # change to your your role - config: # [..] - podOverrides: # on role level + nameNodes: + config: + ... + podOverrides: <1> spec: tolerations: - key: "key1" @@ -102,13 +117,17 @@ spec: effect: "NoSchedule" roleGroups: default: - config: # [..] - podOverrides: # on rolegroup level, takes precedence over the ones specified at role level + config: + ... + podOverrides: <2> metadata: labels: my-custom-label: super-important-label replicas: 1 + ... ---- +<1> `podOverrides` at role level. +<2> `podOverrides` on role group level, takes precedence over the overrides specified at role level. The `podOverrides` can be any valid `PodTemplateSpec` (which means every property that you can set on a regular Kubernetes Pod). @@ -119,7 +138,7 @@ The priority of how to construct the final Pod submitted to Kubernetes looks as 3. PodTemplateSpec given in rolegroup level `podOverrides` Each of these are combined top to bottom using a deep merge. -The exact merge algorithm is described in the https://arnavion.github.io/k8s-openapi/v0.18.x/k8s_openapi/trait.DeepMerge.html[k8s-openapi docs], which basically tries to mimic the way Kubernetes merges patches onto objects. +The exact merge algorithm is described in the {k8s-openapi-deepmerge}[k8s-openapi docs], which basically tries to mimic the way Kubernetes merges patches onto objects. The `podOverrides` will be merged onto the following resources the operators deploy: diff --git a/modules/concepts/pages/overview.adoc b/modules/concepts/pages/overview.adoc index 99986c40c..8452b5f23 100644 --- a/modules/concepts/pages/overview.adoc +++ b/modules/concepts/pages/overview.adoc @@ -1,48 +1,57 @@ = Stackable Data Platform explained The Stackable Data Platform (SDP) is built on Kubernetes. -Its core is a collection of Kubernetes Operators and custom resources which are designed to work together. +Its core is a collection of Kubernetes Operators and CustomResourceDefinitions which are designed to work together. image::overview.drawio.svg[] -The operators are deployed into a Kubernetes cluster, one operator per product (such as Apache ZooKeeper, Apache HDFS, Apache Druid). Every operator has at its core a custom resource (CR) which defines a product instance (shown in green above). The operator creates Kubernetes objects based on the CRs, such as ConfigMaps, StatefulSets and Services. +The operators are deployed into a Kubernetes cluster, one operator per product (such as Apache ZooKeeper, Apache HDFS, Apache Druid). +Every operator has at its core a custom resource (CR) which defines a product instance (shown in green above). +The operator creates Kubernetes objects based on the CRs, such as ConfigMaps, StatefulSets and Services. The operators are deployed with xref:management:stackablectl:index.adoc[] (the Stackable CLI tool) and product instances are created by deploying manifests into Kubernetes. -Aspects like SQL database configuration, xref:resources.adoc[storage configuration] or xref:authentication.adoc[authentication] and xref:opa.adoc[authorization] work the same way across all operators. +Aspects like SQL database configuration, xref:resources.adoc[storage configuration] or xref:authentication.adoc[authentication] and xref:opa.adoc[authorization] are configured the same way in every Stacklet. Most operators support LDAP as a common way to authenticate with product instances and OPA as a common way to set up authorization. [#operators] == Operators -The Operators form the core of the Stackable platform. There is one operator for every supported product, as well as a few supporting operators. All Stackable Operators are built on top of a common framework, so they look and behave in a similar way. +The operators form the core of the Stackable platform. +There is one operator for every supported product, as well as a few supporting operators. +All Stackable operators are built on top of a common framework, so they look and behave in a similar way. -Every Operator relies on a central custom resource (CR) which is specific to the product it operates (i.e. DruidCluster for Apache Druid). -It reads this resource and creates kubernetes resources in accordance with the product CR. +Every operator relies on a central custom resource (CR) which is specific to the product it operates (i.e. DruidCluster for Apache Druid). +It reads this resource and creates Kubernetes resources in accordance with the product CR. image::operator_overview.drawio.svg[] -The diagram above shows the custom resource in green. It contains all the configuration needed to create a product instance. This includes which services the product should connect to, with how many replicas it should operate and how meany resources it should use, among other things. +The diagram above shows the custom resource in green. It contains all the configuration needed to create a product instance. +This includes which services the product should connect to, with how many replicas it should operate and how much of a given resource it should use, among other things. [#discovery] === Discovery -The operator also creates a xref:service-discovery.adoc[**discovery ConfigMap**] for every product instance which is used by other products to connect to it. The ConfigMap has the same name as the product instance and contains information about how to connect to the product. This ConfigMap can then be referenced in other product instance resources. +The operator also creates a xref:service-discovery.adoc[**discovery ConfigMap**] for every product instance which is used by other products to connect to it. +The ConfigMap has the same name as the product instance and contains information about how to connect to the product. +This ConfigMap can then be referenced in other product instance resources. image::discovery.drawio.svg[] -For example, Apache ZooKeeper is a dependency of many other products, such as Apache HDFS and Apache Druid. The HDFS and Druid resources can simply reference the ZooKeeper cluster by name and the operators will use the discovery ConfigMap to configure the Druid and HDFS Pods to connect to the ZooKeeper Service. +For example, Apache ZooKeeper is a dependency of many other products, such as Apache HDFS and Apache Druid. +The HDFS and Druid resources can simply reference the ZooKeeper cluster by name and the operators will use the discovery ConfigMap to configure the Druid and HDFS Pods to connect to the ZooKeeper Service. -You can also create these discovery ConfigMaps yourself to make products discoverable that are not operatored by a Stackable Operator. Learn more about product discovery at xref:service-discovery.adoc[]. +You can also create these discovery ConfigMaps yourself to make products discoverable that are not operated by a Stackable operator. +Learn more about product discovery at xref:service-discovery.adoc[]. [#roles] === Roles -Almost all products that Stackable supports need multiple different processes to run. Because they are often still the same software but running with different parameters, Stackable calls them _roles_. For example HDFS has three roles: DataNode, NameNode and JournalNode. - -All roles are configured together in the custom resource for the product, but they each get their own StatefulSet, ConfigMaps and Service. - -Learn more about roles: xref:roles-and-role-groups.adoc[] +Many of the data products that Stackable supports require multiple different components to run, which together make up the product instance. +For example an HDFS cluster is made up of DataNodes, NameNodes and JournalNodes. +Stackable calls the components that make up the product xref:stacklet.adoc#roles[_roles_]. +All roles are configured together in the custom resource for the product and there is a dedicated configuration section for each role. +Every role is running using the same underlying container image, but with different parameters and they each get their own StatefulSet, ConfigMaps and Service. [#deployment] == Deployment diff --git a/modules/concepts/pages/resources.adoc b/modules/concepts/pages/resources.adoc index 20457f100..0a41e7be4 100644 --- a/modules/concepts/pages/resources.adoc +++ b/modules/concepts/pages/resources.adoc @@ -1,21 +1,25 @@ = Resource management -The https://stackable.tech/en/[Stackable data platform] and its xref:operators:index.adoc[operators] deploy their products in https://kubernetes.io/docs/concepts/containers/[Containers] within https://kubernetes.io/docs/concepts/workloads/pods/[Pods] using https://kubernetes.io/docs/concepts/workloads/controllers/statefulset/[StatefulSets] or https://kubernetes.io/docs/concepts/workloads/controllers/daemonset/[DaemonSets]. In order for the Kubernetes scheduler to select a proper https://kubernetes.io/docs/concepts/architecture/nodes/[Node] for a Pod, https://kubernetes.io/docs/concepts/configuration/manage-resources-containers/[resource] requests and limits for CPU and memory can be specified. The Kubernetes scheduler ensures that the sum of the CPU and memory requests does not exceed the capacity of a given Node. +The Stackable Data Platform and its xref:operators:index.adoc[operators] deploy their products in https://kubernetes.io/docs/concepts/containers/[containers] within https://kubernetes.io/docs/concepts/workloads/pods/[Pods] using https://kubernetes.io/docs/concepts/workloads/controllers/statefulset/[StatefulSets] or https://kubernetes.io/docs/concepts/workloads/controllers/daemonset/[DaemonSets]. +In order for the Kubernetes scheduler to select a suitable https://kubernetes.io/docs/concepts/architecture/nodes/[Node] for a Pod, https://kubernetes.io/docs/concepts/configuration/manage-resources-containers/[resource] requests and limits for CPU and memory can be specified. +The Kubernetes scheduler ensures that the sum of the CPU and memory requests does not exceed the capacity of a given Node. == Terminology -The most commonly defined resources are CPU and memory (RAM). Keep in mind that there are other resources as well. +The most commonly defined resources are CPU and memory (RAM). +Keep in mind that there are other resources as well. For more information have a look at the Kubernetes https://kubernetes.io/docs/concepts/configuration/manage-resources-containers/#requests-and-limits[documentation] on resources. === CPU ==== Requests -The CPU request defines which containers are allowed to have more CPU resources. Larger requests lead to more CPU resources than smaller requests if not enough resources are available. +The CPU request defines which containers are allowed to have more CPU resources in case of a shortage of compute resources. ==== Limits -The CPU limit is a hard bound on how much CPU resources the Container can use. The Linux Kernel checks to see if the limit is exceeded and waits if this is the case to resume the process. +The CPU limit is a hard bound on how much CPU resources the container can use. +The Linux kernel checks to see if the limit is exceeded and waits if this is the case to resume the process. === Memory @@ -25,15 +29,18 @@ The memory request is used during Kubernetes scheduling and checks which Nodes o ==== Limits -The memory limit is a hard bound. If a Container tries to use more memory than specified, the Container is usually marked for restart. To avoid the restart it is critical to specify sufficient resources. +The memory limit is a hard bound. +If a container tries to use more memory than specified, the container is usually marked for restart. +To avoid the restart it is critical to specify sufficient resources. === Storage -Some Stackable products require data storage. This is done using https://kubernetes.io/docs/concepts/storage/persistent-volumes/#persistentvolumeclaims[Persistent Volume Claims] where the size of storage can be specified. +Some Stackable products require data storage. +This is done using https://kubernetes.io/docs/concepts/storage/persistent-volumes/#persistentvolumeclaims[Persistent Volume Claims] where the size of storage can be specified. -== Kubernetes Resource Requests +== Kubernetes resource requests -In Kubernetes, you can specify resource requests and limits within Containers of Pods. +In Kubernetes, you can specify resource requests and limits within containers of Pods. [source, yaml] ---- @@ -55,11 +62,13 @@ spec: cpu: "500m" ---- -This Pod/Container will be scheduled on a Node that has at least `64Mi` of free memory. It is allowed to use a maximum amount of `128Mi` of memory or will be restarted if this value is exceeded. It can not use more CPU resources than `500m` (which is half of a physical or virtual core), but has a guaranteed CPU resources of `250m`. +This Pod/container will be scheduled on a Node that has at least `64Mi` of free memory. +It is allowed to use a maximum amount of `128Mi` of memory or will be restarted if this value is exceeded. +It can not use more CPU resources than `500m` (which is half of a physical or virtual core), but has a guaranteed CPU resources of `250m`. -== Stackable Resource Requests +== Stackable resource requests -=== CPU and Memory +=== CPU and memory include::stackable_resource_requests.adoc[] @@ -93,7 +102,8 @@ spec: capacity: 4Gi ---- -In this case, the role group `resources-from-role` will inherit the resources specified on the role level. Resulting in a `3Gi` storage space for `data`. +In this case, the role group `resources-from-role` will inherit the resources specified on the role level. +Resulting in a `3Gi` storage space for `data`. The role group `resources-from-role-group` has maximum of `4Gi` storage space for `data` (which overrides the role resources). @@ -107,7 +117,8 @@ The StorageClasses that are available on a Kubernetes cluster are configured by Different classes can be configured to provide different levels of reliability or speed, or be configured to be more suited for read or write heavy loads. This configuration is either done in the storage backend or Kubernetes settings (find more information in the https://kubernetes.io/docs/concepts/storage/storage-classes/[Kubernetes documentation]). -For Stackable resources, setting a StorageClass is not mandatory; if not StorageClass is set, the https://kubernetes.io/docs/concepts/storage/storage-classes/#default-storageclass[default StorageClass] will be used. If you want to use a specific StorageClass for a particular storage, the StorageClass can be set on the resource: +For Stackable resources, setting a StorageClass is not mandatory; if a StorageClass is not set, the https://kubernetes.io/docs/concepts/storage/storage-classes/#default-storageclass[default StorageClass] will be used. +If you want to use a specific StorageClass for a particular storage, the StorageClass can be set on the resource: [source,yaml] ---- diff --git a/modules/concepts/pages/roles-and-role-groups.adoc b/modules/concepts/pages/roles-and-role-groups.adoc deleted file mode 100644 index c6b1b6cdc..000000000 --- a/modules/concepts/pages/roles-and-role-groups.adoc +++ /dev/null @@ -1,76 +0,0 @@ -= Roles and role groups - -Software - especially the distributed kind - often consists of multiple different processes that work together to achieve their goal. Think of a client communicating with a server. Stackable calls these _roles_. The roles or processes are often replicated multiple times. These replicas can further be subdivided into _role groups_. Roles and role groups are defined in the resource manifest. - -== Motivation -// configuration and scheduling -The use of roles and role groups is required to control *how* processes are configured and *where* they run. - -=== Configuration -// - the "how" -// internals of the process -// thread counts, heap size, - -Different processes have different tasks that they need to fulfill, which in turn have different configuration settings that only apply to that task. For example coordination, storage, logging and processing tasks require different amounts of threads, memory and storage capacity. These different settings can be put in the per role configuration spec. - -=== Scheduling -// the "where" -// node labels for location and hardware are typical uses -// also nodes that might have GPU attached, or special storage - -While configuration is concerned with modifying how the process works on the inside, scheduling is concerned with the context where it should run inside the cluster. A processing task should be scheduled on a node with faster CPUs or nodes with GPUs attached. A caching process is best scheduled on a node with SSD storage, for example. - -This is done by using link:https://kubernetes.io/docs/concepts/overview/working-with-objects/labels/[label selectors]. Nodes can be labeled and then processes can be assigned to nodes based on these labels. - -=== Role groups - -Role groups are motivated by the need to subdivide the replicas of a process into groups as well, either to configure or to schedule multiple groups differently. For example in a cluster spanning multiple physical locations, you may want to make sure half of your processes run in one location and half in the other. - -== How it works - -A distributed software consists of one or more roles. Every role must be configured with at least one role group. Configuration can be specified at role level or at role group level, where the role group level overrides the role level. Label selectors for scheduling and the number of replicas are specified on the role group, not role. The role group is the lowest level, so it makes sense to have to specify this there. - -=== Example - -HDFS uses three distinct processes that work together to fulfill its duty: NameNodes, DataNodes and JournalNodes. With roles you can specify different configuration settings for each of these, as well as schedule them on specific machines. - -[source,yaml] ----- -apiVersion: hdfs.stackable.tech/v1alpha1 -kind: HdfsCluster -metadata: - name: simple-hdfs -spec: - journalNodes: - roleGroups: - default: - replicas: 3 # <1> - nameNodes: - roleGroups: - default: - replicas: 3 - dataNodes: - config: - resources: - storage: - data: - capacity: 1Gi # <2> - roleGroups: - default: - replicas: 2 - hiCapacity: # <3> - config: - resources: - storage: - data: 2Gi - replicas: 1 - ... ----- - -<1> The JournalNode role with only a single default role group. For the role group 3 replicas are specified, specifying a replica count is optional, the default is 1. -<2> A common config setting for the DataNodes role. This configuration setting applies only to pods for this role. -<3> The DataNode role has two role groups, the default group and the hiCapacity group. In it the config setting for the group is overridden with a higher value of 2Gi. This role group has only one replica. - -== Further Reading - -Every operator uses roles. Have a look at the usage page of an operator for an example of roles. \ No newline at end of file diff --git a/modules/concepts/pages/service-exposition.adoc b/modules/concepts/pages/service-exposition.adoc index 27e6ae499..0471de9a2 100644 --- a/modules/concepts/pages/service-exposition.adoc +++ b/modules/concepts/pages/service-exposition.adoc @@ -1,41 +1,63 @@ = Service exposition +:k8s-service: https://kubernetes.io/docs/concepts/services-networking/service/ +:k8s-service-types: https://kubernetes.io/docs/concepts/services-networking/service/#publishing-services-service-types -All data products expose an interface to the outside world. This interface can be accessed by other products or by end users. Other products accessing the interface can run inside or outside the same Kubernetes cluster. For example, Apache ZooKeeper is a dependency for other products, and it usually needs to be accessible only from within Kubernetes, while Superset is a data analysis product for end users and therefore needs to be accessible from outside the Kubernetes cluster. Users connecting to Superset can be restricted within the local company network, or they can connect over the internet depending on the company security policies and demands. -This page gives an overview over the different options for service exposition, when to chose which option and how these options are configured. +Data products expose interfaces to the outside world. +These interfaces (whether UIs, or APIs) can be accessed by other products or by end users. +Other products accessing the interfaces can run inside or outside of the same Kubernetes cluster. +For example, xref:zookeeper:index.adoc[Apache ZooKeeper] is a dependency for other products, and it usually needs to be accessible only from within Kubernetes, while xref:superset:index.adoc[Apache Superset] is a data analysis product for end users and therefore needs to be accessible from outside the Kubernetes cluster. +Users connecting to Superset can be restricted within the local company network, or they can connect over the internet depending on the company security policies and demands. +This page gives an overview over the different options for service exposition, when to choose which option and how these options are configured. == Service exposition options -The service offered by a data product is the utility it is used for, but https://kubernetes.io/docs/concepts/services-networking/service/[Service] also means the Kubernetes resource. The Stackable Data Platform supports three https://kubernetes.io/docs/concepts/services-networking/service/#publishing-services-service-types[types of Service]: +The Stackable Data Platform supports three {k8s-service-types}[types of Kubernetes Service] for exposing data product endpoints: * ClusterIP * NodePort * LoadBalancer -All custom resources for data products provide a resource field named `spec.clusterConfig.listenerClass` which determines how the product can be accessed . There are three ListenerClasses, named after the goal for which they are used (more on this in the <>): +All custom resources for data products provide a resource field named `spec.clusterConfig.listenerClass` which determines how the product can be accessed. +There are three ListenerClasses, named after the goal for which they are used (more on this in the <>): * `cluster-internal` => Use ClusterIP (default) * `external-unstable` => Use NodePort * `external-stable` => Use LoadBalancer -The `cluster-internal` class exposes the interface of a product by using a ClusterIP Service. This service is only reachable from within the Kubernetes cluster. This setting is the most secure and was chosen as the default for that reason. +The `cluster-internal` class exposes the interface of a product by using a ClusterIP Service. +This service is only reachable from within the Kubernetes cluster. +This setting is the most secure and was chosen as the default for that reason. -NOTE: Not all Operators support all classes. Consult the Operator specific documentation to find out about the supported service types. +NOTE: Not all operators support all classes. +Consult the operator specific documentation to find out about the supported service types. -== [[when-to-choose-which-option]]When to choose which option +[#when-to-choose-which-option] +== When to choose which option -There are three options, one for internal traffic and two for external access, where internal and external refer to the Kubernetes cluster. Internal means inside of the Kuberenetes cluster, and external means access from outside of it. +There are three options, one for internal traffic and two for external access, where internal and external refer to the Kubernetes cluster. +Internal means inside of the Kuberenetes cluster, and external means access from outside of it. === Internal -`cluster-internal` is the default class and, the Service behind it is only exposed within Kubernetes. This is useful for middleware products such as xref:zookeeper:index.adoc[Apache ZooKeeper], the xref:hive:index.adoc[Apache Hive metastore] or a xref:kafka:index.adoc[Apache Kafka] cluster used for internal data flow. Products using this ListenerClass are not accessible from outside Kubernetes. +`cluster-internal` is the default class and the Service behind it is only reachable from within Kubernetes. +This is useful for middleware products such as xref:zookeeper:index.adoc[Apache ZooKeeper], xref:hive:index.adoc[Apache Hive metastore], or an xref:kafka:index.adoc[Apache Kafka] cluster used for internal data flow. +Products using this ListenerClass are not accessible from outside Kubernetes. === External -External access is needed when a product needs to be accessed from _outside_ of Kubernetes. This is necessary for all end user products such as xref:superset:index.adoc[Apache Superset]. Some tools can expose APIs for data ingestion like xref:kafka:index.adoc[Apache Kafka] or xref:nifi:index.adoc[Apache NiFi]. If data needs to be ingested from outside of the cluster, one of the external listener classes should be chosen. +External access is needed when a product needs to be accessed from _outside_ of Kubernetes. +This is necessary for all end user products such as xref:superset:index.adoc[Apache Superset]. +Some tools can expose APIs for data ingestion like xref:kafka:index.adoc[Apache Kafka] or xref:nifi:index.adoc[Apache NiFi]. +If data needs to be ingested from outside of the cluster, one of the external listener classes should be chosen. -When to use `stable` and when to use `unstable`? The `external-unstable` setting exposes a product interface via a Kuberneres NodePort. In this case the service's IP address and port can change if Kubernetes needs to restart or reschedule the Pod to another node. +When to use `stable` and when to use `unstable`? +The `external-unstable` setting exposes a product interface via a Kuberneres NodePort. +In this case the service's IP address and port can change if Kubernetes needs to restart or reschedule the Pod to another node. -The `external-stable` class uses a LoadBalancer. The LoadBalancer is running at a fixed address and is therefore `stable`. Managed Kubernetes services in the cloud usually offer a LoadBalancer, but for an on premise cluster you have to configure a LoadBalancer yourself. For a production setup, it is recommended to use a LoadBalancer or `external-stable` ListenerClass. +The `external-stable` class uses a LoadBalancer. +The LoadBalancer is running at a fixed address and is therefore `stable`. +Managed Kubernetes services in the cloud usually offer a LoadBalancer, but for an on premise cluster you have to configure a LoadBalancer yourself. +For a production setup, it is recommended to use a LoadBalancer and the `external-stable` ListenerClass. == Outlook diff --git a/modules/concepts/pages/stackable_resource_requests.adoc b/modules/concepts/pages/stackable_resource_requests.adoc index 563c096a0..b6a66c85e 100644 --- a/modules/concepts/pages/stackable_resource_requests.adoc +++ b/modules/concepts/pages/stackable_resource_requests.adoc @@ -1,7 +1,10 @@ // This is meant to be inlined using the "include" directive in other pages. // WARNING: do not add headers here as they can break the structure of pages // that include this file. -Stackable operators handle resource requests in a sligtly different manner than Kubernetes. Resource requests are defined on role or group level. See xref:concepts:roles-and-role-groups.adoc[] for details on these concepts. On a role level this means that e.g. all workers will use the same resource requests and limits. This can be further specified on role group level (which takes priority to the role level) to apply different resources. +Stackable operators handle resource requests in a sligtly different manner than Kubernetes. +Resource requests are defined on xref:concepts:stacklet.adoc#roles[role] or xref:concepts:stacklet.adoc#role-groups[role group] level. +On a role level this means that by default, all workers will use the same resource requests and limits. +This can be further specified on role group level (which takes priority to the role level) to apply different resources. This is an example on how to specify CPU and memory resources using the Stackable https://kubernetes.io/docs/concepts/extend-kubernetes/api-extension/custom-resources/[Custom Resources]: @@ -35,10 +38,12 @@ spec: limit: 4Gi ---- -In this case, the role group `resources-from-role` will inherit the resources specified on the role level. Resulting in a maximum of `3Gi` memory and `600m` CPU resources. +In this case, the role group `resources-from-role` will inherit the resources specified on the role level, resulting in a maximum of `3Gi` memory and `600m` CPU resources. -The role group `resources-from-role-group` has maximum of `4Gi` memory and `800m` CPU resources (which overrides the role CPU resources). +The role group `resources-from-role-group` has a maximum of `4Gi` memory and `800m` CPU resources (which overrides the role CPU resources). -IMPORTANT: For Java products the actual used Heap memory is lower than the specified memory limit due to other processes in the Container requiring memory to run as well. Currently, 80% of the specified memory limits is passed to the JVM. +IMPORTANT: For Java products the actual used heap memory is lower than the specified memory limit due to other processes in the Container requiring memory to run as well. +Currently, 80% of the specified memory limit is passed to the JVM. -For memory only a limit can be specified, which will be set as memory request and limit in the Container. This is to always guarantee a Container the full amount memory during Kubernetes scheduling. +For memory, only a limit can be specified, which will be set as memory request and limit in the container. +This is to always guarantee a container the full amount memory during Kubernetes scheduling. diff --git a/modules/concepts/pages/stacklet.adoc b/modules/concepts/pages/stacklet.adoc new file mode 100644 index 000000000..dc2b77ddb --- /dev/null +++ b/modules/concepts/pages/stacklet.adoc @@ -0,0 +1,86 @@ += Stacklet +:page-aliases: roles-and-role-groups.adoc + +A _Stacklet_ is a deployed product that is managed by a Stackable operator. +The running instance is made up of multiple pieces of software called _roles_ and can be further subdivided into _role groups_. + +A Stacklet is defined by a Kubernetes resource that is managed by an operator, for example a DruidCluster, TrinoCluster or SparkApplication. +The associated operator then creates a number of Kubernetes resources that are needed to run, configure and expose the product. +For every operator (for example xref:zookeeper:index.adoc[]) the documentation describes the Stacklet it manages and the resources that it creates. +The collection of all the resources that make up the functioning product is the Stacklet. + +The products are usually deployed with StatefulSets from which Pods are created. +Configuration is done with ConfigMaps, and the the software is exposed using Services. +To allow for easier connectivity between Stacklets, some operators also deploy a xref:service-discovery.adoc[] for Stacklets they manage. + +CAUTION: Watch out for name collisions when deploying multiple Stacklets in the same namespace! +Even though the resource might be different (TrinoCluster, HbaseCluster), there is a name conflict for the discovery ConfigMap if two Stacklets in the same Kubernetes namespace share a name. +It is best to always use unique names for Stacklets. + +Have a look at the <> at the end of the page for a YAML example of a HdfsCluster Stacklet. + +[#roles] +== Roles + +_Roles_ are the core of a Stacklet, and every Stacklet will have at least one role defined. +The roles are the various components that make up the running product. +They are called this way, because under the hood, all roles use the same container image, but running with different parameters; sam image, different role. +Some products have only one running component (and therefore one role) like xref:kafka:index.adoc[Apache Kafka], xref:nifi:index.adoc[Apache NiFi], xref:zookeeper:index.adoc[Apache ZooKeeper] or xref:opa:index.adoc[OPA] while others need multiple different roles, often a combination of worker and coordinator processes, sometimes with a gateway or router process like xref:airflow:index.adoc[Apache Airflow], xref:trino:index.adoc[Trino] or xref:druid:index.adoc[Apache Druid]. + +The different roles have different configuration and scheduling requirements; for example some roles are resource intensive, some require multiple replicas for consistency/availability/partition-tolerance, and for others, only a single replica is enough. +Some configuration options only make sense when applied to a particular role. + +For example, you only need a single coordinator process with little compute and memory, but many worker processes that then need more compute and memory. + +[#role-groups] +== Role groups + +Role groups are a further subdivision of roles, that allow more fine grained control over subsets of replicas that all belong to the same role. +This is useful again to configure scheduling, i.e. have two role groups run in different regions, or run on different classes of nodes (eg: faster CPUs, faster disk access, GPU). +Role groups are a flexible mechanism that you can adapt to whatever is useful for you, but you can also use just a single role group per role - often this is then called `default` by convention. + +Configuration settings can be made at role and role group level. +Any configuration made at role group level overrides role level settings, as it is more specific. + +[#example] +== Example + +HDFS uses three distinct components that work together to fulfill its duty: NameNodes, DataNodes and JournalNodes. +With roles you can specify different configuration settings for each of these. + +[source,yaml] +---- +apiVersion: hdfs.stackable.tech/v1alpha1 +kind: HdfsCluster +metadata: + name: simple-hdfs +spec: + journalNodes: + roleGroups: + default: + replicas: 3 # <1> + nameNodes: + roleGroups: + default: + replicas: 3 + dataNodes: + config: + resources: + storage: + data: + capacity: 1Gi # <2> + roleGroups: + default: + replicas: 2 + highCapacity: # <3> + config: + resources: + storage: + data: 2Gi + replicas: 1 + ... +---- + +<1> The JournalNode role with only a single default role group. For the role group 3 replicas are specified, specifying a replica count is optional, the default is 1. +<2> A common config setting for the DataNodes role. This configuration setting applies only to Pods for this role. +<3> The DataNode role has two role groups, the default group and the highCapacity group. The highCapacity group settings override the role defaults with a higher storage value of 2Gi and only one replica. diff --git a/modules/operators/pages/monitoring.adoc b/modules/operators/pages/monitoring.adoc index 3deb46a70..d671af5f7 100644 --- a/modules/operators/pages/monitoring.adoc +++ b/modules/operators/pages/monitoring.adoc @@ -1,23 +1,23 @@ = Monitoring +:prometheus: https://prometheus.io/ +:prometheus-operator: https://prometheus-operator.dev/ -Services managed by Stackable support monitoring via https://prometheus.io/[Prometheus]. +Services managed by Stackable support monitoring via {prometheus}[Prometheus]. -== Prometheus Operator +== Prometheus operator -Stackable does not currently provide Prometheus, instead we suggest using -https://prometheus-operator.dev/[Prometheus Operator]. +Stackable does not currently provide Prometheus, instead we suggest using {prometheus-operator}[Prometheus operator]. === Installing Prometheus Prometheus Operator can be installed via the Helm chart `kube-prometheus-stack`, which includes both the Operator, and a basic Prometheus configuration that should be sufficient for basic use. -[source,bash] -helm repo add prometheus-community https://prometheus-community.github.io/helm-charts -helm upgrade prometheus prometheus-community/kube-prometheus-stack --install --version 31.0.0 +[source,shell] +$ helm repo add prometheus-community https://prometheus-community.github.io/helm-charts +$ helm upgrade prometheus prometheus-community/kube-prometheus-stack --install --version 31.0.0 -When using the `kube-prometheus-stack` Helm chart (as above), an additional `ServiceMonitor` must be applied to -the Kubernetes cluster, which discovers services registered to the Kubernetes cluster: +When using the `kube-prometheus-stack` Helm chart (as above), an additional `ServiceMonitor` must be applied to the Kubernetes cluster, which discovers services registered to the Kubernetes cluster: [source,yaml] ---- @@ -28,8 +28,8 @@ include::example$monitoring-servicemonitor.yaml[] Prometheus should now be accessible inside of the Kubernetes cluster, and can be access can be forwarded using `kubectl`: -[source,bash] -kubectl port-forward svc/prometheus-kube-prometheus-prometheus 9090 +[source,shell] +$ kubectl port-forward svc/prometheus-kube-prometheus-prometheus 9090 Afterwards, we can go to http://localhost:9090/ to access the query UI. diff --git a/modules/operators/pages/supported_versions.adoc b/modules/operators/pages/supported_versions.adoc index dc86f0d56..16b805c99 100644 --- a/modules/operators/pages/supported_versions.adoc +++ b/modules/operators/pages/supported_versions.adoc @@ -3,7 +3,7 @@ Below you find the list of versions that are supported for each product. New versions get added every release, old ones are deprecated or removed. Read the xref:ROOT:release-notes.adoc[] for each platform version to see which product versions are available. -Read about xref:concepts:product-image-selection.adoc[] to learn how to use a particular version in your Stacklet, as well as how you can use custom images or a custom registry. +Read about xref:concepts:product-image-selection.adoc[] to learn how to use a particular version in your xref:concepts:stacklet.adoc[], as well as how you can use custom images or a custom registry. == Apache Airflow diff --git a/modules/reference/pages/glossary.adoc b/modules/reference/pages/glossary.adoc index 4cf84099d..101d0d9de 100644 --- a/modules/reference/pages/glossary.adoc +++ b/modules/reference/pages/glossary.adoc @@ -34,7 +34,7 @@ dt { [%collapsible%open] ==== For example HDFS consists of 3 roles: Name nodes, journal nodes and data nodes. -Learn more about xref:concepts:roles-and-role-groups.adoc[]. +Learn more about xref:concepts:stacklet.adoc#roles[roles]. ==== [[role-group]]Role group <>:: {empty} @@ -43,7 +43,7 @@ Learn more about xref:concepts:roles-and-role-groups.adoc[]. [%collapsible%open] ==== A role group can override configuration set at role level, allowing for different configurations for sets of processes. -Learn more about xref:concepts:roles-and-role-groups.adoc[]. +Learn more about xref:concepts:stacklet.adoc#role-groups[role groups]. ==== [[stacklet]]Stacklet <>:: {empty}