Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
47 commits
Select commit Hold shift + click to select a range
c0c7f59
draft
fhennig May 27, 2024
f000ed4
fix typos
fhennig May 28, 2024
3b92224
overhauled some pages
fhennig May 28, 2024
993bce5
~
fhennig May 28, 2024
c78d4ea
~
fhennig May 28, 2024
520d106
~
fhennig May 28, 2024
17294e0
~
fhennig May 28, 2024
5a20a8d
~
fhennig May 28, 2024
d69303b
fixed incorrect ref
fhennig May 28, 2024
6078b43
Update modules/concepts/pages/index.adoc
fhennig May 29, 2024
50b0c9f
Update modules/concepts/pages/labels.adoc
fhennig May 29, 2024
f55d8c9
Update modules/concepts/pages/labels.adoc
fhennig May 29, 2024
e4ca51a
Update modules/concepts/pages/labels.adoc
fhennig May 29, 2024
58080e6
Update modules/concepts/pages/logging.adoc
fhennig May 29, 2024
4477a22
Update modules/concepts/pages/opa.adoc
fhennig May 29, 2024
4408f9b
Update modules/concepts/pages/overrides.adoc
fhennig May 29, 2024
e4d93a7
Update modules/concepts/pages/overrides.adoc
fhennig May 29, 2024
e3887fd
Update modules/concepts/pages/service-exposition.adoc
fhennig May 29, 2024
c9e38a0
Update modules/concepts/pages/service-exposition.adoc
fhennig May 29, 2024
82773ce
Update modules/concepts/pages/resources.adoc
fhennig May 29, 2024
9b8feb8
Update modules/concepts/pages/resources.adoc
fhennig May 29, 2024
816cc04
Update modules/concepts/pages/overrides.adoc
fhennig May 29, 2024
8da0870
Update modules/concepts/pages/overrides.adoc
fhennig May 29, 2024
b3b4c60
Update modules/concepts/pages/overrides.adoc
fhennig May 29, 2024
647f7db
Update modules/concepts/pages/overview.adoc
fhennig May 29, 2024
a1faa3f
Update modules/concepts/pages/overview.adoc
fhennig May 29, 2024
4f219af
Update modules/concepts/pages/overview.adoc
fhennig May 29, 2024
91d785b
Update modules/concepts/pages/stacklet.adoc
fhennig May 29, 2024
53bbbc0
Update modules/concepts/pages/stacklet.adoc
fhennig May 29, 2024
288c9be
Update modules/concepts/pages/stacklet.adoc
fhennig May 29, 2024
23efc01
Update modules/concepts/pages/stacklet.adoc
fhennig May 29, 2024
3554689
Update modules/concepts/pages/stackable_resource_requests.adoc
fhennig May 29, 2024
b0c2ce8
Update modules/concepts/pages/stackable_resource_requests.adoc
fhennig May 29, 2024
4a95cac
Update modules/concepts/pages/service-exposition.adoc
fhennig May 29, 2024
21b8fc9
Update modules/concepts/pages/service-exposition.adoc
fhennig May 29, 2024
341d220
Update modules/concepts/pages/service-exposition.adoc
fhennig May 29, 2024
7638f1c
clarify role text
fhennig May 30, 2024
40d2b4e
change stacklet text
fhennig May 30, 2024
af3caf4
Update modules/concepts/pages/stacklet.adoc
fhennig May 30, 2024
5c238a6
Update modules/concepts/pages/stackable_resource_requests.adoc
fhennig May 30, 2024
b363e60
change CPU resource text
fhennig May 30, 2024
0e7f8d4
change process to component
fhennig May 30, 2024
3d04e18
remove confusing sentence
fhennig May 30, 2024
ef3bc24
add suggestion
fhennig May 30, 2024
a36ab60
Update modules/concepts/pages/overview.adoc
fhennig Jun 12, 2024
7acb16a
Merge branch 'main' into add-stacklet-concepts-page
fhennig Jun 12, 2024
51ba5a7
Merge branch 'main' into add-stacklet-concepts-page
fhennig Jun 19, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions modules/concepts/nav.adoc
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
* xref:concepts:index.adoc[]
** xref:overview.adoc[Overview]
** xref:overview.adoc[Platform overview]
** xref:stacklet.adoc[]
** Common configuration mechanisms
*** xref:roles-and-role-groups.adoc[]
*** xref:product_image_selection.adoc[]
*** xref:overrides.adoc[Advanced: overrides]
** Resources
Expand Down
21 changes: 11 additions & 10 deletions modules/concepts/pages/authentication.adoc
Original file line number Diff line number Diff line change
@@ -1,7 +1,8 @@
= Authentication
:keycloak: https://www.keycloak.org/

The Stackable Platform uses the AuthenticationClass as a central mechanism to handle user authentication across supported products.
The authentication mechanism needs to be configured only in the AuthenticationClass which is then referenced in the product.
The authentication mechanism needs to be configured only in the AuthenticationClass which is then referenced in the xref:stacklet.adoc[Stacklet] definition.
Multiple different authentication providers are supported.

[#authenticationclass]
Expand Down Expand Up @@ -40,7 +41,7 @@ NOTE: Learn more in the xref:tutorials:authentication_with_openldap.adoc[OpenLDA
[#OIDC]
=== OpenID Connect

An OIDC provider like https://www.keycloak.org/[Keycloak {external-link-icon}^] could be configured as follows:
An OIDC provider like {keycloak}[Keycloak] could be configured as follows:

[source,yaml]
----
Expand All @@ -60,10 +61,10 @@ NOTE: Get a full overview of all the properties in the {crd-docs}/authentication
=== TLS
The `TLS` provider configures a product to authenticate users using TLS certificates.
When establishing a connection the client will first validate the certificate of the server.
This step is not influenced by this `AuthenticationClass`, it only affects the next step:
This step is not influenced by this AuthenticationClass, it only affects the next step:
Afterwards the server checks the validity of the certificate the client has provided.
This includes the usual checks - such as checking that it hasn't expired and matches the hostname of the client.
Additionally the client certificate needs to be signed with the `ca` certificate, which is provided by the `SecretClass` specified in `clientCertSecretClass`.
Additionally the client certificate needs to be signed with the `ca` certificate, which is provided by the SecretClass specified in `clientCertSecretClass`.

A sample TLS provider looks as follows:

Expand All @@ -77,25 +78,25 @@ include::example$authenticationclass-tls.yaml[]
The `static` provider is used to represent a simple - static - set of users.
Users are identified by a username and a password.

First, the `AuthenticationClass` needs to be defined as follows:
First, the AuthenticationClass needs to be defined as follows:

[source,yaml]
----
include::example$authenticationclass-static-authenticationclass.yaml[]
----
<1> The name of the `Secret` containing the credentials
<1> The name of the Secret containing the credentials

Afterwards the referenced `Secret` needs to be created:
Afterwards the referenced Secret needs to be created:

[source,yaml]
----
include::example$authenticationclass-static-secret.yaml[]
----
<1> The name of the `Secret`, which needs to match the `Secret` name specified in the `AuthenticationClass` above
<2> The namespace of the `Secret`. The `Secret` needs to be in the same namespace as the product that tries to use the static `AuthenticationClass`
<1> The name of the Secret, which needs to match the Secret name specified in the AuthenticationClass above
<2> The namespace of the Secret. The Secret needs to be in the same namespace as the product that tries to use the static AuthenticationClass

[#further-reading]
== Further Reading
== Further reading

* xref:tutorials:authentication_with_openldap.adoc[] tutorial
* {crd-docs}/authentication.stackable.tech/authenticationclass/v1alpha1/[AuthenticationClass CRD reference]
2 changes: 1 addition & 1 deletion modules/concepts/pages/index.adoc
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@ The xref:overview.adoc[Platform overview] is a good starting point to understand

== General configuration mechanisms

Learn about xref:roles-and-role-groups.adoc[], how product image selection works.
Learn what a xref:stacklet.adoc[Stacklet] is, what roles and role groups are, and how product image selection works.
There is also the common xref:overrides.adoc[override] mechanism for configuration settings, although this tool should be used with care!

== Resources
Expand Down
23 changes: 12 additions & 11 deletions modules/concepts/pages/labels.adoc
Original file line number Diff line number Diff line change
@@ -1,25 +1,26 @@
= Labels
:common-labels: https://kubernetes.io/docs/concepts/overview/working-with-objects/common-labels/

Labels are key/value pairs in the metadata of Kubernetes objects that add identifying information to the object.
They do not have direct semantic implications but can be used to organize and manage resources.
The `stackablectl` tool, the cockpit and the Helm Charts add labels to the resources that are part of a Stacklet, and the operators add labels to the resources they create.
The xref:management:stackablectl:index.adoc[`stackablectl`] tool, the cockpit, and the Helm Charts add labels to the resources that are part of a xref:stacklet.adoc[Stacklet], and the operators add labels to the resources they create.

== Resource labels added by the operators

Every resource created by a Stackable operator is labelled with a common set of labels.
Some of these labels are https://kubernetes.io/docs/concepts/overview/working-with-objects/common-labels/[recommended] to use by the Kubernetes documentation.
Every resource created by a Stackable operator has a common set of labels applied.
Some of these labels are {common-labels}[recommended] to use by the Kubernetes documentation.
The following labels are added to resources created by our operators:

- `app.kubernetes.io/name` - The name of the product, i.e. `druid` or `zookeeper`.
- `app.kubernetes.io/version` - The version of the product and Stackable version, i.e. `3.3.4-stackable23.11.0`
- `app.kubernetes.io/instance` - The name of the Stacklet.
- `app.kubernetes.io/component` - This is the xref:concepts:roles-and-role-groups.adoc[role] that this resource belongs to.
- `app.kubernetes.io/role-group` - The name of the xref:concepts:roles-and-role-groups.adoc[role group] that a resource belongs to.
- `app.kubernetes.io/managed-by` - Which software manages this resource? This will be the operator, i.e. `airflow-operator`.
* `app.kubernetes.io/name` - The name of the product, i.e. `druid` or `zookeeper`.
* `app.kubernetes.io/version` - The version of the product and Stackable version, i.e. `3.3.4-stackable24.3.0`
* `app.kubernetes.io/instance` - The name of the Stacklet.
* `app.kubernetes.io/component` - This is the xref:stacklet.adoc#roles[role] that this resource belongs to.
* `app.kubernetes.io/role-group` - The name of the xref:stacklet.adoc#role-groups[role group] that a resource belongs to.
* `app.kubernetes.io/managed-by` - Which software manages this resource? This will be the operator, i.e. `kafka-operator`.

also this:

- `stackable.tech/vendor` with value `Stackable`
- `stackable.tech/vendor` with value `Stackable`.

== Labels added by tools

Expand Down Expand Up @@ -55,4 +56,4 @@ The default set of labels includes:

== Further reading

Have a look at https://docs.stackable.tech/home/nightly/contributor/adr/adr031-resource-labels[] if you want to find out about the design decisions for our labels.
Take a look at xref:contributor:adr/ADR031-resource-labels.adoc[] if you want to find out about the design decisions for our labels.
59 changes: 33 additions & 26 deletions modules/concepts/pages/logging.adoc
Original file line number Diff line number Diff line change
@@ -1,48 +1,54 @@
= Logging
:description: A conceptual explanation of the logging architecture of the Stackable Data Platform, and how it is configured.
:keywords: logging, observability, log aggregation, Kubernetes, k8s, Vector, Elasticsearch, OpenSearch
:vector: https://vector.dev/
:vector-sinks: https://vector.dev/docs/reference/configuration/sinks/#
:vector-sidecar: https://vector.dev/docs/setup/deployment/roles/#sidecar
:vector-aggregator: https://vector.dev/docs/setup/deployment/roles/#aggregator
:vector-agg-install: https://vector.dev/docs/setup/installation/package-managers/helm/#aggregator
:vector-source-vector: https://vector.dev/docs/reference/configuration/sources/vector/
:vector-topology-centralized: https://vector.dev/docs/setup/deployment/topologies/#centralized

// Abstract
Logging is important for observability of the platform.
Stackable provides human-readable plaintext logs for each running container, as well as aggregated and persisted logs with identical structure across the whole platform.
Log levels can be set for individual modules and configuration is identical across all products, but custom logging configuration files can be supplied as well.
Log levels can be set for individual modules and configuration is identical in each xref:stacklet.adoc[Stacklet], but custom logging configuration files can be supplied as well.

image::logging_overview.drawio.svg[]

== Motivation

**Aggregate and persist** - The logging on the platform was designed to aggregate logs from all parts of the platform to make it easy to correlate events from different parts.
For this, logs should share the same structure, and should be viewable in a central location.
Logs should also be persisted in a central location, so if a component crashes, the logs are still there to identify the reason.
**Aggregate and persist** - The logging architecture is designed to aggregate logs from all parts of the platform to simplify correlating events from different Stacklets.
For this, logs from different sources are configured to share the same structure, and are collected and made viewable in a central location.
The collected logs are also be persisted centrally, so if a component crashes its logs are still there to investigate the cause of the crash.

**Easy to read on the fly** - At the same time, logs should still be accessible in an easy to read format on the containers, to allow for easy on the fly inspection of each part of the platform.
**Easy to read on the fly** - At the same time logs are kept accessible in an plain format on the containers, supporting easy on the fly inspection of running Stacklets.
The logging configuration also supports setting different thresholds for the logs readable on the container and the aggregated logs.
This way you can get a detailed view of the operations of a component while viewing it on the container, but aggregate logs at a coarser granularity when aggregating across the whole platform.

**Consistent configuration** - Finally, logging should be always configured the same way, no matter which product and which underlying technology is used to produce the logs.
Logging for each product is configured in the ProductCluster resource.
It is still supported to supply custom logging configuration files, these are then product specific.
Logging for each product is configured in the Stacklet resource.
For advanced log configurations, supplying custom product specific log configuration files is also supported.

== Architecture

Below you can see the overall architecture using ZooKeeper as an example.
Stackable uses https://vector.dev/[Vector] for log aggregation and any of the supported https://vector.dev/docs/reference/configuration/sinks/[sinks] can be used to persist the logs.
Stackable uses {vector}[Vector] for log aggregation and any of the supported {vector-sinks}[sinks] can be used to persist the logs.

image::logging_architecture.drawio.svg[]
image::logging_architecture.drawio.svg[An architecture diagram showing the Kubernetes resources involved in the logging stack]

=== Log source

You configure your logging settings in the resource describing the cluster (ZookeeperCluster in this example), seen in the top left in the diagram (see the <<configuration, configuration>> section below).
You configure your logging settings in the Stacklet definition (ZookeeperCluster in this example), seen in the top left in the diagram (see the <<configuration, configuration>> section below).
The operator reads this resource and creates the appropriate log configuration files in the ConfigMap which also holds other product configuration.
The ConfigMap is then mounted into the containers.
The ZooKeeper Pod has three containers: The `prepare` sidecar container, the `zookeeper` container and the `vector` https://vector.dev/docs/setup/deployment/roles/#sidecar[sidecar container].
All logs get written into a shared mounted directory, from which the Vector agent reads them and sends them to the Vector https://vector.dev/docs/setup/deployment/roles/#aggregator[aggregator].
The ZooKeeper Pod has three containers: The `prepare` sidecar container, the `zookeeper` container and the `vector` {vector-sidecar}[sidecar container].
All logs get written into a shared mounted directory, from which the Vector agent reads them and sends them to the Vector {vector-aggregator}[aggregator].

=== Aggregator and sinks

The aggregator is configured to use one or multiple https://vector.dev/docs/reference/configuration/sinks/[sinks] (for example OpenSearch, Elasticsearch), it sends all logs to all sinks.
If a sink is unavailable, it will buffer the log messages.
It is also the single point where the sinks are configured, so the sinks are decoupled from the individual product configurations and only needs to be configured in this single location for the whole platform.
The aggregator is configured to use one or multiple {vector-sinks}[sinks] (for example OpenSearch, Elasticsearch), it sends all logs to all sinks.
If a sink is unavailable, the aggregator buffers the log messages.
It is also the single point where the sinks are configured, so the sinks are decoupled from the Stacklet definitions and only need to be configured in this single location for the whole platform.

[#configuration]
== Configuration
Expand Down Expand Up @@ -109,12 +115,12 @@ A typical use case is that the log level for the console is set to a more detail
**Per container configuration** - A Pod actually consists of multiple containers and you might want different log levels for each of them.
Also the log levels per module are container specific.

Following the Stackable xref::roles-and-role-groups.adoc[roles and role groups] concept, this configuration fragment can be set either at role or role group level.
Following the Stackable xref:stacklet.adoc#roles[roles] and xref::stacklet.adoc#roles[role groups] concept, this configuration fragment can be set either at role or role group level.

=== Configuring the Aggregator

Follow the https://vector.dev/docs/setup/installation/package-managers/helm/#aggregator[installation instructions] for the aggregator.
Configure a https://vector.dev/docs/reference/configuration/sources/vector/[Vector source] at adress `0.0.0.0:6000` and configure sinks and additional settings according to your needs.
Follow the {vector-agg-install}[installation instructions] for the aggregator.
Configure a {vector-source-vector}[Vector source] at adress `0.0.0.0:6000` and configure sinks and additional settings according to your needs.

=== Configuring the aggregator location

Expand All @@ -123,21 +129,22 @@ The field is called `ADDRESS` and the value could be `vector-aggregator:6000` if

== Custom overrides

As with many parts of the Stackable platform, custom overrides are supported as well, by supplying your own logging configuration file. This is then product specific.
As with many parts of the Stackable platform, custom overrides are supported as well, by supplying your own logging configuration file.
This is then product specific.

```yaml
[source,yaml]
----
logging:
enableVectorAgent: false // <1>
containers:
my-container:
custom:
configMap: my-configmap // <2>
```

<1> The vector logging agent is not deployed
<2> A custom logging configuration is loaded from a ConfigMap called `my-configmap`
----
<1> The vector logging agent is not deployed.
<2> A custom logging configuration is loaded from a ConfigMap called `my-configmap`.

== Further reading

To get some hands on experience and see logging in action, try out the xref:demos:logging.adoc[logging demo] or follow the xref:tutorials:logging-vector-aggregator.adoc[logging tutorial].
The Vector documentation contains more information about the https://vector.dev/docs/setup/deployment/topologies/#centralized[deployment topology] and https://vector.dev/docs/reference/configuration/sinks/[sinks].
The Vector documentation contains more information about the {vector-topology-centralized}[deployment topology] and {vector-sinks}[sinks] that can be used.
Loading