Skip to content

Commit 61aea6c

Browse files
Add Stacklet docs page (#613)
* draft * fix typos * overhauled some pages * ~ * ~ * ~ * ~ * ~ * fixed incorrect ref * Update modules/concepts/pages/index.adoc Co-authored-by: Nick <NickLarsenNZ@users.noreply.github.com> * Update modules/concepts/pages/labels.adoc Co-authored-by: Nick <NickLarsenNZ@users.noreply.github.com> * Update modules/concepts/pages/labels.adoc Co-authored-by: Nick <NickLarsenNZ@users.noreply.github.com> * Update modules/concepts/pages/labels.adoc Co-authored-by: Nick <NickLarsenNZ@users.noreply.github.com> * Update modules/concepts/pages/logging.adoc Co-authored-by: Nick <NickLarsenNZ@users.noreply.github.com> * Update modules/concepts/pages/opa.adoc Co-authored-by: Nick <NickLarsenNZ@users.noreply.github.com> * Update modules/concepts/pages/overrides.adoc Co-authored-by: Nick <NickLarsenNZ@users.noreply.github.com> * Update modules/concepts/pages/overrides.adoc Co-authored-by: Nick <NickLarsenNZ@users.noreply.github.com> * Update modules/concepts/pages/service-exposition.adoc Co-authored-by: Nick <NickLarsenNZ@users.noreply.github.com> * Update modules/concepts/pages/service-exposition.adoc Co-authored-by: Nick <NickLarsenNZ@users.noreply.github.com> * Update modules/concepts/pages/resources.adoc Co-authored-by: Nick <NickLarsenNZ@users.noreply.github.com> * Update modules/concepts/pages/resources.adoc Co-authored-by: Nick <NickLarsenNZ@users.noreply.github.com> * Update modules/concepts/pages/overrides.adoc Co-authored-by: Nick <NickLarsenNZ@users.noreply.github.com> * Update modules/concepts/pages/overrides.adoc Co-authored-by: Nick <NickLarsenNZ@users.noreply.github.com> * Update modules/concepts/pages/overrides.adoc Co-authored-by: Nick <NickLarsenNZ@users.noreply.github.com> * Update modules/concepts/pages/overview.adoc Co-authored-by: Nick <NickLarsenNZ@users.noreply.github.com> * Update modules/concepts/pages/overview.adoc Co-authored-by: Nick <NickLarsenNZ@users.noreply.github.com> * Update modules/concepts/pages/overview.adoc Co-authored-by: Nick <NickLarsenNZ@users.noreply.github.com> * Update modules/concepts/pages/stacklet.adoc Co-authored-by: Nick <NickLarsenNZ@users.noreply.github.com> * Update modules/concepts/pages/stacklet.adoc Co-authored-by: Nick <NickLarsenNZ@users.noreply.github.com> * Update modules/concepts/pages/stacklet.adoc Co-authored-by: Nick <NickLarsenNZ@users.noreply.github.com> * Update modules/concepts/pages/stacklet.adoc Co-authored-by: Nick <NickLarsenNZ@users.noreply.github.com> * Update modules/concepts/pages/stackable_resource_requests.adoc Co-authored-by: Nick <NickLarsenNZ@users.noreply.github.com> * Update modules/concepts/pages/stackable_resource_requests.adoc Co-authored-by: Nick <NickLarsenNZ@users.noreply.github.com> * Update modules/concepts/pages/service-exposition.adoc Co-authored-by: Nick <NickLarsenNZ@users.noreply.github.com> * Update modules/concepts/pages/service-exposition.adoc Co-authored-by: Nick <NickLarsenNZ@users.noreply.github.com> * Update modules/concepts/pages/service-exposition.adoc Co-authored-by: Nick <NickLarsenNZ@users.noreply.github.com> * clarify role text * change stacklet text * Update modules/concepts/pages/stacklet.adoc Co-authored-by: Nick <NickLarsenNZ@users.noreply.github.com> * Update modules/concepts/pages/stackable_resource_requests.adoc Co-authored-by: Nick <NickLarsenNZ@users.noreply.github.com> * change CPU resource text * change process to component * remove confusing sentence * add suggestion * Update modules/concepts/pages/overview.adoc Co-authored-by: Nick <NickLarsenNZ@users.noreply.github.com> --------- Co-authored-by: Nick <NickLarsenNZ@users.noreply.github.com>
1 parent 2e19fb3 commit 61aea6c

19 files changed

+380
-265
lines changed

modules/concepts/nav.adoc

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -1,7 +1,7 @@
11
* xref:concepts:index.adoc[]
2-
** xref:overview.adoc[Overview]
2+
** xref:overview.adoc[Platform overview]
3+
** xref:stacklet.adoc[]
34
** Common configuration mechanisms
4-
*** xref:roles-and-role-groups.adoc[]
55
*** xref:product_image_selection.adoc[]
66
*** xref:overrides.adoc[Advanced: overrides]
77
** Resources

modules/concepts/pages/authentication.adoc

Lines changed: 11 additions & 10 deletions
Original file line numberDiff line numberDiff line change
@@ -1,7 +1,8 @@
11
= Authentication
2+
:keycloak: https://www.keycloak.org/
23

34
The Stackable Platform uses the AuthenticationClass as a central mechanism to handle user authentication across supported products.
4-
The authentication mechanism needs to be configured only in the AuthenticationClass which is then referenced in the product.
5+
The authentication mechanism needs to be configured only in the AuthenticationClass which is then referenced in the xref:stacklet.adoc[Stacklet] definition.
56
Multiple different authentication providers are supported.
67

78
[#authenticationclass]
@@ -40,7 +41,7 @@ NOTE: Learn more in the xref:tutorials:authentication_with_openldap.adoc[OpenLDA
4041
[#OIDC]
4142
=== OpenID Connect
4243

43-
An OIDC provider like https://www.keycloak.org/[Keycloak {external-link-icon}^] could be configured as follows:
44+
An OIDC provider like {keycloak}[Keycloak] could be configured as follows:
4445

4546
[source,yaml]
4647
----
@@ -60,10 +61,10 @@ NOTE: Get a full overview of all the properties in the {crd-docs}/authentication
6061
=== TLS
6162
The `TLS` provider configures a product to authenticate users using TLS certificates.
6263
When establishing a connection the client will first validate the certificate of the server.
63-
This step is not influenced by this `AuthenticationClass`, it only affects the next step:
64+
This step is not influenced by this AuthenticationClass, it only affects the next step:
6465
Afterwards the server checks the validity of the certificate the client has provided.
6566
This includes the usual checks - such as checking that it hasn't expired and matches the hostname of the client.
66-
Additionally the client certificate needs to be signed with the `ca` certificate, which is provided by the `SecretClass` specified in `clientCertSecretClass`.
67+
Additionally the client certificate needs to be signed with the `ca` certificate, which is provided by the SecretClass specified in `clientCertSecretClass`.
6768

6869
A sample TLS provider looks as follows:
6970

@@ -77,25 +78,25 @@ include::example$authenticationclass-tls.yaml[]
7778
The `static` provider is used to represent a simple - static - set of users.
7879
Users are identified by a username and a password.
7980

80-
First, the `AuthenticationClass` needs to be defined as follows:
81+
First, the AuthenticationClass needs to be defined as follows:
8182

8283
[source,yaml]
8384
----
8485
include::example$authenticationclass-static-authenticationclass.yaml[]
8586
----
86-
<1> The name of the `Secret` containing the credentials
87+
<1> The name of the Secret containing the credentials
8788

88-
Afterwards the referenced `Secret` needs to be created:
89+
Afterwards the referenced Secret needs to be created:
8990

9091
[source,yaml]
9192
----
9293
include::example$authenticationclass-static-secret.yaml[]
9394
----
94-
<1> The name of the `Secret`, which needs to match the `Secret` name specified in the `AuthenticationClass` above
95-
<2> The namespace of the `Secret`. The `Secret` needs to be in the same namespace as the product that tries to use the static `AuthenticationClass`
95+
<1> The name of the Secret, which needs to match the Secret name specified in the AuthenticationClass above
96+
<2> The namespace of the Secret. The Secret needs to be in the same namespace as the product that tries to use the static AuthenticationClass
9697

9798
[#further-reading]
98-
== Further Reading
99+
== Further reading
99100

100101
* xref:tutorials:authentication_with_openldap.adoc[] tutorial
101102
* {crd-docs}/authentication.stackable.tech/authenticationclass/v1alpha1/[AuthenticationClass CRD reference]

modules/concepts/pages/index.adoc

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -6,7 +6,7 @@ The xref:overview.adoc[Platform overview] is a good starting point to understand
66

77
== General configuration mechanisms
88

9-
Learn about xref:roles-and-role-groups.adoc[], how product image selection works.
9+
Learn what a xref:stacklet.adoc[Stacklet] is, what roles and role groups are, and how product image selection works.
1010
There is also the common xref:overrides.adoc[override] mechanism for configuration settings, although this tool should be used with care!
1111

1212
== Resources

modules/concepts/pages/labels.adoc

Lines changed: 12 additions & 11 deletions
Original file line numberDiff line numberDiff line change
@@ -1,25 +1,26 @@
11
= Labels
2+
:common-labels: https://kubernetes.io/docs/concepts/overview/working-with-objects/common-labels/
23

34
Labels are key/value pairs in the metadata of Kubernetes objects that add identifying information to the object.
45
They do not have direct semantic implications but can be used to organize and manage resources.
5-
The `stackablectl` tool, the cockpit and the Helm Charts add labels to the resources that are part of a Stacklet, and the operators add labels to the resources they create.
6+
The xref:management:stackablectl:index.adoc[`stackablectl`] tool, the cockpit, and the Helm Charts add labels to the resources that are part of a xref:stacklet.adoc[Stacklet], and the operators add labels to the resources they create.
67

78
== Resource labels added by the operators
89

9-
Every resource created by a Stackable operator is labelled with a common set of labels.
10-
Some of these labels are https://kubernetes.io/docs/concepts/overview/working-with-objects/common-labels/[recommended] to use by the Kubernetes documentation.
10+
Every resource created by a Stackable operator has a common set of labels applied.
11+
Some of these labels are {common-labels}[recommended] to use by the Kubernetes documentation.
1112
The following labels are added to resources created by our operators:
1213

13-
- `app.kubernetes.io/name` - The name of the product, i.e. `druid` or `zookeeper`.
14-
- `app.kubernetes.io/version` - The version of the product and Stackable version, i.e. `3.3.4-stackable23.11.0`
15-
- `app.kubernetes.io/instance` - The name of the Stacklet.
16-
- `app.kubernetes.io/component` - This is the xref:concepts:roles-and-role-groups.adoc[role] that this resource belongs to.
17-
- `app.kubernetes.io/role-group` - The name of the xref:concepts:roles-and-role-groups.adoc[role group] that a resource belongs to.
18-
- `app.kubernetes.io/managed-by` - Which software manages this resource? This will be the operator, i.e. `airflow-operator`.
14+
* `app.kubernetes.io/name` - The name of the product, i.e. `druid` or `zookeeper`.
15+
* `app.kubernetes.io/version` - The version of the product and Stackable version, i.e. `3.3.4-stackable24.3.0`
16+
* `app.kubernetes.io/instance` - The name of the Stacklet.
17+
* `app.kubernetes.io/component` - This is the xref:stacklet.adoc#roles[role] that this resource belongs to.
18+
* `app.kubernetes.io/role-group` - The name of the xref:stacklet.adoc#role-groups[role group] that a resource belongs to.
19+
* `app.kubernetes.io/managed-by` - Which software manages this resource? This will be the operator, i.e. `kafka-operator`.
1920

2021
also this:
2122

22-
- `stackable.tech/vendor` with value `Stackable`
23+
- `stackable.tech/vendor` with value `Stackable`.
2324

2425
== Labels added by tools
2526

@@ -55,4 +56,4 @@ The default set of labels includes:
5556

5657
== Further reading
5758

58-
Have a look at https://docs.stackable.tech/home/nightly/contributor/adr/adr031-resource-labels[] if you want to find out about the design decisions for our labels.
59+
Take a look at xref:contributor:adr/ADR031-resource-labels.adoc[] if you want to find out about the design decisions for our labels.

modules/concepts/pages/logging.adoc

Lines changed: 33 additions & 26 deletions
Original file line numberDiff line numberDiff line change
@@ -1,48 +1,54 @@
11
= Logging
22
:description: A conceptual explanation of the logging architecture of the Stackable Data Platform, and how it is configured.
33
:keywords: logging, observability, log aggregation, Kubernetes, k8s, Vector, Elasticsearch, OpenSearch
4+
:vector: https://vector.dev/
5+
:vector-sinks: https://vector.dev/docs/reference/configuration/sinks/#
6+
:vector-sidecar: https://vector.dev/docs/setup/deployment/roles/#sidecar
7+
:vector-aggregator: https://vector.dev/docs/setup/deployment/roles/#aggregator
8+
:vector-agg-install: https://vector.dev/docs/setup/installation/package-managers/helm/#aggregator
9+
:vector-source-vector: https://vector.dev/docs/reference/configuration/sources/vector/
10+
:vector-topology-centralized: https://vector.dev/docs/setup/deployment/topologies/#centralized
411

5-
// Abstract
612
Logging is important for observability of the platform.
713
Stackable provides human-readable plaintext logs for each running container, as well as aggregated and persisted logs with identical structure across the whole platform.
8-
Log levels can be set for individual modules and configuration is identical across all products, but custom logging configuration files can be supplied as well.
14+
Log levels can be set for individual modules and configuration is identical in each xref:stacklet.adoc[Stacklet], but custom logging configuration files can be supplied as well.
915

1016
image::logging_overview.drawio.svg[]
1117

1218
== Motivation
1319

14-
**Aggregate and persist** - The logging on the platform was designed to aggregate logs from all parts of the platform to make it easy to correlate events from different parts.
15-
For this, logs should share the same structure, and should be viewable in a central location.
16-
Logs should also be persisted in a central location, so if a component crashes, the logs are still there to identify the reason.
20+
**Aggregate and persist** - The logging architecture is designed to aggregate logs from all parts of the platform to simplify correlating events from different Stacklets.
21+
For this, logs from different sources are configured to share the same structure, and are collected and made viewable in a central location.
22+
The collected logs are also be persisted centrally, so if a component crashes its logs are still there to investigate the cause of the crash.
1723

18-
**Easy to read on the fly** - At the same time, logs should still be accessible in an easy to read format on the containers, to allow for easy on the fly inspection of each part of the platform.
24+
**Easy to read on the fly** - At the same time logs are kept accessible in an plain format on the containers, supporting easy on the fly inspection of running Stacklets.
1925
The logging configuration also supports setting different thresholds for the logs readable on the container and the aggregated logs.
2026
This way you can get a detailed view of the operations of a component while viewing it on the container, but aggregate logs at a coarser granularity when aggregating across the whole platform.
2127

2228
**Consistent configuration** - Finally, logging should be always configured the same way, no matter which product and which underlying technology is used to produce the logs.
23-
Logging for each product is configured in the ProductCluster resource.
24-
It is still supported to supply custom logging configuration files, these are then product specific.
29+
Logging for each product is configured in the Stacklet resource.
30+
For advanced log configurations, supplying custom product specific log configuration files is also supported.
2531

2632
== Architecture
2733

2834
Below you can see the overall architecture using ZooKeeper as an example.
29-
Stackable uses https://vector.dev/[Vector] for log aggregation and any of the supported https://vector.dev/docs/reference/configuration/sinks/[sinks] can be used to persist the logs.
35+
Stackable uses {vector}[Vector] for log aggregation and any of the supported {vector-sinks}[sinks] can be used to persist the logs.
3036

31-
image::logging_architecture.drawio.svg[]
37+
image::logging_architecture.drawio.svg[An architecture diagram showing the Kubernetes resources involved in the logging stack]
3238

3339
=== Log source
3440

35-
You configure your logging settings in the resource describing the cluster (ZookeeperCluster in this example), seen in the top left in the diagram (see the <<configuration, configuration>> section below).
41+
You configure your logging settings in the Stacklet definition (ZookeeperCluster in this example), seen in the top left in the diagram (see the <<configuration, configuration>> section below).
3642
The operator reads this resource and creates the appropriate log configuration files in the ConfigMap which also holds other product configuration.
3743
The ConfigMap is then mounted into the containers.
38-
The ZooKeeper Pod has three containers: The `prepare` sidecar container, the `zookeeper` container and the `vector` https://vector.dev/docs/setup/deployment/roles/#sidecar[sidecar container].
39-
All logs get written into a shared mounted directory, from which the Vector agent reads them and sends them to the Vector https://vector.dev/docs/setup/deployment/roles/#aggregator[aggregator].
44+
The ZooKeeper Pod has three containers: The `prepare` sidecar container, the `zookeeper` container and the `vector` {vector-sidecar}[sidecar container].
45+
All logs get written into a shared mounted directory, from which the Vector agent reads them and sends them to the Vector {vector-aggregator}[aggregator].
4046

4147
=== Aggregator and sinks
4248

43-
The aggregator is configured to use one or multiple https://vector.dev/docs/reference/configuration/sinks/[sinks] (for example OpenSearch, Elasticsearch), it sends all logs to all sinks.
44-
If a sink is unavailable, it will buffer the log messages.
45-
It is also the single point where the sinks are configured, so the sinks are decoupled from the individual product configurations and only needs to be configured in this single location for the whole platform.
49+
The aggregator is configured to use one or multiple {vector-sinks}[sinks] (for example OpenSearch, Elasticsearch), it sends all logs to all sinks.
50+
If a sink is unavailable, the aggregator buffers the log messages.
51+
It is also the single point where the sinks are configured, so the sinks are decoupled from the Stacklet definitions and only need to be configured in this single location for the whole platform.
4652

4753
[#configuration]
4854
== Configuration
@@ -109,12 +115,12 @@ A typical use case is that the log level for the console is set to a more detail
109115
**Per container configuration** - A Pod actually consists of multiple containers and you might want different log levels for each of them.
110116
Also the log levels per module are container specific.
111117

112-
Following the Stackable xref::roles-and-role-groups.adoc[roles and role groups] concept, this configuration fragment can be set either at role or role group level.
118+
Following the Stackable xref:stacklet.adoc#roles[roles] and xref::stacklet.adoc#roles[role groups] concept, this configuration fragment can be set either at role or role group level.
113119

114120
=== Configuring the Aggregator
115121

116-
Follow the https://vector.dev/docs/setup/installation/package-managers/helm/#aggregator[installation instructions] for the aggregator.
117-
Configure a https://vector.dev/docs/reference/configuration/sources/vector/[Vector source] at adress `0.0.0.0:6000` and configure sinks and additional settings according to your needs.
122+
Follow the {vector-agg-install}[installation instructions] for the aggregator.
123+
Configure a {vector-source-vector}[Vector source] at adress `0.0.0.0:6000` and configure sinks and additional settings according to your needs.
118124

119125
=== Configuring the aggregator location
120126

@@ -123,21 +129,22 @@ The field is called `ADDRESS` and the value could be `vector-aggregator:6000` if
123129

124130
== Custom overrides
125131

126-
As with many parts of the Stackable platform, custom overrides are supported as well, by supplying your own logging configuration file. This is then product specific.
132+
As with many parts of the Stackable platform, custom overrides are supported as well, by supplying your own logging configuration file.
133+
This is then product specific.
127134

128-
```yaml
135+
[source,yaml]
136+
----
129137
logging:
130138
enableVectorAgent: false // <1>
131139
containers:
132140
my-container:
133141
custom:
134142
configMap: my-configmap // <2>
135-
```
136-
137-
<1> The vector logging agent is not deployed
138-
<2> A custom logging configuration is loaded from a ConfigMap called `my-configmap`
143+
----
144+
<1> The vector logging agent is not deployed.
145+
<2> A custom logging configuration is loaded from a ConfigMap called `my-configmap`.
139146

140147
== Further reading
141148

142149
To get some hands on experience and see logging in action, try out the xref:demos:logging.adoc[logging demo] or follow the xref:tutorials:logging-vector-aggregator.adoc[logging tutorial].
143-
The Vector documentation contains more information about the https://vector.dev/docs/setup/deployment/topologies/#centralized[deployment topology] and https://vector.dev/docs/reference/configuration/sinks/[sinks].
150+
The Vector documentation contains more information about the {vector-topology-centralized}[deployment topology] and {vector-sinks}[sinks] that can be used.

0 commit comments

Comments
 (0)