From 8e6a924dc2abad9f54627e433ba4ccc020116684 Mon Sep 17 00:00:00 2001 From: ahardin-rh Date: Thu, 18 Aug 2016 17:49:24 -0400 Subject: [PATCH] Follow-up edits to PR#2538 --- install_config/aggregate_logging.adoc | 450 ++++++++++++++------------ 1 file changed, 240 insertions(+), 210 deletions(-) diff --git a/install_config/aggregate_logging.adoc b/install_config/aggregate_logging.adoc index b6f904602d81..97d1a534e63f 100644 --- a/install_config/aggregate_logging.adoc +++ b/install_config/aggregate_logging.adoc @@ -104,12 +104,12 @@ $ oc new-app logging-deployer-account-template + [NOTE] ==== -If you have deployed logging previously, for example in a different project, then -it is normal for the cluster roles to fail to be created as they already exist. +If you deployed logging previously, for example in a different project, then +it is normal for the cluster roles to fail to be created because they already exist. ==== . Enable the deployer service account to create an OAuthClient (normally -a cluster-admin privilege) for Kibana to use later when authenticating +a cluster administrator privilege) for Kibana to use later when authenticating against the master. + ==== @@ -117,7 +117,7 @@ against the master. $ oadm policy add-cluster-role-to-user oauth-editor \ system:serviceaccount:logging:logging-deployer <1> ---- -<1> Use the project you created earlier (e.g., *logging*) when specifying +<1> Use the project you created earlier (for example, *logging*) when specifying this service account. ==== @@ -132,7 +132,7 @@ $ oadm policy add-scc-to-user privileged \ $ oadm policy add-cluster-role-to-user cluster-reader \ system:serviceaccount:logging:aggregated-logging-fluentd <1> ---- -<1> Use the project you created earlier (e.g., *logging*) when specifying +<1> Use the project you created earlier (for example, *logging*) when specifying this service account. ==== @@ -140,19 +140,21 @@ this service account. == Specifying Deployer Parameters Parameters for the EFK deployment may be specified in the -form of a xref:../dev_guide/configmaps.adoc[configmap], +form of a xref:../dev_guide/configmaps.adoc#dev-guide-configmaps[ConfigMap], a xref:../dev_guide/secrets.adoc#dev-guide-secrets[secret], or template parameters (which are passed to the deployer in environment variables). The deployer looks for each value first in a -*logging-deployer* configmap, then a *logging-deployer* secret, then as +*logging-deployer* ConfigMap, then a *logging-deployer* secret, then as an environment variable. Any or all may be omitted if not needed. -Read about the available parameters below. Typically you should at -least specify the hostname at which Kibana should be exposed to client -browsers, and also the master URL where client browsers will be directed -for authenticating to OpenShift. +The available parameters are outlined below. Typically, you should at +least specify the host name at which Kibana should be exposed to client +browsers, and also the master URL where client browsers will be directed to +for authenticating to {product-title}. -. Create a xref:../dev_guide/configmaps.adoc[configmap] to provide most deployer parameters. An invocation supplying the most important parameters might be: +. Create a xref:../dev_guide/configmaps.adoc#dev-guide-configmaps[ConfigMap] to +provide most deployer parameters. An invocation supplying the most important +parameters might be: + ---- $ oc create configmap logging-deployer \ @@ -162,17 +164,17 @@ $ oc create configmap logging-deployer \ --from-literal es-instance-ram=8G ---- + -It is also easy to edit ConfigMap YAML after creating it: +. Edit the ConfigMap YAML file after creating it: + ---- $ oc edit configmap logging-deployer ---- + -These and other parameters are available (you should read the +Other parameters are available. Read the xref:aggregate_logging.adoc#aggregated-elasticsearch[ElasticSearch -section] below before choosing ElasticSearch parameters for the deployer, +section] before choosing ElasticSearch parameters for the deployer, and the xref:aggregate_logging.adoc#aggregated-fluentd[Fluentd section] -for some possible parameters): +for some possible parameters: + [cols="3,7",options="header"] |=== @@ -180,13 +182,13 @@ for some possible parameters): |Description |*_kibana-hostname_* -| The external host name for web clients to reach Kibana. +|The external host name for web clients to reach Kibana. |*_public-master-url_* -|The external URL for the master. For OAuth purposes. +|The external URL for the master; used for OAuth purposes. |*_es-cluster-size_* (default: 1) -| The number of instances of +|The number of instances of Elasticsearch to deploy. Redundancy requires at least three, and more can be used for scaling. @@ -195,18 +197,25 @@ used for scaling. must be at least 512M. Possible suffixes are G,g,M,m. |*_es-pvc-prefix_* (default: *logging-es-*) -| Prefix for the names of PersistentVolumeClaims to be used as storage for Elasticsearch instances; a number will be appended per instance (e.g. *logging-es-1*). If they don't already exist, they will be created with size *_es-pvc-size_*. +|Prefix for the names of persistent volume claims to be used as storage for +Elasticsearch instances; a number will be appended per instance (for example, +*logging-es-1*). If they do not already exist, they will be created with size +*_es-pvc-size_*. |*_es-pvc-size_* -| Size of the PersistentVolumeClaim to create per ElasticSearch instance, e.g. 100G. If omitted, no PVCs will be created and ephemeral volumes are used instead. +|Size of the persistent volume claim to create per ElasticSearch instance, 100G, +for example. If omitted, no PVCs are created and ephemeral volumes are used instead. |*_es-pvc-dynamic_* -| Set to true to have created PersistentVolumeClaims annotated such that their backing storage can be dynamically provisioned (if that is available for your cluster). +|Set to `*true*` to have created persistent volume claims annotated so that their +backing storage can be dynamically provisioned (if that is available for your +cluster). |*_storage-group_* -| Number of a supplemental group ID for access to Elasticsearch storage volumes; backing volumes should allow access by this group ID (defaults to 65534). +|Number of a supplemental group ID for access to Elasticsearch storage volumes; +backing volumes should allow access by this group ID (defaults to 65534). -|*_fluentd-nodeselector_* (default: *logging-infra-fluentd=true*) +a|*_fluentd-nodeselector_* (default: *logging-infra-fluentd=true*) | A node selector that specifies which nodes are eligible targets for deploying Fluentd instances. All nodes where Fluentd should run (typically, all) must have this label @@ -238,10 +247,11 @@ are distinguishable by the *-ops* included in their names and have parallel deployment options listed below. |*_kibana-ops-hostname, es-ops-instance-ram, es-ops-pvc-size, es-ops-pvc-prefix, es-ops-cluster-size, es-ops-nodeselector, kibana-ops-nodeselector, curator-ops-nodeselector_* -| Parallel parameters for the ops log cluster. +|Parallel parameters for the ops log cluster. |*_image-pull-secret_* -| Specify the name of an existing pull secret to be used for pulling component images from an authenticated registry. +|Specify the name of an existing pull secret to be used for pulling component +images from an authenticated registry. |=== . Create a xref:../dev_guide/secrets.adoc#dev-guide-secrets[secret] to provide security-related files to the deployer. The contents of the secret are optional, and will be randomly generated if not supplied. @@ -292,7 +302,8 @@ xref:../dev_guide/templates.adoc#dev-guide-templates[template] to create a deployer pod that reads the deployment parameters and manages the deployment. -Run the deployer, optionally specifying parameters (described in the table below), for example: +Run the deployer, optionally specifying parameters (described in the table +below), for example: ==== Without template parameters: @@ -348,54 +359,58 @@ endif::openshift-enterprise[] |=== Running the deployer creates a deployer pod and prints its name. Wait until the -pod is running. This can take up to a few minutes for {product-title} to retrieve the deployer image -from the registry. You can watch its process with: +pod is running. This can take up to a few minutes for {product-title} to +retrieve the deployer image from the registry. Watch its process with: ---- $ oc get pod/ -w ---- -It should eventually enter Running status and end in Complete status. -If it seems to be taking too long to start, you can retrieve more details about -the pod and any associated events with: +It will eventually enter *Running* status and end in *Complete* status. If takes +too long to start, retrieve more details about the pod and any associated events +with: ---- $ oc describe pod/ ---- -You can also check the logs if the deployment does not complete successfully: +Check the logs if the deployment does not complete successfully: ---- $ oc logs -f ---- -Once deployment completes successfully, you will probably need to +Once deployment completes successfully, you may need to xref:aggregate_logging.adoc#aggregated-fluentd[label the nodes for Fluentd to deploy on], and may have other adjustments to make to the -deployed components. These are described in the next section. +deployed components. These tasks are described in the next section. +[[aggregate-logging-understanding-the-deployment]] == Understanding and Adjusting the Deployment +This section describes adjustments that you can make to deployed components. [[aggregated-ops]] === Ops Cluster [NOTE] ==== -The logs for the *default*, *openshift*, and *openshift-infra* projects are automatically aggregated and grouped into the *.operations* item in the Kibana interface. +The logs for the *default*, *openshift*, and *openshift-infra* projects are +automatically aggregated and grouped into the *.operations* item in the Kibana +interface. -The project where you have deployed the EFK stack (*logging*, as documented here) is _not_ aggregated into *.operations* and is found under its ID. +The project where you have deployed the EFK stack (*logging*, as documented +here) is _not_ aggregated into *.operations* and is found under its ID. ==== -If you set `*enable-ops-cluster*` to `true` for the deployer, Fluentd -is configured to split logs between the main ElasticSearch cluster -and another cluster reserved for operations logs (which are defined -as node system logs and the projects `default`, `openshift`, and -`openshift-infra`). Thus a separate Elasticsearch cluster, a separate -Kibana, and a separate Curator are deployed to index, access, and manage -operations logs. These deployments are set apart with names that include -`-ops`. Keep these separate deployments in mind if you have enabled this -option. Most of the following discussion also applies to the operations -cluster if present, just with the names changed to include `-ops`. +If you set `*enable-ops-cluster*` to *true* for the deployer, Fluentd is +configured to split logs between the main ElasticSearch cluster and another +cluster reserved for operations logs (which are defined as node system logs and +the projects *default*, *openshift*, and *openshift-infra*). Therefore, a +separate Elasticsearch cluster, a separate Kibana, and a separate Curator are +deployed to index, access, and manage operations logs. These deployments are set +apart with names that include `-ops`. Keep these separate deployments in mind if +you enabled this option. Most of the following discussion also applies to the +operations cluster if present, just with the names changed to include `-ops`. [[aggregated-elasticsearch]] === Elasticsearch @@ -409,7 +424,7 @@ replica of Elasticsearch has its own deployment configuration. It is possible to scale your cluster up after creation by adding more deployments from a template; however, scaling up (or down) requires the correct procedure and an awareness of clustering parameters (to be -described in a separate section). It is best if you can indicate the +described in a separate section). It is best to indicate the desired scale at first deployment. Refer to @@ -430,39 +445,40 @@ $ oc get dc --selector logging-infra=elasticsearch [[logging-node-selector]] *Node Selector* -Because Elasticsearch can use a lot of resources, all members of a -cluster should have low latency network connections to each other -and to any remote storage. Ensure this by directing the instances to -dedicated nodes, or a dedicated region within your cluster, using a -xref:../admin_guide/managing_projects.adoc#using-node-selectors[node -selector]. +Because Elasticsearch can use a lot of resources, all members of a cluster +should have low latency network connections to each other and to any remote +storage. Ensure this by directing the instances to dedicated nodes, or a +dedicated region within your cluster, using a +xref:../admin_guide/managing_projects.adoc#using-node-selectors[node selector]. -To configure a node selector, specify the `*es-nodeselector*` -configuration option at deployment. This applies to all Elasticsearch -deployments; if you need to individualize the node selectors, you must -manually edit each deployment configuration after deployment. +To configure a node selector, specify the `*es-nodeselector*` configuration +option at deployment. This applies to all Elasticsearch deployments; if you need +to individualize the node selectors, you must manually edit each deployment +configuration after deployment. [[aggregated-logging-persistent-storage]] *Persistent Elasticsearch Storage* -By default, the deployer creates an ephemeral deployment in which all of a pod's data is -lost upon restart. For production usage, specify a persistent storage volume for each -Elasticsearch deployment configuration. You may either create the necessary -xref:../architecture/additional_concepts/storage.adoc#persistent-volume-claims[PersistentVolumeClaims] -before deploying or have them created for you. The PVCs must be named based on -the `*es-pvc-prefix*` setting, which defaults to `logging-es-`; each PVC name will have -a sequence number added to it, so `logging-es-1`, `logging-es-2`, etc. If a PVC needed for -the deployment exists already, it will be used; if not, and `*es-pvc-size*` has -been specified, it will be created with a request for that size. +By default, the deployer creates an ephemeral deployment in which all of a pod's +data is lost upon restart. For production usage, specify a persistent storage +volume for each Elasticsearch deployment configuration. You can create +the necessary +xref:../architecture/additional_concepts/storage.adoc#persistent-volume-claims[persistent +volume claims] before deploying or have them created for you. The PVCs must be +named based on the `*es-pvc-prefix*` setting, which defaults to `logging-es-`; +each PVC name will have a sequence number added to it, so `logging-es-1`, +`logging-es-2`, and so on. If a PVC needed for the deployment exists already, it +is used; if not, and `*es-pvc-size*` has been specified, it is created with a +request for that size. [WARNING] ==== -Using NFS storage as a volume or a PersistentVolume (or via NAS -such as Gluster) is not supported for Elasticsearch storage, as Lucene -relies on filesystem behavior that NFS does not supply. Data corruption -and other problems can occur. If NFS storage is a requirement, you can -allocate a large file on that storage to serve as a storage device and -treat it as a local mount on each host. For example: +Using NFS storage as a volume or a persistent volume (or via NAS such as +Gluster) is not supported for Elasticsearch storage, as Lucene relies on file +system behavior that NFS does not supply. Data corruption and other problems can +occur. If NFS storage is a requirement, you can allocate a large file on that +storage to serve as a storage device and treat it as a local mount on each host. +For example: ---- $ truncate -s 1T /nfs/storage/elasticsearch-1 @@ -471,9 +487,9 @@ $ mount -o loop /nfs/storage/elasticsearch-1 /usr/local/es-storage $ chown 1000:1000 /usr/local/es-storage ---- -Then, use *_/usr/local/es-storage_* as a host-mount as -described below. Performance under this solution is significantly -worse than using actual local drives. +Then, use *_/usr/local/es-storage_* as a host-mount as described below. +Performance under this solution is significantly worse than using actual local +drives. ==== It is possible to use a local disk volume (if available) on each @@ -488,7 +504,7 @@ local volume: $ oadm policy add-scc-to-user privileged \ system:serviceaccount:logging:aggregated-logging-elasticsearch <1> ---- -<1> Use the project you created earlier (e.g., *logging*) when specifying +<1> Use the project you created earlier (for example, *logging*) when specifying this service account. ==== @@ -503,13 +519,13 @@ $ for dc in $(oc get deploymentconfig --selector logging-infra=elasticsearch -o done ---- -. The Elasticsearch replicas must be located on the correct nodes to use -the local storage, and should not move around even if those nodes are -taken down for a period of time. This requires giving each Elasticsearch -replica a node selector that is unique to a node where an administrator -has allocated storage for it. To configure a node selector, edit each -Elasticsearch deployment configuration and add or edit the *nodeSelector* section to specify -a unique label that you have applied for each desired node: +. The Elasticsearch replicas must be located on the correct nodes to use the local +storage, and should not move around even if those nodes are taken down for a +period of time. This requires giving each Elasticsearch replica a node selector +that is unique to a node where an administrator has allocated storage for it. To +configure a node selector, edit each Elasticsearch deployment configuration and +add or edit the *nodeSelector* section to specify a unique label that you have +applied for each desired node: + ==== ---- @@ -521,7 +537,9 @@ spec: nodeSelector: logging-es-node: "1" <1> ---- -<1> This label should uniquely identify a replica with a single node that bears that label, in this case `*logging-es-node=1*`. Use the `oc label` command to apply labels to nodes as needed. +<1> This label should uniquely identify a replica with a single node that bears that +label, in this case `*logging-es-node=1*`. Use the `oc label` command to apply +labels to nodes as needed. To automate applying the node selector you can instead use the `oc patch` command: @@ -548,10 +566,11 @@ $ for dc in $(oc get deploymentconfig --selector logging-infra=elasticsearch -o *Changing the Scale of Elasticsearch* If you need to scale up the number of Elasticsearch instances your cluster uses, -it is not as simple as scaling up an Elasticsearch deployment configuration. This -is due to the nature of persistent volumes and how Elasticsearch is configured -to store its data and recover the cluster. Instead, scaling up requires creating a deployment -configuration for each Elasticsearch cluster node. +it is not as simple as scaling up an Elasticsearch deployment configuration. +This is due to the nature of persistent volumes and how Elasticsearch is +configured to store its data and recover the cluster. Instead, scaling up +requires creating a deployment configuration for each Elasticsearch cluster +node. By far the simplest way to change the scale of Elasticsearch is to reinstall the whole deployment. Assuming you have supplied persistent @@ -565,21 +584,21 @@ $ oc edit configmap logging-deployer $ oc new-app logging-deployer-template --param MODE=reinstall ---- -If you previously deployed using template parameters rather than a configmap, -this would be a good time to create a configmap instead for future deployer execution. +If you previously deployed using template parameters rather than a ConfigMap, +this would be a good time to create a ConfigMap instead for future deployer +execution. If you do not wish to reinstall, for instance because you have made -customizations that you would like to preserve, then it is possible to -add new Elasticsearch deployment configurations to the cluster using -a template supplied by the deployer. This requires a more complicated -procedure however. +customizations that you would like to preserve, then it is possible to add new +Elasticsearch deployment configurations to the cluster using a template supplied +by the deployer. This requires a more complicated procedure however. During installation, the deployer xref:../install_config/imagestreams_templates.adoc#install-config-imagestreams-templates[creates templates] with the Elasticsearch configurations provided to it: -`logging-es-template` (and `logging-es-ops-template` if the deployer was -run with `*ENABLE_OPS_CLUSTER=true*`). You can use these for scaling, but -you need to adjust the size-related parameters in the templates: +`logging-es-template` (and `logging-es-ops-template` if the deployer was run +with `*ENABLE_OPS_CLUSTER=true*`). You can use these for scaling, but you need +to adjust the size-related parameters in the templates: [cols="3,7",options="header"] |=== @@ -599,13 +618,12 @@ By default, the same as the cluster size. |=== The node quorum and recovery settings in the template were set based on the -`*es-[ops-]cluster-size*` value initially provided to the deployer. Since the cluster size is -changing, those values need to be overridden. +`*es-[ops-]cluster-size*` value initially provided to the deployer. Since the +cluster size is changing, those values need to be overridden. -. The existing deployment configurations for -that cluster also need to have the three environment variable values above -updated. To edit each of the configurations for the cluster in series, you -may use the following command: +. The existing deployment configurations for that cluster also need to have the +three environment variable values above updated. To edit each of the +configurations for the cluster in series, you may use the following command: + ==== ---- @@ -618,10 +636,10 @@ they will begin with the correct values. For example, for a cluster of size 5, you would set `*NODE_QUORUM*` to `3`, `*RECOVER_AFTER_NODES*` to `4`, and `*RECOVER_EXPECTED_NODES*` to `5`. -. Create additional deployment configuration(s) by running the following -command against the Elasticsearch cluster you want to to scale up for -(`logging-es-template` or `logging-es-ops-template`), overriding the -parameters as above. +. Create additional deployment configurations by running the following command +against the Elasticsearch cluster you want to to scale up for +(`logging-es-template` or `logging-es-ops-template`), overriding the parameters +as above. + ==== ---- @@ -635,9 +653,9 @@ $ oc new-app logging-es[-ops]-template \ These deployments will be named differently, but all will have the `logging-es` prefix. -. Each new deployment configuration is created without a persistent volume. If you want to -attach a persistent volume to it, after creation you can -use the `oc set volume` command to do so, for example: +. Each new deployment configuration is created without a persistent volume. If you +want to attach a persistent volume to it, after creation you can use the `oc set +volume` command to do so, for example: + ---- $ oc volume dc/logging-es- \ @@ -655,29 +673,36 @@ $ oc scale --replicas=1 dc/logging-es- [[aggregated-fluentd]] === Fluentd -Fluentd is deployed as a DaemonSet that deploys replicas according -to a node label selector (which you can specify with the deployer parameter +Fluentd is deployed as a DaemonSet that deploys replicas according to a node +label selector (which you can specify with the deployer parameter `*fluentd-nodeselector*`; the default is `logging-infra-fluentd`). -Once you have ElasticSearch running as -desired, label the nodes intended for Fluentd deployment to feed -their logs into ES. The example below would label a node named -`node.example.com` using the default Fluentd node selector. - $ oc label node/node.example.com logging-infra-fluentd=true +Once you have ElasticSearch running as desired, label the nodes intended for +Fluentd deployment to feed their logs into ES. The example below would label a +node named `node.example.com` using the default Fluentd node selector: -Alternatively, you can label all nodes with the following: +---- +$ oc label node/node.example.com logging-infra-fluentd=true +---- - $ oc label node --all logging-infra-fluentd=true +Alternatively, you can label all nodes with: -Note: Labeling nodes requires cluster-admin capability. +---- +$ oc label node --all logging-infra-fluentd=true +---- + +[NOTE] +==== +Labeling nodes requires cluster administrator capability. +==== [[fluentd-use-journald]] -*Having Fluentd use the systemd journal as the log source* +*Having Fluentd Use the Systemd Journal as the Log Source* By default, Fluentd reads from *_/var/log/messages_* and -*_/var/log/containers/.log_* for system logs and container logs, respectively. -You can instead use the systemd journal as the log source. There are three -deployer configuration parameters available in the deployer configmap: +*_/var/log/containers/.log_* for system logs and container logs, +respectively. You can instead use the systemd journal as the log source. There +are three deployer configuration parameters available in the deployer ConfigMap: [cols="3,7",options="header"] |=== @@ -685,25 +710,25 @@ deployer configuration parameters available in the deployer configmap: |Description | `*use-journal*` -| The default is empty, which tells the deployer to have Fluentd check which -log driver Docker is using. If docker is using `--log-driver=journald`, Fluentd reads -from the systemd journal, otherwise, it assumes docker is using the +|The default is empty, which tells the deployer to have Fluentd check which log +driver Docker is using. If Docker is using `--log-driver=journald`, Fluentd +reads from the systemd journal, otherwise, it assumes docker is using the `json-file` log driver and reads from the *_/var/log_* file sources. You can -specify the `*use-journal*` option as `true` or `false` to be explicit about which log source to use. Using the -systemd journal requires `docker-1.10` or later, and Docker must be configured to -use `--log-driver=journald`. +specify the `*use-journal*` option as `true` or `false` to be explicit about +which log source to use. Using the systemd journal requires `docker-1.10` or +later, and Docker must be configured to use `--log-driver=journald`. | `*journal-source*` -| The default is empty, so that when using the systemd journal, Fluentd first looks for +|The default is empty, so that when using the systemd journal, Fluentd first looks for *_/var/log/journal_*, and if that is not available, uses *_/run/log/journal_* as the journal source. You can specify `*journal-source*` with an explicit -journal path. For example, if you want fluentd to always read logs +journal path. For example, if you want Fluentd to always read logs from the transient in-memory journal, set `*journal-source*`=*_/run/log/journal_*. | `*journal-read-from-head*` -| If this setting is `false`, Fluentd -starts reading from the end of the journal, ignoring historical logs. If this -setting is `true`, fluentd starts reading logs from the beginning of the journal. +|If this setting is *false*, Fluentd starts reading from the end of the journal, +ignoring historical logs. If this setting is *true*, Fluentd starts reading +logs from the beginning of the journal. |=== [NOTE] @@ -714,21 +739,23 @@ journal, before any new log entries are available in Elasticsearch, when using ==== [[fluentd-log-external-elasticsearch]] -*Having Fluentd send logs to another Elasticsearch* +*Having Fluentd Send Logs to Another Elasticsearch* You can configure Fluentd to send a copy of each log message to both the -Elasticsearch instance included with OpenShift aggregated logging, *and* to an -external Elasticsearch instance. For example, if you already have an +Elasticsearch instance included with {product-title} aggregated logging, _and_ +to an external Elasticsearch instance. For example, if you already have an Elasticsearch instance set up for auditing purposes, or data warehousing, you can send a copy of each log message to that Elasticsearch. This feature is controlled via environment variables on Fluentd, which can be modified as described below. -If its environment variable `*ES_COPY*` is `true`, Fluentd sends a copy of -the logs to another Elasticsearch. The names for the copy variables are just like the -current `*ES_HOST*`, `*OPS_HOST*`, etc. variables, except that they add -`_COPY`: `*ES_COPY_HOST*`, `*OPS_COPY_HOST*`, etc. There are some additional -parameters added: + +If its environment variable `*ES_COPY*` is *true*, Fluentd sends a copy of the +logs to another Elasticsearch. The names for the copy variables are just like +the current `*ES_HOST*`, `*OPS_HOST*`, and other variables, except that they add +`_COPY`: `*ES_COPY_HOST*`, `*OPS_COPY_HOST*`, and so on. There are some +additional parameters added: + * `*ES_COPY_SCHEME*`, `*OPS_COPY_SCHEME*` - can use either `http` or `https` - defaults to `https` * `*ES_COPY_USERNAME*`, `*OPS_COPY_USERNAME*` - user name to use to authenticate to @@ -757,37 +784,38 @@ $ oc new-app logging-fluentd-template [[fluentd-throttling]] *Throttling logs in Fluentd* -For projects that are especially verbose, an administrator can throttle -down the rate at which the logs are read in by Fluentd before being -processed. +For projects that are especially verbose, an administrator can throttle down the +rate at which the logs are read in by Fluentd before being processed. [WARNING] ==== -Throttling may contribute to log aggregation falling behind -for the configured projects; log entries could be lost if a pod were deleted -before Fluentd caught up. +Throttling can contribute to log aggregation falling behind for the configured +projects; log entries can be lost if a pod is deleted before Fluentd catches up. ==== [NOTE] ==== Throttling does not work when using the systemd journal as the log -source. The throttling implementation depends on being able to throttle the -reading of the individual log files for each project. When reading from the +source. The throttling implementation depends on being able to throttle the +reading of the individual log files for each project. When reading from the journal, there is only a single log source, no log files, so no file-based -throttling is available, and there isn't a method of restricting which log -entries are read into the Fluentd process. +throttling is available. There is not a method of restricting the log +entries that are read into the Fluentd process. ==== -To tell Fluentd which projects it should be restricting you will need -to edit the throttle configuration in its configmap after deployment: +To tell Fluentd which projects it should be restricting, edit the throttle +configuration in its ConfigMap after deployment: - $ oc edit configmap/logging-fluentd +---- +$ oc edit configmap/logging-fluentd +---- -The format of the throttle-config.yaml key is a yaml file that contains +The format of the *_throttle-config.yaml_* key is a YAML file that contains project names and the desired rate at which logs are read in on each node. The default is 1000 lines at a time per node. For example: -``` +==== +---- logging: read_lines_limit: 500 @@ -796,12 +824,13 @@ test-project: .operations: read_lines_limit: 100 -``` - +---- +==== +[[aggregate-logging-kibana]] === Kibana To access the Kibana console from the {product-title} web console, add the -`loggingPublicURL` parameter in the *_/etc/origin/master/master-config.yaml_* +`*loggingPublicURL*` parameter in the *_/etc/origin/master/master-config.yaml_* file, with the URL of the Kibana console (the `*kibana-hostname*` parameter). The value must be an HTTPS URL: @@ -815,7 +844,7 @@ assetConfig: ---- ==== -Setting the `loggingPublicURL` parameter creates a *View Archive* button on the +Setting the `*loggingPublicURL*` parameter creates a *View Archive* button on the {product-title} web console under the *Browse* -> *Pods* -> ** -> *Logs* tab. This links to the Kibana console. @@ -827,10 +856,10 @@ $ oc scale dc/logging-kibana --replicas=2 ---- ==== -You can see the UI by visiting the site specified at the `*KIBANA_HOSTNAME*` -variable. +You can see the user interface by visiting the site specified at the +`*KIBANA_HOSTNAME*` variable. -See the https://www.elastic.co/guide/en/kibana/4.1/discover.html[Kibana +See the link:https://www.elastic.co/guide/en/kibana/4.1/discover.html[Kibana documentation] for more information on Kibana. [[configuring-curator]] @@ -864,7 +893,7 @@ The available parameters are: |Description |`*$PROJECT_NAME*` -|The actual name of a project, such as `myapp-devel`. For {product-title} `operations` +|The actual name of a project, such as *myapp-devel*. For {product-title} *operations* logs, use the name `.operations` as the project name. |`*$ACTION*` @@ -877,7 +906,7 @@ logs, use the name `.operations` as the project name. |An integer for the number of units. |`*.defaults*` -|Use `.defaults` as the `$PROJECT_NAME` to set the defaults for projects that are +|Use `.defaults` as the `*$PROJECT_NAME*` to set the defaults for projects that are not specified. |`*runhour*` @@ -888,15 +917,15 @@ use with `.defaults`. |(Number) the minute of the hour at which to run the Curator jobs. For use with `.defaults`. |=== -For example, to configure Curator to +For example, to configure Curator to: -- delete indices in the `myapp-dev` project older than `1 day` -- delete indices in the `myapp-qe` project older than `1 week` -- delete `operations` logs older than `8 weeks` +- delete indices in the *myapp-dev* project older than `1 day` +- delete indices in the *myapp-qe* project older than `1 week` +- delete *operations* logs older than `8 weeks` - delete all other projects indices after they are `30 days` old - run the Curator jobs at midnight every day -you would use: +Use: ---- myapp-dev: @@ -921,7 +950,7 @@ myapp-qe: [IMPORTANT] ==== -When you use `month` as the `$UNIT` for an operation, Curator starts counting at +When you use `month` as the `*$UNIT*` for an operation, Curator starts counting at the first day of the current month, not the current day of the current month. For example, if today is April 15, and you want to delete indices that are 2 months older than today (delete: months: 2), Curator does not delete indices that are dated @@ -934,20 +963,20 @@ from that date. If you want to be exact with Curator, it is best to use days [[aggregate-logging-creating-the-curator-configuration]] ==== Creating the Curator Configuration -The deployer provides a configmap from which Curator reads its -configuration. You may edit or replace this configmap to reconfigure -Curator. Currently the `logging-curator` configmap is used to +The deployer provides a ConfigMap from which Curator reads its +configuration. You may edit or replace this ConfigMap to reconfigure +Curator. Currently the `logging-curator` ConfigMap is used to configure both your ops and non-ops Curator instances. Any `.operations` configurations will be in the same location as your application logs configurations. -. To edit the provided configmap to configure your Curator instances: +. To edit the provided ConfigMap to configure your Curator instances: + ---- $ oc edit configmap/logging-curator ---- -. To replace the provided configmap instead: +. To replace the provided ConfigMap instead: + ---- $ create /path/to/mycuratorconfig.yaml @@ -963,10 +992,11 @@ $ oc deploy --latest dc/logging-curator $ oc deploy --latest dc/logging-curator-ops ---- +[[aggregate-logging-cleanup]] == Cleanup -You can remove everything generated during the deployment while -leaving other project contents intact: +Remove everything generated during the deployment while leaving other project +contents intact: ---- $ oc new-app logging-deployer-template --param MODE=uninstall @@ -994,7 +1024,7 @@ host's OAuth2 server. If the secret is not identical on both servers, it can cause a login loop where you are continuously redirected back to the Kibana login page. -To fix this issue, delete the current oauthclient, and create a new one, using the +To fix this issue, delete the current OAuthClient, and create a new one, using the same template as before: ==== @@ -1006,8 +1036,8 @@ $ oc new-app logging-support-template *Cryptic Error When Viewing the Console* -When attempting to visit the Kibana console, you may instead receive a browser -error: +When attempting to visit the Kibana console, you may receive a browser +error instead: ==== ---- @@ -1020,7 +1050,7 @@ This can be caused by a mismatch between the OAuth2 client and server. The return address for the client must be in a whitelist so the server can securely redirect back after logging in. -Fix this issue by replacing the OAuth client entry: +Fix this issue by replacing the OAuthClient entry: ==== ---- @@ -1078,10 +1108,10 @@ $ oc get route --all-namespaces --selector logging-infra=support [[sending-logs-to-an-external-elasticsearch-instance]] == Sending Logs to an External Elasticsearch Instance -Fluentd sends logs to the value of the `ES_HOST`, `ES_PORT`, `OPS_HOST`, and -`OPS_PORT` environment variables of the Elasticsearch deployment configuration. -The application logs are directed to the `ES_HOST` destination, and operations -logs to `OPS_HOST`. +Fluentd sends logs to the value of the `*ES_HOST*`, `*ES_PORT*`, `*OPS_HOST*`, +and `*OPS_PORT*` environment variables of the Elasticsearch deployment +configuration. The application logs are directed to the `*ES_HOST*` destination, +and operations logs to `*OPS_HOST*`. To direct logs to a specific Elasticsearch instance, edit the deployment configuration and replace the value of the above variables with the desired @@ -1092,19 +1122,19 @@ $ oc edit dc/ ---- For an external Elasticsearch instance to contain both application and -operations logs, you can set `ES_HOST` and `OPS_HOST` to the same destination, -while ensuring that `ES_PORT` and `OPS_PORT` also have the same value. +operations logs, you can set `*ES_HOST*` and `*OPS_HOST*` to the same destination, +while ensuring that `*ES_PORT*` and `*OPS_PORT*` also have the same value. If your externally hosted Elasticsearch instance does not use TLS, update the -`*_CLIENT_CERT`, `*_CLIENT_KEY`, and `*_CA` variables to be empty. If it does -use TLS, but not mutual TLS, update the `*_CLIENT_CERT` and `*_CLIENT_KEY` -variables to be empty and patch or recreate the `logging-fluentd` secret with -the appropriate `*_CA` value for communicating with your Elasticsearch instance. +`*_CLIENT_CERT*`, `*_CLIENT_KEY*`, and `*_CA*` variables to be empty. If it does +use TLS, but not mutual TLS, update the `*_CLIENT_CERT*` and `*_CLIENT_KEY*` +variables to be empty and patch or recreate the *logging-fluentd* secret with +the appropriate `*_CA*` value for communicating with your Elasticsearch instance. If it uses Mutual TLS as the provided Elasticsearch instance does, patch or -recreate the `logging-fluentd` secret with your client key, client cert, and CA. +recreate the *logging-fluentd* secret with your client key, client cert, and CA. Since Fluentd is deployed by a DaemonSet, update the -`logging-fluentd-template` template, delete your current DaemonSet, and recreate +*logging-fluentd-template* template, delete your current DaemonSet, and recreate it with `oc new-app logging-fluentd-template` after seeing all previous Fluentd pods have terminated. @@ -1125,9 +1155,9 @@ endif::openshift-origin[] ifdef::openshift-enterprise[] 3.2.0, endif::openshift-enterprise[] -an admin certificate, key, and CA that can be used to communicate with and perform +an administrator certificate, key, and CA that can be used to communicate with and perform administrative operations on Elasticsearch are provided within the -`logging-elasticsearch` secret. +*logging-elasticsearch* secret. [NOTE] ==== @@ -1141,11 +1171,11 @@ If they are not available, refer to xref:../install_config/upgrading/manual_upgrades.adoc#manual-upgrading-efk-logging-stack[Manual Upgrades] to ensure you are on the latest version first. -Connect to an Elasticsearch pod that is in the cluster on which you are +. Connect to an Elasticsearch pod that is in the cluster on which you are attempting to perform maintenance. -To find a pod in a cluster use either: - +. To find a pod in a cluster use either: ++ ==== ---- $ oc get pods -l component=es -o name | head -1 @@ -1153,24 +1183,24 @@ $ oc get pods -l component=es-ops -o name | head -1 ---- ==== -Then, connect to a pod: - +. Connect to a pod: ++ ==== ---- $ oc rsh ---- ==== -Once connected to an Elasticsearch container, you can use the certificates mounted +. Once connected to an Elasticsearch container, you can use the certificates mounted from the secret to communicate with Elasticsearch per its 1.5 link:https://www.elastic.co/guide/en/elasticsearch/reference/1.5/docs.html[Document APIs]. - -Fluentd sends its logs to Elasticsearch using the index format "{project_name}.{project_uuid}.YYYY.MM.DD" ++ +Fluentd sends its logs to Elasticsearch using the index format *{project_name}.{project_uuid}.YYYY.MM.DD* where YYYY.MM.DD is the date of the log record. - -For example, to delete all logs for the `logging` project with uuid `3b3594fa-2ccd-11e6-acb7-0eb6b35eaee3` ++ +For example, to delete all logs for the *logging* project with uuid *3b3594fa-2ccd-11e6-acb7-0eb6b35eaee3* from June 15, 2016, we can run: - ++ ==== ---- $ curl --key /etc/elasticsearch/keys/admin-key --cert /etc/elasticsearch/keys/admin-cert \