Application Logging on Red Hat OpenShift Container Platform (RHOCP) 4.4 with Elasticsearch, Fluentd, and Kibana
The following guide has been tested with Red Hat OpenShift Container Platform (RHOCP) 4.2/Kabanero 0.3.0, RHOCP 4.3/Kabanero 0.6.0, and RHOCP 4.4/Kabanero 0.8.0.
Pod processes running in Kubernetes frequently produce logs. To effectively manage this log data and ensure no loss of log data occurs when a pod terminates, a log aggregation tool should be deployed on the Kubernetes cluster. Log aggregation tools help users persist, search, and visualize the log data that is gathered from the pods across the cluster. Log aggregation tools in the market today include: EFK, LogDNA, Splunk, Datadog, IBM Operations Analytics, etc. When considering log aggregation tools, enterprises will make choices that are inclusive of their journey to cloud, both new cloud native applications running in Kubernetes and their existing traditional IT choices.
One choice for application logging with log aggregation, based on open source, is EFK (Elasticsearch, Fluentd, and Kibana). This guide describes the process of deploying EFK using the Elasticsearch Operator and the Cluster Logging Operator. Use this preconfigured EFK stack to aggregate all container logs. After a successful installation, the EFK pods should reside inside the openshift-logging namespace of the cluster.
Install cluster logging
To install the cluster logging component, follow the OpenShift guide Deploying cluster logging. Ensure you set up valid storage for Elasticsearch via Persistent Volumes. When the example Cluster Logging instance YAML from the guide is deployed, the Elasticsearch pods that are created will automatically search for Persistent Volumes to bind to; if there are none available for binding, the Elasticsearch pods will be stuck in a pending state. Using in-memory storage is also possible by removing the
storage definition from the Cluster Logging instance YAML, but this is not suitable for production.
After the installation completes without any error, you can see the following pods that are running in the openshift-logging namespace. The exact number of pods running for each of the EFK components can vary depending on the configuration specified in the ClusterLogging Custom Resource (CR).
[root@rhel7-ocp ~]# oc get pods -n openshift-logging NAME READY STATUS RESTARTS AGE cluster-logging-operator-874597bcb-qlmlf 1/1 Running 0 150m curator-1578684600-2lgqp 0/1 Completed 0 4m46s elasticsearch-cdm-4qrvthgd-1-5444897599-7rqx8 2/2 Running 0 9m6s elasticsearch-cdm-4qrvthgd-2-865c6b6d85-69b4r 2/2 Running 0 8m3s fluentd-rmdbn 1/1 Running 0 9m5s fluentd-vtk48 1/1 Running 0 9m5s kibana-756fcdb7f-rw8k8 2/2 Running 0 9m6s
The cluster logging also exposes a route for external access to the Kibana console.
[root@rhel7-okd ~]# oc get routes -n openshift-logging NAME HOST/PORT PATH SERVICES PORT TERMINATION WILDCARD kibana kibana-openshift-logging.apps.host.kabanero.com kibana <all> reencrypt/Redirect None
Configure Fluentd to merge JSON log message body
If there are application pods outputting logs in JSON format, then it is recommended to set Fluentd to parse the JSON fields from the message body and merge the parsed objects with the JSON payload document posted to Elasticsearch.
This feature is disabled by default. To enable this feature, first set the cluster logging instance’s managementState field from “Managed” to “Unmanaged”. Setting the cluster logging instance to unmanaged state gives the administrator full control of the components managed by the Cluster Logging Operator, and is the prerequisite for many cluster logging configurations.
[root@rhel7-ocp ~]# oc edit ClusterLogging instance apiVersion: "logging.openshift.io/v1" kind: "ClusterLogging" metadata: name: "instance" .... spec: managementState: "Unmanaged"
Then, set the environment variable MERGE_JSON_LOG to true with the following command:
[root@rhel7-ocp ~]# oc set env ds/fluentd MERGE_JSON_LOG=true
View application logs in Kibana
In cases where the application server provides the option, output application logs in JSON format. This will let you fully take advantage of Kibana’s dashboard functions. Kibana is then able to process the data from each individual field of the JSON object to create customized visualizations for that field.
See the Kibana dashboard page by using the routes URL https://kibana-openshift-logging.apps.host.kabanero.com. Log in using your Kubernetes user and password, then the browser should redirect you to Kibana’s Discover page where the newest logs of the selected index are being streamed. Select the project.* index to view the application logs generated by the deployed application.
The project.* index contains only a set of default fields at the start, which does not include all of the fields from the deployed application’s JSON log object. Therefore, the index needs to be refreshed to have all the fields from the application’s log object available to Kibana.
To refresh the index, click the Management option from the Kibana menu.
Click Index Pattern, and find the project.* index in Index Pattern. Then, click the refresh fields button. After Kibana is updated with all the available fields in the project.* index, import any preconfigured dashboards to view the application’s logs.
To import the dashboard and its associated objects, navigate back to the Management page and click Saved Objects. Click Import and select the dashboard file. When prompted, click the Yes, overwrite all option
Head back to the Dashboard page and enjoy navigating logs on the newly imported dashboard.
Configuring and uninstalling cluster logging
If changes need to be made for the installed EFK stack, edit the ClusterLogging Custom Resource (CR) of the deployed cluster logging instance. Make sure to set the managementState to “Unmanaged” first before making any changes to the existing configuration. If the EFK stack is no longer needed, remove the cluster logging instance from Cluster Logging Operator Details page.