Chapter 3. Troubleshooting logging

Procedure

1. Change to the openshift-logging project by running the following command:

   Copy to Clipboard Toggle word wrap

   $ oc project openshift-logging

2. Get the ClusterLogging instance status by running the following command:

   Copy to Clipboard Toggle word wrap

   $ oc get clusterlogging instance -o yaml

   Example output

   Copy to Clipboard Toggle word wrap

   apiVersion: logging.openshift.io/v1
   kind: ClusterLogging
   # ...
   status:  

   1

   
     collection:
       logs:
         fluentdStatus:
           daemonSet: fluentd  

   2

   
           nodes:
             collector-2rhqp: ip-10-0-169-13.ec2.internal
             collector-6fgjh: ip-10-0-165-244.ec2.internal
             collector-6l2ff: ip-10-0-128-218.ec2.internal
             collector-54nx5: ip-10-0-139-30.ec2.internal
             collector-flpnn: ip-10-0-147-228.ec2.internal
             collector-n2frh: ip-10-0-157-45.ec2.internal
           pods:
             failed: []
             notReady: []
             ready:
             - collector-2rhqp
             - collector-54nx5
             - collector-6fgjh
             - collector-6l2ff
             - collector-flpnn
             - collector-n2frh
     logstore: 

   3

   
       elasticsearchStatus:
       - ShardAllocationEnabled:  all
         cluster:
           activePrimaryShards:    5
           activeShards:           5
           initializingShards:     0
           numDataNodes:           1
           numNodes:               1
           pendingTasks:           0
           relocatingShards:       0
           status:                 green
           unassignedShards:       0
         clusterName:             elasticsearch
         nodeConditions:
           elasticsearch-cdm-mkkdys93-1:
         nodeCount:  1
         pods:
           client:
             failed:
             notReady:
             ready:
             - elasticsearch-cdm-mkkdys93-1-7f7c6-mjm7c
           data:
             failed:
             notReady:
             ready:
             - elasticsearch-cdm-mkkdys93-1-7f7c6-mjm7c
           master:
             failed:
             notReady:
             ready:
             - elasticsearch-cdm-mkkdys93-1-7f7c6-mjm7c
   visualization:  

   4

   
       kibanaStatus:
       - deployment: kibana
         pods:
           failed: []
           notReady: []
           ready:
           - kibana-7fb4fd4cc9-f2nls
         replicaSets:
         - kibana-7fb4fd4cc9
         replicas: 1

   1

   In the output, the cluster status fields appear in the status stanza.

   2

   Information on the Fluentd pods.

   3

   Information on the Elasticsearch pods, including Elasticsearch cluster health, green, yellow, or red.

   4

   Information on the Kibana pods.

Procedure

1. Change to the openshift-logging project.

   Copy to Clipboard Toggle word wrap

   $ oc project openshift-logging

2. View the status of logging environment:

   Copy to Clipboard Toggle word wrap

   $ oc describe deployment cluster-logging-operator

   Example output

   Copy to Clipboard Toggle word wrap

   Name:                   cluster-logging-operator
   
   ....
   
   Conditions:
     Type           Status  Reason
     ----           ------  ------
     Available      True    MinimumReplicasAvailable
     Progressing    True    NewReplicaSetAvailable
   
   ....
   
   Events:
     Type    Reason             Age   From                   Message
     ----    ------             ----  ----                   -------
     Normal  ScalingReplicaSet  62m   deployment-controller  Scaled up replica set cluster-logging-operator-574b8987df to 1----

3. View the status of the logging replica set:

   1. Get the name of a replica set:

      Example output

      Copy to Clipboard Toggle word wrap

      $ oc get replicaset

      Example output

      Copy to Clipboard Toggle word wrap

      NAME                                      DESIRED   CURRENT   READY   AGE
      cluster-logging-operator-574b8987df       1         1         1       159m
      elasticsearch-cdm-uhr537yu-1-6869694fb    1         1         1       157m
      elasticsearch-cdm-uhr537yu-2-857b6d676f   1         1         1       156m
      elasticsearch-cdm-uhr537yu-3-5b6fdd8cfd   1         1         1       155m
      kibana-5bd5544f87                         1         1         1       157m

   2. Get the status of the replica set:

      Copy to Clipboard Toggle word wrap

      $ oc describe replicaset cluster-logging-operator-574b8987df

      Example output

      Copy to Clipboard Toggle word wrap

      Name:           cluster-logging-operator-574b8987df
      
      ....
      
      Replicas:       1 current / 1 desired
      Pods Status:    1 Running / 0 Waiting / 0 Succeeded / 0 Failed
      
      ....
      
      Events:
        Type    Reason            Age   From                   Message
        ----    ------            ----  ----                   -------
        Normal  SuccessfulCreate  66m   replicaset-controller  Created pod: cluster-logging-operator-574b8987df-qjhqv----

Procedure

1. Check the Elasticsearch cluster health and verify that the cluster status is red by running the following command:

   Copy to Clipboard Toggle word wrap

   $ oc exec -n openshift-logging -c elasticsearch $ES_POD_NAME -- health

2. List the nodes that have joined the cluster by running the following command:

   Copy to Clipboard Toggle word wrap

   $ oc exec -n openshift-logging -c elasticsearch $ES_POD_NAME \
     -- es_util --query=_cat/nodes?v

3. List the Elasticsearch pods and compare them with the nodes in the command output from the previous step, by running the following command:

   Copy to Clipboard Toggle word wrap

   $ oc -n openshift-logging get pods -l component=elasticsearch

4. If some of the Elasticsearch nodes have not joined the cluster, perform the following steps.

   1. Confirm that Elasticsearch has an elected master node by running the following command and observing the output:

      Copy to Clipboard Toggle word wrap

      $ oc exec -n openshift-logging -c elasticsearch $ES_POD_NAME \
        -- es_util --query=_cat/master?v

   2. Review the pod logs of the elected master node for issues by running the following command and observing the output:

      Copy to Clipboard Toggle word wrap

      $ oc logs <elasticsearch_master_pod_name> -c elasticsearch -n openshift-logging

   3. Review the logs of nodes that have not joined the cluster for issues by running the following command and observing the output:

      Copy to Clipboard Toggle word wrap

      $ oc logs <elasticsearch_node_name> -c elasticsearch -n openshift-logging

5. If all the nodes have joined the cluster, check if the cluster is in the process of recovering by running the following command and observing the output:

   Copy to Clipboard Toggle word wrap

   $ oc exec -n openshift-logging -c elasticsearch $ES_POD_NAME \
     -- es_util --query=_cat/recovery?active_only=true

   If there is no command output, the recovery process might be delayed or stalled by pending tasks.

6. Check if there are pending tasks by running the following command and observing the output:

   Copy to Clipboard Toggle word wrap

   $ oc exec -n openshift-logging -c elasticsearch $ES_POD_NAME \
     -- health | grep number_of_pending_tasks

7. If there are pending tasks, monitor their status. If their status changes and indicates that the cluster is recovering, continue waiting. The recovery time varies according to the size of the cluster and other factors. Otherwise, if the status of the pending tasks does not change, this indicates that the recovery has stalled.

8. If it seems like the recovery has stalled, check if the cluster.routing.allocation.enable value is set to none, by running the following command and observing the output:

   Copy to Clipboard Toggle word wrap

   $ oc exec -n openshift-logging -c elasticsearch $ES_POD_NAME \
     -- es_util --query=_cluster/settings?pretty

9. If the cluster.routing.allocation.enable value is set to none, set it to all, by running the following command:

   Copy to Clipboard Toggle word wrap

   $ oc exec -n openshift-logging -c elasticsearch $ES_POD_NAME \
     -- es_util --query=_cluster/settings?pretty \
     -X PUT -d '{"persistent": {"cluster.routing.allocation.enable":"all"}}'

10. Check if any indices are still red by running the following command and observing the output:

    Copy to Clipboard Toggle word wrap

    $ oc exec -n openshift-logging -c elasticsearch $ES_POD_NAME \
      -- es_util --query=_cat/indices?v

11. If any indices are still red, try to clear them by performing the following steps.

    1. Clear the cache by running the following command:

       Copy to Clipboard Toggle word wrap

       $ oc exec -n openshift-logging -c elasticsearch $ES_POD_NAME \
         -- es_util --query=<elasticsearch_index_name>/_cache/clear?pretty

    2. Increase the max allocation retries by running the following command:

       Copy to Clipboard Toggle word wrap

       $ oc exec -n openshift-logging -c elasticsearch $ES_POD_NAME \
         -- es_util --query=<elasticsearch_index_name>/_settings?pretty \
         -X PUT -d '{"index.allocation.max_retries":10}'

    3. Delete all the scroll items by running the following command:

       Copy to Clipboard Toggle word wrap

       $ oc exec -n openshift-logging -c elasticsearch $ES_POD_NAME \
         -- es_util --query=_search/scroll/_all -X DELETE

    4. Increase the timeout by running the following command:

       Copy to Clipboard Toggle word wrap

       $ oc exec -n openshift-logging -c elasticsearch $ES_POD_NAME \
         -- es_util --query=<elasticsearch_index_name>/_settings?pretty \
         -X PUT -d '{"index.unassigned.node_left.delayed_timeout":"10m"}'

12. If the preceding steps do not clear the red indices, delete the indices individually.

    1. Identify the red index name by running the following command:

       Copy to Clipboard Toggle word wrap

       $ oc exec -n openshift-logging -c elasticsearch $ES_POD_NAME \
         -- es_util --query=_cat/indices?v

    2. Delete the red index by running the following command:

       Copy to Clipboard Toggle word wrap

       $ oc exec -n openshift-logging -c elasticsearch $ES_POD_NAME \
         -- es_util --query=<elasticsearch_red_index_name> -X DELETE

13. If there are no red indices and the cluster status is red, check for a continuous heavy processing load on a data node.

    1. Check if the Elasticsearch JVM Heap usage is high by running the following command:

       Copy to Clipboard Toggle word wrap

       $ oc exec -n openshift-logging -c elasticsearch $ES_POD_NAME \
         -- es_util --query=_nodes/stats?pretty

       In the command output, review the node_name.jvm.mem.heap_used_percent field to determine the JVM Heap usage.

    2. Check for high CPU utilization. For more information about CPU utilitzation, see the OpenShift Dedicated "Reviewing monitoring dashboards" documentation.

Procedure

1. Identify the node on which Elasticsearch is deployed by running the following command:

   Copy to Clipboard Toggle word wrap

   $ oc -n openshift-logging get po -o wide

2. Check if there are unassigned shards by running the following command:

   Copy to Clipboard Toggle word wrap

   $ oc exec -n openshift-logging -c elasticsearch $ES_POD_NAME \
     -- es_util --query=_cluster/health?pretty | grep unassigned_shards

3. If there are unassigned shards, check the disk space on each node, by running the following command:

   Copy to Clipboard Toggle word wrap

   $ for pod in `oc -n openshift-logging get po -l component=elasticsearch -o jsonpath='{.items[*].metadata.name}'`; \
     do echo $pod; oc -n openshift-logging exec -c elasticsearch $pod \
     -- df -h /elasticsearch/persistent; done

4. In the command output, check the Use column to determine the used disk percentage on that node.

   Example output

   Copy to Clipboard Toggle word wrap

   elasticsearch-cdm-kcrsda6l-1-586cc95d4f-h8zq8
   Filesystem      Size  Used Avail Use% Mounted on
   /dev/nvme1n1     19G  522M   19G   3% /elasticsearch/persistent
   elasticsearch-cdm-kcrsda6l-2-5b548fc7b-cwwk7
   Filesystem      Size  Used Avail Use% Mounted on
   /dev/nvme2n1     19G  522M   19G   3% /elasticsearch/persistent
   elasticsearch-cdm-kcrsda6l-3-5dfc884d99-59tjw
   Filesystem      Size  Used Avail Use% Mounted on
   /dev/nvme3n1     19G  528M   19G   3% /elasticsearch/persistent

   If the used disk percentage is above 85%, the node has exceeded the low watermark, and shards can no longer be allocated to this node.

5. To check the current redundancyPolicy, run the following command:

   Copy to Clipboard Toggle word wrap

   $ oc -n openshift-logging get es elasticsearch \
     -o jsonpath='{.spec.redundancyPolicy}'

   If you are using a ClusterLogging resource on your cluster, run the following command:

   Copy to Clipboard Toggle word wrap

   $ oc -n openshift-logging get cl \
     -o jsonpath='{.items[*].spec.logStore.elasticsearch.redundancyPolicy}'

   If the cluster redundancyPolicy value is higher than the SingleRedundancy value, set it to the SingleRedundancy value and save this change.

6. If the preceding steps do not fix the issue, delete the old indices.

   1. Check the status of all indices on Elasticsearch by running the following command:

      Copy to Clipboard Toggle word wrap

      $ oc exec -n openshift-logging -c elasticsearch $ES_POD_NAME -- indices

   2. Identify an old index that can be deleted.

   3. Delete the index by running the following command:

      Copy to Clipboard Toggle word wrap

      $ oc exec -n openshift-logging -c elasticsearch $ES_POD_NAME \
        -- es_util --query=<elasticsearch_index_name> -X DELETE

Procedure

1. Identify the node on which Elasticsearch is deployed by running the following command:

   Copy to Clipboard Toggle word wrap

   $ oc -n openshift-logging get po -o wide

2. Check the disk space on each node:

   Copy to Clipboard Toggle word wrap

   $ for pod in `oc -n openshift-logging get po -l component=elasticsearch -o jsonpath='{.items[*].metadata.name}'`; \
     do echo $pod; oc -n openshift-logging exec -c elasticsearch $pod \
     -- df -h /elasticsearch/persistent; done

3. Check if the cluster is rebalancing:

   Copy to Clipboard Toggle word wrap

   $ oc exec -n openshift-logging -c elasticsearch $ES_POD_NAME \
     -- es_util --query=_cluster/health?pretty | grep relocating_shards

   If the command output shows relocating shards, the high watermark has been exceeded. The default value of the high watermark is 90%.

4. Increase the disk space on all nodes. If increasing the disk space is not possible, try adding a new data node to the cluster, or decrease the total cluster redundancy policy.

5. To check the current redundancyPolicy, run the following command:

   Copy to Clipboard Toggle word wrap

   $ oc -n openshift-logging get es elasticsearch \
     -o jsonpath='{.spec.redundancyPolicy}'

   If you are using a ClusterLogging resource on your cluster, run the following command:

   Copy to Clipboard Toggle word wrap

   $ oc -n openshift-logging get cl \
     -o jsonpath='{.items[*].spec.logStore.elasticsearch.redundancyPolicy}'

   If the cluster redundancyPolicy value is higher than the SingleRedundancy value, set it to the SingleRedundancy value and save this change.

6. If the preceding steps do not fix the issue, delete the old indices.

   1. Check the status of all indices on Elasticsearch by running the following command:

      Copy to Clipboard Toggle word wrap

      $ oc exec -n openshift-logging -c elasticsearch $ES_POD_NAME -- indices

   2. Identify an old index that can be deleted.

   3. Delete the index by running the following command:

      Copy to Clipboard Toggle word wrap

      $ oc exec -n openshift-logging -c elasticsearch $ES_POD_NAME \
        -- es_util --query=<elasticsearch_index_name> -X DELETE

Procedure

1. Get the disk space of the Elasticsearch node:

   Copy to Clipboard Toggle word wrap

   $ for pod in `oc -n openshift-logging get po -l component=elasticsearch -o jsonpath='{.items[*].metadata.name}'`; \
     do echo $pod; oc -n openshift-logging exec -c elasticsearch $pod \
     -- df -h /elasticsearch/persistent; done

2. In the command output, check the Avail column to determine the free disk space on that node.

   Example output

   Copy to Clipboard Toggle word wrap

   elasticsearch-cdm-kcrsda6l-1-586cc95d4f-h8zq8
   Filesystem      Size  Used Avail Use% Mounted on
   /dev/nvme1n1     19G  522M   19G   3% /elasticsearch/persistent
   elasticsearch-cdm-kcrsda6l-2-5b548fc7b-cwwk7
   Filesystem      Size  Used Avail Use% Mounted on
   /dev/nvme2n1     19G  522M   19G   3% /elasticsearch/persistent
   elasticsearch-cdm-kcrsda6l-3-5dfc884d99-59tjw
   Filesystem      Size  Used Avail Use% Mounted on
   /dev/nvme3n1     19G  528M   19G   3% /elasticsearch/persistent

3. Increase the disk space on all nodes. If increasing the disk space is not possible, try adding a new data node to the cluster, or decrease the total cluster redundancy policy.

4. To check the current redundancyPolicy, run the following command:

   Copy to Clipboard Toggle word wrap

   $ oc -n openshift-logging get es elasticsearch \
     -o jsonpath='{.spec.redundancyPolicy}'

   If you are using a ClusterLogging resource on your cluster, run the following command:

   Copy to Clipboard Toggle word wrap

   $ oc -n openshift-logging get cl \
     -o jsonpath='{.items[*].spec.logStore.elasticsearch.redundancyPolicy}'

   If the cluster redundancyPolicy value is higher than the SingleRedundancy value, set it to the SingleRedundancy value and save this change.

5. If the preceding steps do not fix the issue, delete the old indices.

   1. Check the status of all indices on Elasticsearch by running the following command:

      Copy to Clipboard Toggle word wrap

      $ oc exec -n openshift-logging -c elasticsearch $ES_POD_NAME -- indices

   2. Identify an old index that can be deleted.

   3. Delete the index by running the following command:

      Copy to Clipboard Toggle word wrap

      $ oc exec -n openshift-logging -c elasticsearch $ES_POD_NAME \
        -- es_util --query=<elasticsearch_index_name> -X DELETE

6. Continue freeing up and monitoring the disk space. After the used disk space drops below 90%, unblock writing to this node by running the following command:

   Copy to Clipboard Toggle word wrap

   $ oc exec -n openshift-logging -c elasticsearch $ES_POD_NAME \
     -- es_util --query=_all/_settings?pretty \
     -X PUT -d '{"index.blocks.read_only_allow_delete": null}'

Procedure

1. Get the disk space of the Elasticsearch node:

   Copy to Clipboard Toggle word wrap

   $ for pod in `oc -n openshift-logging get po -l component=elasticsearch -o jsonpath='{.items[*].metadata.name}'`; \
     do echo $pod; oc -n openshift-logging exec -c elasticsearch $pod \
     -- df -h /elasticsearch/persistent; done

2. In the command output, check the Avail column to determine the free disk space on that node.

   Example output

   Copy to Clipboard Toggle word wrap

   elasticsearch-cdm-kcrsda6l-1-586cc95d4f-h8zq8
   Filesystem      Size  Used Avail Use% Mounted on
   /dev/nvme1n1     19G  522M   19G   3% /elasticsearch/persistent
   elasticsearch-cdm-kcrsda6l-2-5b548fc7b-cwwk7
   Filesystem      Size  Used Avail Use% Mounted on
   /dev/nvme2n1     19G  522M   19G   3% /elasticsearch/persistent
   elasticsearch-cdm-kcrsda6l-3-5dfc884d99-59tjw
   Filesystem      Size  Used Avail Use% Mounted on
   /dev/nvme3n1     19G  528M   19G   3% /elasticsearch/persistent

3. Increase the disk space on all nodes. If increasing the disk space is not possible, try adding a new data node to the cluster, or decrease the total cluster redundancy policy.

4. To check the current redundancyPolicy, run the following command:

   Copy to Clipboard Toggle word wrap

   $ oc -n openshift-logging get es elasticsearch -o jsonpath='{.spec.redundancyPolicy}'

   If you are using a ClusterLogging resource on your cluster, run the following command:

   Copy to Clipboard Toggle word wrap

   $ oc -n openshift-logging get cl \
     -o jsonpath='{.items[*].spec.logStore.elasticsearch.redundancyPolicy}'

   If the cluster redundancyPolicy value is higher than the SingleRedundancy value, set it to the SingleRedundancy value and save this change.

5. If the preceding steps do not fix the issue, delete the old indices.

   1. Check the status of all indices on Elasticsearch by running the following command:

      Copy to Clipboard Toggle word wrap

      $ oc exec -n openshift-logging -c elasticsearch $ES_POD_NAME -- indices

   2. Identify an old index that can be deleted.

   3. Delete the index by running the following command:

      Copy to Clipboard Toggle word wrap

      $ oc exec -n openshift-logging -c elasticsearch $ES_POD_NAME \
        -- es_util --query=<elasticsearch_index_name> -X DELETE

Procedure

1. Change to the openshift-logging project by running the following command:

   Copy to Clipboard Toggle word wrap

   $ oc project openshift-logging

2. To view the status:

   1. Get the name of the Elasticsearch log store instance by running the following command:

      Copy to Clipboard Toggle word wrap

      $ oc get Elasticsearch

      Example output

      Copy to Clipboard Toggle word wrap

      NAME            AGE
      elasticsearch   5h9m

   2. Get the Elasticsearch log store status by running the following command:

      Copy to Clipboard Toggle word wrap

      $ oc get Elasticsearch <Elasticsearch-instance> -o yaml

      For example:

      Copy to Clipboard Toggle word wrap

      $ oc get Elasticsearch elasticsearch -n openshift-logging -o yaml

      The output includes information similar to the following:

      Example output

      Copy to Clipboard Toggle word wrap

      status: 

      1

      
        cluster: 

      2

      
          activePrimaryShards: 30
          activeShards: 60
          initializingShards: 0
          numDataNodes: 3
          numNodes: 3
          pendingTasks: 0
          relocatingShards: 0
          status: green
          unassignedShards: 0
        clusterHealth: ""
        conditions: [] 

      3

      
        nodes: 

      4

      
        - deploymentName: elasticsearch-cdm-zjf34ved-1
          upgradeStatus: {}
        - deploymentName: elasticsearch-cdm-zjf34ved-2
          upgradeStatus: {}
        - deploymentName: elasticsearch-cdm-zjf34ved-3
          upgradeStatus: {}
        pods: 

      5

      
          client:
            failed: []
            notReady: []
            ready:
            - elasticsearch-cdm-zjf34ved-1-6d7fbf844f-sn422
            - elasticsearch-cdm-zjf34ved-2-dfbd988bc-qkzjz
            - elasticsearch-cdm-zjf34ved-3-c8f566f7c-t7zkt
          data:
            failed: []
            notReady: []
            ready:
            - elasticsearch-cdm-zjf34ved-1-6d7fbf844f-sn422
            - elasticsearch-cdm-zjf34ved-2-dfbd988bc-qkzjz
            - elasticsearch-cdm-zjf34ved-3-c8f566f7c-t7zkt
          master:
            failed: []
            notReady: []
            ready:
            - elasticsearch-cdm-zjf34ved-1-6d7fbf844f-sn422
            - elasticsearch-cdm-zjf34ved-2-dfbd988bc-qkzjz
            - elasticsearch-cdm-zjf34ved-3-c8f566f7c-t7zkt
        shardAllocationEnabled: all

      1

      In the output, the cluster status fields appear in the status stanza.

      2

      The status of the Elasticsearch log store:

      • The number of active primary shards.
      • The number of active shards.
      • The number of shards that are initializing.
      • The number of Elasticsearch log store data nodes.
      • The total number of Elasticsearch log store nodes.
      • The number of pending tasks.
      • The Elasticsearch log store status: green, red, yellow.
      • The number of unassigned shards.

      3

      Any status conditions, if present. The Elasticsearch log store status indicates the reasons from the scheduler if a pod could not be placed. Any events related to the following conditions are shown:

      • Container Waiting for both the Elasticsearch log store and proxy containers.
      • Container Terminated for both the Elasticsearch log store and proxy containers.
      • Pod unschedulable. Also, a condition is shown for a number of issues; see Example condition messages.

      4

      The Elasticsearch log store nodes in the cluster, with upgradeStatus.

      5

      The Elasticsearch log store client, data, and master pods in the cluster, listed under failed, notReady, or ready state.