Chapter 4. Configuring the Distributed Tracing Platform

Procedure

1. In the OpenTelemetryCollector custom resource of the OpenTelemetry Collector, enable the Spanmetrics Connector (spanmetrics), which derives metrics from traces and exports the metrics in the Prometheus format.

   Example OpenTelemetryCollector custom resource for span RED

   Copy to Clipboard Toggle word wrap

   apiVersion: opentelemetry.io/v1beta1
   kind: OpenTelemetryCollector
   metadata:
     name: otel
   spec:
     mode: deployment
     observability:
       metrics:
         enableMetrics: true 

   1

   
     config: |
       connectors:
         spanmetrics: 

   2

   
           metrics_flush_interval: 15s
   
       receivers:
         otlp: 

   3

   
           protocols:
             grpc:
             http:
   
       exporters:
         prometheus: 

   4

   
           endpoint: 0.0.0.0:8889
           add_metric_suffixes: false
           resource_to_telemetry_conversion:
             enabled: true 

   5

   
   
         otlp:
           auth:
             authenticator: bearertokenauth
           endpoint: tempo-redmetrics-gateway.mynamespace.svc.cluster.local:8090
           headers:
             X-Scope-OrgID: dev
           tls:
             ca_file: /var/run/secrets/kubernetes.io/serviceaccount/service-ca.crt
             insecure: false
   
       extensions:
         bearertokenauth:
           filename: /var/run/secrets/kubernetes.io/serviceaccount/token
   
       service:
         extensions:
         - bearertokenauth
         pipelines:
           traces:
             receivers: [otlp]
             exporters: [otlp, spanmetrics] 

   6

   
           metrics:
             receivers: [spanmetrics] 

   7

   
             exporters: [prometheus]
   
   # ...

   1

   Creates the ServiceMonitor custom resource to enable scraping of the Prometheus exporter.

   2

   The Spanmetrics connector receives traces and exports metrics.

   3

   The OTLP receiver to receive spans in the OpenTelemetry protocol.

   4

   The Prometheus exporter is used to export metrics in the Prometheus format.

   5

   The resource attributes are dropped by default.

   6

   The Spanmetrics connector is configured as exporter in traces pipeline.

   7

   The Spanmetrics connector is configured as receiver in metrics pipeline.

2. In the TempoStack custom resource, enable the Monitor tab and set the Prometheus endpoint to the Thanos querier service to query the data from your user-defined monitoring stack.

   Example TempoStack custom resource with the enabled Monitor tab

   Copy to Clipboard Toggle word wrap

   apiVersion: tempo.grafana.com/v1alpha1
   kind: TempoStack
   metadata:
     name: redmetrics
   spec:
     storage:
       secret:
         name: minio-test
         type: s3
     storageSize: 1Gi
     tenants:
       mode: openshift
       authentication:
         - tenantName: dev
           tenantId: "1610b0c3-c509-4592-a256-a1871353dbfa"
     template:
       gateway:
         enabled: true
       queryFrontend:
         jaegerQuery:
           monitorTab:
             enabled: true 

   1

   
             prometheusEndpoint: https://thanos-querier.openshift-monitoring.svc.cluster.local:9092 

   2

   
             redMetricsNamespace: "" 

   3

   
   
   # ...

   1

   Enables the monitoring tab in the Jaeger console.

   2

   The service name for Thanos Querier from user-workload monitoring.

   3

   Optional: The metrics namespace on which the Jaeger query retrieves the Prometheus metrics. Include this line only if you are using an OpenTelemetry Collector version earlier than 0.109.0. If you are using an OpenTelemetry Collector version 0.109.0 or later, omit this line.

3. Optional: Use the span RED metrics generated by the spanmetrics connector with alerting rules. For example, for alerts about a slow service or to define service level objectives (SLOs), the connector creates a duration_bucket histogram and the calls counter metric. These metrics have labels that identify the service, API name, operation type, and other attributes.

   Table 4.4. Labels of the metrics created in the spanmetrics connector

   Label	Description	Values
   service_name	Service name set by the otel_service_name environment variable.	frontend
   span_name	Name of the operation.	/ /customer
   span_kind	Identifies the server, client, messaging, or internal operation.	SPAN_KIND_SERVER SPAN_KIND_CLIENT SPAN_KIND_PRODUCER SPAN_KIND_CONSUMER SPAN_KIND_INTERNAL

   Example PrometheusRule custom resource that defines an alerting rule for SLO when not serving 95% of requests within 2000ms on the front-end service

   Copy to Clipboard Toggle word wrap

   apiVersion: monitoring.coreos.com/v1
   kind: PrometheusRule
   metadata:
     name: span-red
   spec:
     groups:
     - name: server-side-latency
       rules:
       - alert: SpanREDFrontendAPIRequestLatency
         expr: histogram_quantile(0.95, sum(rate(duration_bucket{service_name="frontend", span_kind="SPAN_KIND_SERVER"}[5m])) by (le, service_name, span_name)) > 2000 

   1

   
         labels:
           severity: Warning
         annotations:
           summary: "High request latency on {{$labels.service_name}} and {{$labels.span_name}}"
           description: "{{$labels.instance}} has 95th request latency above 2s (current value: {{$value}}s)"

   1

   The expression for checking if 95% of the front-end server response time values are below 2000 ms. The time range ([5m]) must be at least four times the scrape interval and long enough to accommodate a change in the metric.