全链路可观测性实战：OpenTelemetry + Grafana Tempo + Loki构建统一监控体系

2026年7月15日互联网

可观测性三支柱的统一之路

在微服务架构下，传统的监控方式已无法应对复杂调用链路带来的排查难题。日志（Logs）、指标（Metrics）、链路追踪（Traces）被称为可观测性三支柱，但长期以来它们由不同的工具采集和存储，缺乏关联。OpenTelemetry作为CNCF的毕业项目，定义了统一的采集标准和API，配合Grafana生态的Tempo（追踪）、Loki（日志）、Mimir（指标），可以实现三支柱的深度关联，让一次排查同时覆盖全链路。

本文将从零搭建一套完整的可观测性体系，涵盖数据采集、存储、查询和告警的全流程。

架构设计与组件选型

整体架构：应用 → OpenTelemetry Collector → Tempo/Loki/Mimir → Grafana。Collector作为数据枢纽，负责接收、处理和路由三类遥测数据。

# docker-compose.yaml 核心配置
version: '3.8'
services:
  otel-collector:
    image: otel/opentelemetry-collector-contrib:0.96.0
    command: ["--config=/etc/otel-collector-config.yaml"]
    volumes:
      - ./otel-collector-config.yaml:/etc/otel-collector-config.yaml
    ports:
      - "4317:4317"   # gRPC OTLP
      - "4318:4318"   # HTTP OTLP
      - "8889:8889"   # Prometheus exporter

  tempo:
    image: grafana/tempo:2.3.1
    command: ["-config.file=/etc/tempo.yaml"]
    volumes:
      - ./tempo.yaml:/etc/tempo.yaml
    ports:
      - "3200:3200"

  loki:
    image: grafana/loki:2.9.3
    command: ["-config.file=/etc/loki.yaml"]
    volumes:
      - ./loki.yaml:/etc/loki.yaml
    ports:
      - "3100:3100"

  mimir:
    image: grafana/mimir:2.11.0
    command: ["-config.file=/etc/mimir.yaml"]
    volumes:
      - ./mimir.yaml:/etc/mimir.yaml
    ports:
      - "9009:9009"

  grafana:
    image: grafana/grafana:10.3.1
    ports:
      - "3000:3000"
    environment:
      - GF_SECURITY_ADMIN_PASSWORD=admin

OpenTelemetry Collector配置

Collector是整个体系的核心，需要配置接收器、处理器和导出器：

# otel-collector-config.yaml
receivers:
  otlp:
    protocols:
      grpc:
        endpoint: 0.0.0.0:4317
      http:
        endpoint: 0.0.0.0:4318

processors:
  batch:
    send_batch_size: 1024
    timeout: 5s
  memory_limiter:
    check_interval: 1s
    limit_percentage: 80
  # 关联日志与追踪
  transform/traces:
    error_mode: ignore
    trace_statements:
      - context: span
        statements:
          - set(attributes["deployment.environment"], "production")

exporters:
  otlphttp/tempo:
    endpoint: http://tempo:4318
  loki:
    endpoint: http://loki:3100/loki/api/v1/push
    default_labels_enabled: false
  prometheusremotewrite/mimir:
    endpoint: http://mimir:9009/api/v1/push

service:
  pipelines:
    traces:
      receivers: [otlp]
      processors: [memory_limiter, batch]
      exporters: [otlphttp/tempo]
    logs:
      receivers: [otlp]
      processors: [memory_limiter, transform/traces, batch]
      exporters: [loki]
    metrics:
      receivers: [otlp]
      processors: [memory_limiter, batch]
      exporters: [prometheusremotewrite/mimir]

应用侧接入SDK

以Java Spring Boot应用为例，使用OpenTelemetry Java Agent实现零代码侵入的自动instrumentation：

# Dockerfile
FROM openjdk:17-slim
COPY target/app.jar /app.jar

# 下载OTel Java Agent
ADD https://github.com/open-telemetry/opentelemetry-java-instrumentation/releases/latest/download/opentelemetry-javaagent.jar /otel-agent.jar

ENTRYPOINT ["java",   "-javaagent:/otel-agent.jar",   "-Dotel.service.name=order-service",   "-Dotel.exporter.otlp.endpoint=http://otel-collector:4317",   "-Dotel.resource.attributes=service.namespace=prod,deployment.environment=production",   "-Dotel.traces.sampler=parentbased_traceidratio",   "-Dotel.traces.sampler.arg=0.1",   "-jar", "/app.jar"]

关键配置解释：sampler=parentbased_traceidratio设置采样率为10%，适合高流量服务；错误span会被优先保留。对于Go/Python应用，OpenTelemetry也提供了对应的自动和手动instrumentation库。

日志与追踪的关联

实现Logs和Traces的深度关联，需要在日志中注入trace_id和span_id。以Logback为例：

<!-- logback-spring.xml -->
<appender name="JSON" class="ch.qos.logback.core.ConsoleAppender">
  <encoder class="net.logstash.logback.encoder.LogstashEncoder">
    <includeMdcKeyName>trace_id</includeMdcKeyName>
    <includeMdcKeyName>span_id</includeMdcKeyName>
    <includeMdcKeyName>service.name</includeMdcKeyName>
  </encoder>
</appender>
</pre>

<p>OpenTelemetry Java Agent会自动将trace_id和span_id注入MDC（Mapped Diagnostic Context），Logback编码器将其输出到JSON日志中。Loki收到这些日志后，Grafana可以根据trace_id直接跳转到Tempo查看完整链路。</p>

<h2>Grafana关联查询配置</h2>
<p>在Grafana中配置三个数据源的关联规则，实现从指标到追踪、从追踪到日志的无缝跳转：</p>
<pre><code># Tempo数据源配置 - 启用Trace to Logs
# 在Grafana UI中: Configuration → Data Sources → Tempo
# Trace to Logs设置:
datasource: Loki
tags: ['service.name', 'deployment.environment']
mappedTags: [{ key: 'service.name', value: 'service' }]
mapTagNamesEnabled: true
spanStartTimeShift: '-1h'
spanEndTimeShift: '1h'
filterByTraceID: true
filterBySpanID: true

# Loki数据源配置 - 启用Logs to Trace
# Derived fields设置:
name: TraceID
matcherRegex: '"trace_id":"(\w+)"'
url: '$${__value.raw}'
datasource: Tempo

配置完成后，在Grafana面板中点击任意trace_id即可跳转到Tempo的瀑布图，从span展开即可查看对应时间窗口的Loki日志，实现真正的全链路关联分析。

告警规则设计

基于Mimir指标和Tempo追踪数据，设计分级告警：

# Grafana告警规则示例
# 1. P99延迟告警
- uid: p99-latency-alert
  title: "P99延迟超过阈值"
  condition: C
  data:
    - refId: A
      relativeTimeRange:
        from: 300
        to: 0
      datasourceUid: mimir
      model:
        expr: histogram_quantile(0.99, sum(rate(http_server_duration_bucket{service_name="order-service"}[5m])) by (le))
        instant: true
    - refId: B
      relativeTimeRange:
        from: 300
        to: 0
      datasourceUid: __expr__
      model:
        type: reduce
        expression: A
        reducer: last
    - refId: C
      relativeTimeRange:
        from: 300
        to: 0
      datasourceUid: __expr__
      model:
        type: threshold
        expression: B
        conditions:
          - evaluator:
              params: [2.0]
              type: gt
            operator:
              type: and

性能优化与成本控制

可观测性数据量可能非常庞大，需要从采集、传输、存储三个环节控制成本：

# Collector采样与过滤处理器
processors:
  filter/health:
    error_mode: ignore
    traces:
      span:
        - 'attributes["http.route"] == "/healthz"'
        - 'attributes["http.route"] == "/readyz"'
  probabilistic_sampler:
    sampling_percentage: 10

# Tempo对象存储后端（生产环境替换filesystem）
storage:
  trace:
    backend: s3
    s3:
      bucket: tempo-traces
      endpoint: s3.amazonaws.com
    block:
      bloom_filter_false_positive: 0.05
      index_downsample_bytes: 1000

建议：健康检查路径的span在Collector层直接过滤；采样率根据服务重要性分级设置（核心服务100%，辅助服务10%）；Tempo使用对象存储降低长期存储成本。

总结

OpenTelemetry + Grafana生态的可观测性体系，通过统一的采集标准和深度关联的存储后端，彻底解决了微服务排查中"日志查不到、链路对不上、指标看不懂"的痛点。Collector作为数据枢纽，让所有应用只需对接一套SDK；Grafana作为统一入口，让三支柱数据自由流转。这套方案的投入产出比极高，建议从核心服务开始逐步接入，用可观测性驱动SRE实践。