Prometheus远程写入与Thanos多集群监控联邦架构部署实战

小编 • • 网站运维

Prometheus是云原生监控体系的核心组件，单实例Prometheus存在数据持久化困难、跨集群查询能力缺失、高可用方案复杂等问题。当Kubernetes集群规模增长到多集群阶段，运维团队需要统一的监控视图来追踪全局SRE稳定性指标。Thanos通过Sidecar长期存储、Remote Write远程写入和Query Federation查询联邦三种模式扩展Prometheus，实现多集群监控数据的统一存储和查询。

Thanos架构组件与部署模式选择

Thanos核心组件包括Sidecar/Receiver（数据采集）、Store Gateway（历史数据查询）、Compactor（数据降采样）、Query（查询聚合）。三种部署模式适用场景不同。

# Thanos组件二进制安装
wget https://github.com/thanos-io/thanos/releases/download/v0.35.0/thanos-0.35.0.linux-amd64.tar.gz
tar xzf thanos-0.35.0.linux-amd64.tar.gz
mv thanos /usr/local/bin/

# 组件版本验证
thanos --version

模式一：Sidecar模式（单集群长期存储）

Sidecar与Prometheus同Pod部署，通过–tsdb.path读取Prometheus本地TSDB数据块，上传到对象存储。适合每个集群已运行Prometheus且需要长期存储的场景。

# Prometheus启动参数增加外部标签
cat > /etc/prometheus/prometheus.yml << 'EOF'
global:
  scrape_interval: 15s
  external_labels:
    cluster: "prod-cluster-1"
    region: "us-east-1"

remote_write:
  - url: "http://thanos-receiver:19291/api/v1/receive"

scrape_configs:
  - job_name: "kubernetes-nodes"
    kubernetes_sd_configs:
      - role: node
    relabel_configs:
      - source_labels: [__address__]
        regex: "(.*):.*"
        target_label: __address__
        replacement: "$1:9100"
  
  - job_name: "kubernetes-pods"
    kubernetes_sd_configs:
      - role: pod
    relabel_configs:
      - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_scrape]
        action: keep
        regex: true
EOF

# Prometheus启用生命周期API（Sidecar需要）
/usr/local/bin/prometheus \
  --config.file=/etc/prometheus/prometheus.yml \
  --storage.tsdb.path=/prometheus/data \
  --storage.tsdb.retention.time=2h \
  --web.enable-lifecycle \
  --web.enable-admin-api

# Thanos Sidecar
thanos sidecar \
  --tsdb.path=/prometheus/data \
  --prometheus.url=http://localhost:9090 \
  --objstore.config-file=/etc/thanos/objstore.yaml \
  --http-address=0.0.0.0:10902 \
  --grpc-address=0.0.0.0:10901

Remote Write远程写入模式配置与对象存储集成

Remote Write模式将Prometheus采集的指标实时推送到Thanos Receiver，避免了Sidecar模式中Prometheus本地存储的依赖。适合大规模集群和边缘节点场景。

# 对象存储配置（MinIO S3兼容）
cat > /etc/thanos/objstore.yaml << 'EOF'
type: S3
config:
  bucket: "thanos-metrics"
  endpoint: "minio.monitoring.svc:9000"
  access_key: "thanos-access-key"
  secret_key: "thanos-secret-key"
  insecure: true
  put_user_metadata:
    "X-Amz-Storage-Class": "STANDARD"
  trace:
    enable: false
EOF

# Thanos Receiver（接收Remote Write数据）
thanos receive \
  --remote-write.address=0.0.0.0:19291 \
  --grpc-address=0.0.0.0:10901 \
  --http-address=0.0.0.0:10902 \
  --objstore.config-file=/etc/thanos/objstore.yaml \
  --label=receive_replica="$(hostname)" \
  --tsdb.path=/var/thanos/receive \
  --tsdb.retention=15d \
  --receive.replication-factor=2

# Prometheus端配置Remote Write
cat > /etc/prometheus/prometheus.yml << 'EOF'
global:
  scrape_interval: 15s
  evaluation_interval: 15s
  external_labels:
    cluster: "edge-cluster-tokyo"
    environment: "production"

remote_write:
  - url: "http://thanos-receiver:19291/api/v1/receive"
    queue_config:
      capacity: 10000
      max_samples_per_send: 2000
      max_shards: 30
      min_shards: 5
    remote_timeout: 30s
    write_relabel_configs:
      - source_labels: [__name__]
        regex: "go_.*|process_.*"
        action: drop
EOF

Store Gateway与Compactor数据降采样配置

Store Gateway从对象存储读取历史数据块供Thanos Query查询。Compactor对历史数据进行降采样（5分钟和1小时精度），减少长期数据查询的扫描量。

# Store Gateway
thanos store \
  --data-dir=/var/thanos/store \
  --objstore.config-file=/etc/thanos/objstore.yaml \
  --http-address=0.0.0.0:10902 \
  --grpc-address=0.0.0.0:10901 \
  --cache-index-header \
  --index-cache-size=1GB \
  --chunk-pool-size=2GB

# Compactor（数据降采样与压缩）
thanos compact \
  --data-dir=/var/thanos/compact \
  --objstore.config-file=/etc/thanos/objstore.yaml \
  --http-address=0.0.0.0:10902 \
  --retention.resolution-raw=30d \
  --retention.resolution-5m=180d \
  --retention.resolution-1h=365d \
  --wait \
  --wait-interval=5m \
  --deduplication.replica-label=receive_replica

# 降采样说明：
# raw (原始15s精度): 保留30天
# 5m (5分钟降采样): 保留180天
# 1h (1小时降采样): 保留365天
# 查询30天以上数据时自动使用降采样数据，大幅减少响应时间

Thanos Query多集群查询联邦与PromQL跨集群聚合

Thanos Query通过gRPC连接所有Sidecar/Receiver/Store Gateway，对外暴露PromQL接口，实现跨集群查询。StoreAPI使得Query可以透明地合并来自不同集群的数据。

# Thanos Query
thanos query \
  --http-address=0.0.0.0:10902 \
  --grpc-address=0.0.0.0:10901 \
  --query.replica-label=receive_replica \
  --query.max-concurrent=20 \
  --query.timeout=2m \
  --endpoint=sidecar-1.cluster1:10901 \
  --endpoint=sidecar-2.cluster2:10901 \
  --endpoint=receiver-1:10901 \
  --endpoint=receiver-2:10901 \
  --endpoint=store-gateway:10901 \
  --endpoint=store-gateway-2:10901

# 跨集群PromQL查询示例
# 1. 查询所有集群的CPU使用率（按cluster标签聚合）
# avg by (cluster) (1 - avg(rate(node_cpu_seconds_total{mode="idle"}[5m])) by (cluster, instance))

# 2. 查询特定集群的内存使用率
# (1 - (node_memory_MemAvailable_bytes / node_memory_MemTotal_bytes)) * 100

# 3. 跨集群Pod重启告警
# rate(kube_pod_container_status_restarts_total[15m]) > 0

# 4. 全局HTTP请求延迟P99（跨集群聚合）
# histogram_quantile(0.99, sum by (le, cluster) (rate(http_request_duration_seconds_bucket[5m])))

Grafana数据源对接与多集群监控面板配置

# Grafana数据源配置（Thanos Query作为Prometheus数据源）
apiVersion: 1
datasources:
  - name: Thanos-Query
    type: prometheus
    access: proxy
    url: http://thanos-query:10902
    isDefault: true
    jsonData:
      timeInterval: "15s"
      httpMethod: POST
      prometheusType: "Thanos"
      prometheusVersion: "0.35.0"

# 多集群CPU使用率面板PromQL
# avg by (cluster) (1 - avg(rate(node_cpu_seconds_total{mode="idle"}[5m])) by (cluster, instance)) * 100

# 多集群内存使用率面板
# avg by (cluster) (1 - (node_memory_MemAvailable_bytes / node_memory_MemTotal_bytes)) * 100

# 跨集群网络流量面板
# sum by (cluster) (rate(node_network_receive_bytes_total{device=~"eth.*"}[5m])) / 1024 / 1024

告警规则联邦与Alertmanager路由分发

多集群监控体系中，告警规则可以部署在Thanos Ruler或各集群本地Prometheus上。推荐分层部署：基础设施告警在各集群本地Prometheus（减少Remote Write延迟影响），业务告警在Thanos Ruler（跨集群聚合判断）。

# Thanos Ruler
thanos rule \
  --data-dir=/var/thanos/ruler \
  --eval-interval=30s \
  --rule-file=/etc/thanos/rules/*.yaml \
  --alertmanagers.url=alertmanager:9093 \
  --query=thanos-query:10902 \
  --objstore.config-file=/etc/thanos/objstore.yaml \
  --http-address=0.0.0.0:10902 \
  --grpc-address=0.0.0.0:10901

# 跨集群告警规则示例
cat > /etc/thanos/rules/cross-cluster.yaml << 'EOF'
groups:
  - name: cross-cluster-alerts
    interval: 30s
    rules:
      - alert: ClusterNodeDown
        expr: count by (cluster) (up{job="kubernetes-nodes"} == 0) > 0
        for: 2m
        labels:
          severity: critical
        annotations:
          summary: "集群 {{ $labels.cluster }} 有节点离线"
          description: "集群 {{ $labels.cluster }} 当前有 {{ $value }} 个节点无法访问"
      
      - alert: HighMemoryUsageAcrossClusters
        expr: |
          avg by (cluster) (
            (1 - node_memory_MemAvailable_bytes / node_memory_MemTotal_bytes) * 100
          ) > 85
        for: 10m
        labels:
          severity: warning
        annotations:
          summary: "集群 {{ $labels.cluster }} 内存使用率过高"
          description: "集群 {{ $labels.cluster }} 内存使用率持续10分钟超过85%，当前值: {{ $value }}%"
EOF

# Alertmanager路由配置 - 按集群分发到不同接收组
cat > /etc/alertmanager/config.yml << 'EOF'
route:
  receiver: default
  group_by: ['cluster', 'alertname']
  group_wait: 30s
  group_interval: 5m
  repeat_interval: 4h
  routes:
    - matchers:
        - cluster="prod-cluster-1"
      receiver: team-platform
    - matchers:
        - cluster="edge-cluster-tokyo"
      receiver: team-edge

receivers:
  - name: default
    webhook_configs:
      - url: 'http://dingtalk-webhook:8060/dingtalk/ops/send'
  - name: team-platform
    webhook_configs:
      - url: 'http://dingtalk-webhook:8060/dingtalk/platform/send'
  - name: team-edge
    webhook_configs:
      - url: 'http://dingtalk-webhook:8060/dingtalk/edge/send'
EOF

完成上述部署后，Thanos Query对外提供统一PromQL查询入口，Grafana通过单一数据源访问所有集群的实时和历史监控数据。Store Gateway配合Compactor降采样，1年内的历史数据查询延迟控制在5秒以内。Remote Write模式下的数据延迟（从采集到可查询）通常在30-60秒之间，满足大部分告警和看板需求。对于延迟敏感的基础设施告警，仍建议保留各集群本地Prometheus + Alertmanager独立告警链路。

原创文章，作者：小编，如若转载，请注明出处：https://www.yunthe.com/prometheus-yuan-cheng-xie-ru-yu-thanos-duo-ji-qun-jian-kong/

赞 (0)
小编小编
0
上一篇 5小时前
下一篇 5小时前

相关推荐