Prometheus高可用联邦集群架构与远程写入数据同步实战

Prometheus单实例在物理故障时会丢失监控数据并导致告警中断，生产环境必须构建高可用架构。Prometheus原生不支持集群化存储，通过联邦集群和远程写入组合方案实现数据冗余与统一视图。本文给出跨数据中心部署、数据去重、告警转发等实战配置。

Prometheus高可用架构选型分析

联邦适合节点规模1000以内、数据中心分布式的场景，远程写入适合超大规模或需要长期保留的场景。生产实践常将两者结合。

联邦集群架构部署与scrape配置

联邦架构分两级：每个数据中心部署本地Prometheus采集本地target；中心Prometheus通过/federate接口从各本地实例拉取聚合数据。

# 中心Prometheus - prometheus.yml
global:
  scrape_interval: 30s
  evaluation_interval: 30s

scrape_configs:
  - job_name: 'federate-dc1'
    scrape_interval: 30s
    honor_labels: true
    metrics_path: '/federate'
    params:
      'match[]':
        - '{__name__=~"job:.*"}'
        - '{__name__="up"}'
        - '{__name__=~"container_.*"}'
        - '{__name__=~"node_.*"}'
    static_configs:
      - targets: ['prometheus-dc1.internal:9090']
        labels:
          datacenter: 'dc1'

  - job_name: 'federate-dc2'
    scrape_interval: 30s
    honor_labels: true
    metrics_path: '/federate'
    params:
      'match[]':
        - '{__name__=~"job:.*"}'
        - '{__name__="up"}'
        - '{__name__=~"container_.*"}'
        - '{__name__=~"node_.*"}'
    static_configs:
      - targets: ['prometheus-dc2.internal:9090']
        labels:
          datacenter: 'dc2'

本地Prometheus通过recording rules预聚合高频指标以减少联邦拉取的数据量：

# recording_rules.yml
groups:
  - name: aggregation
    rules:
      - record: job:cpu_usage:rate5m
        expr: avg(rate(container_cpu_usage_seconds_total[5m])) by (job, instance)
      - record: job:memory_usage:bytes
        expr: sum(container_memory_working_set_bytes) by (job, instance)
      - record: job:http_request_rate:5m
        expr: sum(rate(http_requests_total[5m])) by (job, status_code)
      - record: instance:node_cpu_utilisation:ratio
        expr: 1 - (avg by (instance) (rate(node_cpu_seconds_total{mode="idle"}[5m])))

远程写入配置与数据压缩传输

远程写入将本地采集数据实时转发到远端存储（Thanos Receive、Mimir、Cortex等），解决本地存储容量限制：

remote_write:
  - url: 'http://thanos-receive.central.svc:19291/api/v1/receive'
    remote_timeout: 30s
    headers:
      X-Thanos-Tenant: 'dc1'
    write_relabel_configs:
      - source_labels: [__name__]
        regex: 'go_.*|process_.*|prometheus_.*'
        action: drop
    queue_config:
      capacity: 10000
      max_shards: 200
      min_shards: 10
      max_samples_per_send: 2000
      batch_send_deadline: 5s
    external_labels:
      datacenter: 'dc1'
      prometheus_replica: 'pod-A'

remote_read:
  - url: 'http://thanos-query.central.svc:9090/api/v1/query'
    read_recent: true
    timeout: 30s

监控远程写入健康的关键PromQL：

# 队列积压样本数
prometheus_remote_storage_queue_length
# 发送失败次数
rate(prometheus_remote_storage_dropped_samples_total[5m])
# 发送延迟P99
histogram_quantile(0.99, rate(prometheus_remote_storage_sent_batch_duration_seconds_bucket[5m]))
# 每秒发送样本数
rate(prometheus_remote_storage_samples_total[5m])

数据去重与external_labels设计

同一指标可能被多个Prometheus采集，去重依赖external_labels生成唯一标识：

global:
  external_labels:
    cluster: 'prod-k8s-01'
    datacenter: 'dc1'
    prometheus_replica: 'replica-A'
    environment: 'production'

# Thanos Query去重配置
# --query.replica-label=prometheus_replica
# 基于除replica label以外的所有labels去重

# PromQL去重查询
max by (instance, job, cluster, datacenter) (up{job="node"})

Alertmanager集群与告警去重

多Prometheus实例会产生重复告警，通过Alertmanager集群的Gossip协议实现去重：

# alertmanager.yml
route:
  group_by: ['alertname', 'cluster', 'datacenter']
  group_wait: 30s
  group_interval: 5m
  repeat_interval: 4h
  receiver: 'default'
  routes:
    - matchers: ['severity="critical"']
      receiver: 'pagerduty'
      group_wait: 10s

receivers:
  - name: 'pagerduty'
    pagerduty_configs:
      - routing_key: 'xxx'
        severity: critical

# 启动三节点集群
# alertmanager --config.file=alertmanager.yml \
#   --cluster.peer=am-1:9094 \
#   --cluster.peer=am-2:9094 \
#   --cluster.peer=am-3:9094 \
#   --cluster.listen-address=0.0.0.0:9094

联邦数据链路监控与故障排查

# 联邦scrape成功率
rate(prometheus_target_scrape_pool_total{job=~"federate-.*"}[5m])
# 联邦拉取延迟
scrape_duration_seconds{job=~"federate-.*"}
# 联邦拉取样本数
scrape_samples_post_metric_relabeling{job=~"federate-.*"}
# 远程写入积压
prometheus_remote_storage_queue_length

联邦数据不完整的排查：

# 直接调试联邦拉取
curl -G 'http://prometheus-dc1:9090/federate' \
  --data-urlencode 'match[]={__name__=~"node_.*"}' \
  -H 'Accept: application/openmetrics-text'

容量规划与扩缩容

# 单实例采集样本数（建议上限2M samples/min）
rate(prometheus_tsdb_head_samples_appended_total[5m]) * 60
# 本地TSDB存储空间（每百万样本约1.5-2GB）
prometheus_tsdb_size_bytes
# 远程写入带宽（每样本压缩后约30-50字节）
rate(prometheus_remote_storage_samples_total[5m]) * 40 / 1024 / 1024

当单实例samples/min超过2M时考虑sharding拆分，Thanos Receiver支持hash分片：

# Thanos Receive hash ring配置
[
  {"hashtag": 0, "endpoints": ["thanos-receive-0:19291"]},
  {"hashtag": 1, "endpoints": ["thanos-receive-1:19291"]}
]

监控系统的可用性直接影响告警可靠性，高可用架构本质上是在数据完整性、查询性能和运维成本之间做权衡。联邦集群解决跨数据中心汇聚，远程写入解决长期存储扩展性，Alertmanager集群解决告警可靠性，三者组合构成生产级监控高可用的标准方案。