日志集中化收集与分析是SRE稳定性工程的基础环节。ELK Stack（Elasticsearch、Logstash、Kibana）配合Filebeat，构建了从日志采集、传输、解析到可视化的完整流水线。在海量日志场景下，日志分析平台的架构设计和配置优化直接影响故障应急响应的效率。DevOps实践中，结构化日志的采集与分析能力是CI/CD流水线和监控告警体系之外的重要可观测性支柱。

ELK Stack架构设计与组件选型

现代ELK架构采用Filebeat + Elasticsearch + Kibana三层结构。Filebeat作为轻量级日志采集器部署在每个应用节点上，直接将日志发送到Elasticsearch，不再经过Logstash中转。Logstash的角色转变为数据预处理管道，仅在需要复杂过滤和转换时使用。

组件职责划分：Filebeat负责日志文件的tail跟踪和行组装；Elasticsearch负责日志的全文索引存储和搜索；Kibana负责查询界面和可视化仪表盘。三者版本必须保持一致，推荐使用8.x版本，内置安全认证和ILM索引生命周期管理。

Elasticsearch集群部署与索引配置

Elasticsearch 8.x默认开启安全认证，部署时需要配置节点间通信证书。日志索引采用按天滚动策略，配合ILM自动管理索引的生命周期。

# docker-compose.yml 部署单节点ES（开发环境）
version: '3.8'
services:
  elasticsearch:
    image: docker.elastic.co/elasticsearch/elasticsearch:8.12.0
    environment:
      - discovery.type=single-node
      - xpack.security.enabled=true
      - ELASTIC_PASSWORD=YourStrongPass123
      - ES_JAVA_OPTS=-Xms2g -Xmx2g
    ports:
      - "9200:9200"
    volumes:
      - es_data:/usr/share/elasticsearch/data

  kibana:
    image: docker.elastic.co/kibana/kibana:8.12.0
    environment:
      - ELASTICSEARCH_HOSTS=http://elasticsearch:9200
      - ELASTICSEARCH_USERNAME=elastic
      - ELASTICSEARCH_PASSWORD=YourStrongPass123
    ports:
      - "5601:5601"
    depends_on:
      - elasticsearch

索引模板配置，为日志索引定义映射和设置：

PUT _index_template/app-logs
{
  "index_patterns": ["app-logs-*"],
  "template": {
    "settings": {
      "number_of_shards": 1,
      "number_of_replicas": 1,
      "index.lifecycle.name": "logs-policy",
      "index.refresh_interval": "5s"
    },
    "mappings": {
      "properties": {
        "@timestamp": { "type": "date" },
        "level": { "type": "keyword" },
        "service": { "type": "keyword" },
        "host": { "type": "keyword" },
        "message": { "type": "text" },
        "trace_id": { "type": "keyword" },
        "duration_ms": { "type": "long" }
      }
    }
  }
}

PUT _ilm/policy/logs-policy
{
  "policy": {
    "phases": {
      "hot": {
        "actions": {
          "rollover": {
            "max_age": "1d",
            "max_primary_shard_size": "10gb"
          }
        }
      },
      "delete": {
        "min_age": "7d",
        "actions": {
          "delete": {}
        }
      }
    }
  }
}

Filebeat采集配置与多日志源接入

Filebeat包含两个核心组件：prospector（发现日志文件）和harvester（逐行读取文件内容）。每个harvester使用一个goroutine持续读取文件，并记录每个文件的读取偏移量到registry文件中，实现断点续传。

# filebeat.yml 核心配置
filebeat.inputs:
# 采集Nginx访问日志（JSON格式）
- type: filestream
  id: nginx-access
  paths:
    - /var/log/nginx/access.log
  parsers:
    - ndjson:
        target: nginx
        overwrite_keys: true

# 采集Java应用日志（多行合并）
- type: filestream
  id: java-app
  paths:
    - /opt/app/logs/*.log
  parsers:
    - multiline:
        type: pattern
        pattern: '^\d{4}-\d{2}-\d{2}'
        negate: true
        match: after

# 采集容器日志
- type: container
  paths:
    - /var/lib/docker/containers/*/*.log
  processors:
    - add_docker_metadata:
        host: "unix:///var/run/docker.sock"

output.elasticsearch:
  hosts: ["elasticsearch:9200"]
  username: "elastic"
  password: "YourStrongPass123"
  indices:
    - index: "app-logs-%{[agent][type]}"
      when.contains:
        agent.type: "filestream"

processors:
  - add_host_metadata:
      when.not.contains.tags: forwarded
  - add_fields:
      target: ''
      fields:
        env: production
  - drop_fields:
      fields: ["agent.ephemeral_id", "agent.id", "agent.version"]
  - decode_json_fields:
      fields: ["message"]
      target: "json"
      overwrite_keys: true

path.data: /var/lib/filebeat
path.logs: /var/log/filebeat

Kibana仪表盘配置与KQL查询

在Kibana中创建Index Pattern后，可以使用Kibana Query Language（KQL）进行日志检索，并构建可视化仪表盘用于日常运维监控。

# KQL查询语法示例

# 按日志级别过滤
level: "ERROR" or level: "FATAL"

# 按服务和时间范围过滤
service: "order-service" and @timestamp >= "now-1h"

# 全文搜索异常堆栈
message: "NullPointerException" and service: "payment-service"

# 范围查询（响应时间超过1秒的慢请求）
duration_ms >= 1000 and service: "api-gateway"

# Trace ID追踪
trace_id: "abc123def456"

构建运维仪表盘时添加以下可视化组件：

# 1. 日志级别分布饼图
# 聚合: Terms on level field

# 2. 日志量时间序列图
# X轴: @timestamp (date histogram, 1分钟间隔)
# Y轴: Count, 按service字段分series

# 3. 慢请求Top 10表格
# 聚合: Terms on service field
# Metric: avg(duration_ms), 降序排列

# 4. 错误日志热力图
# X轴: @timestamp (hourly)
# Y轴: service
# Metric: count where level=ERROR

# 通过Kibana API批量创建仪表盘
curl -u elastic:YourStrongPass123 -X POST \
  "localhost:5601/api/saved_objects/dashboard/ops-overview" \
  -H 'kbn-xsrf: true' -H 'Content-Type: application/json' -d '{
  "attributes": { "title": "运维监控总览" }
}'

告警规则配置与故障通知

Kibana 8.x内置Alerting功能，可以基于ES查询条件配置告警规则。当日志匹配特定模式时，通过邮件、Webhook或Slack发送通知。

# 创建告警规则：ERROR日志5分钟内超过50条
POST /api/detection_engine/rules
{
  "name": "High Error Rate Alert",
  "params": {
    "index": ["app-logs-*"],
    "esqlQuery": "FROM app-logs-* | WHERE level == \"ERROR\" | STATS count = COUNT(*) BY service",
    "threshold": 50,
    "timeWindowSize": 300000,
    "timeWindowUnit": "ms"
  },
  "actions": [
    {
      "action_type_id": ".webhook",
      "params": {
        "url": "https://hooks.slack.com/services/xxx",
        "body": "{{context.alert}}"
      }
    }
  ]
}

ELK日志分析平台的价值在于将分散在各节点上的日志汇聚到统一的分析平台。Filebeat的多行合并和JSON解析能力使得非结构化日志也能被有效索引。Kibana的KQL查询和仪表盘可视化将日志检索效率从分钟级降到秒级。配合ILM索引生命周期管理和告警规则，日志平台本身也具备了自动化运维能力，无需人工干预即可完成日志归档和异常通知。