ELK日志分析平台搭建实战：Elasticsearch索引管理与Kibana可视化查询配置

Elasticsearch集群部署与索引生命周期管理

日志是SRE稳定性工程和DevOps实践的基础数据。ELK Stack（Elasticsearch + Logstash + Kibana）是构建集中式日志分析平台的标准方案，在容器编排和微服务架构中广泛应用。Elasticsearch作为底层存储和检索引擎，集群部署配置直接影响整个日志平台的可用性和查询性能。

Elasticsearch集群通过分片（Shard）和副本（Replica）实现数据分布和高可用。生产环境建议最少3个节点，配置1个主分片和1个副本，保证单节点故障时数据不丢失。索引生命周期管理（ILM）自动处理索引的创建、滚动、归档和删除，避免手动维护的运维负担。

# elasticsearch.yml 核心配置
cluster.name: log-prod
node.name: node-1
path.data: /data/es
path.logs: /var/log/es
network.host: 0.0.0.0
http.port: 9200
discovery.seed_hosts: ["10.0.1.11", "10.0.1.12", "10.0.1.13"]
cluster.initial_master_nodes: ["node-1", "node-2", "node-3"]
xpack.security.enabled: true
xpack.security.transport.ssl.enabled: true

# 创建ILM策略
PUT _ilm/policy/logs-policy
{
  "policy": {
    "phases": {
      "hot": {
        "min_age": "0ms",
        "actions": {
          "rollover": {
            "max_size": "50gb",
            "max_age": "7d"
          },
          "set_priority": {"priority": 100}
        }
      },
      "warm": {
        "min_age": "7d",
        "actions": {
          "forcemerge": {"max_num_segments": 1},
          "set_priority": {"priority": 50}
        }
      },
      "delete": {
        "min_age": "30d",
        "actions": {"delete": {}}
      }
    }
  }
}

hot阶段设置索引滚动条件，当单个索引达到50GB或7天时自动创建新索引。warm阶段对历史索引执行force merge减少段数量，提升查询效率。delete阶段在30天后自动删除过期索引，控制存储成本。

Logstash管道配置与日志解析过滤

Logstash负责日志数据的采集、过滤和输出。通过input、filter、output三段式管道配置实现数据处理流程。filter中的grok模式匹配是日志解析的核心环节：

input {
  beats {
    port => 5044
  }
}

filter {
  if [service] == "nginx" {
    grok {
      match => {
        "message" => "%{IPORHOST:client_ip} - %{DATA:user} \[%{HTTPDATE:timestamp}\] "%{WORD:method} %{URIPATHPARAM:path} HTTP/%{NUMBER:http_version}" %{NUMBER:status} %{NUMBER:bytes} "%{DATA:referer}" "%{DATA:agent}" %{NUMBER:request_time}"
      }
    }
    date {
      match => ["timestamp", "dd/MMM/yyyy:HH:mm:ss Z"]
      target => "@timestamp"
    }
    mutate {
      convert => { "status" => "integer" }
      convert => { "bytes" => "integer" }
      convert => { "request_time" => "float" }
    }
  }

  if [service] == "app" and [message] =~ /^\{/ {
    json {
      source => "message"
      target => "log"
    }
  }
}

output {
  elasticsearch {
    hosts => ["https://10.0.1.11:9200"]
    user => "elastic"
    password => "${ES_PASSWORD}"
    index => "%{[@metadata][beat]}-%{+YYYY.MM.dd}"
    ilm_enabled => true
    ilm_rollover_alias => "app-logs"
    ilm_pattern => "{now/d}-000001"
    ilm_policy => "logs-policy"
  }
}

grok基于正则表达式提取结构化字段，预定义了HTTP日志、系统日志等常用模式。date插件将日志中的时间戳映射到@timestamp字段，确保Kibana时间轴显示正确。mutate插件做类型转换，将status、bytes等字段从字符串转为数值，便于后续聚合统计。

Kibana可视化仪表盘与查询DSL实战

Kibana提供日志查询、可视化和告警功能。Kibana Query Language（KQL）支持快速字段过滤和布尔查询：

# KQL查询示例
status:>=500 and service:nginx
response.time:>1000 and path:"/api/*"
not status:200 and client.ip:10.0.1.*

# Elasticsearch DSL聚合查询
GET /app-logs-*/_search
{
  "size": 0,
  "query": {
    "range": {
      "@timestamp": {
        "gte": "now-1h",
        "lte": "now"
      }
    }
  },
  "aggs": {
    "status_codes": {
      "terms": { "field": "status", "size": 10 }
    },
    "avg_response_time": {
      "avg": { "field": "response_time" }
    },
    "time_series": {
      "date_histogram": {
        "field": "@timestamp",
        "fixed_interval": "1m"
      },
      "aggs": {
        "error_rate": {
          "filter": { "range": { "status": { "gte": 500 } } }
        }
      }
    }
  }
}

date_histogram聚合按时间维度分桶统计，结合filter子聚合计算错误率，可在仪表盘中渲染实时错误率趋势图。设置告警规则监控5xx错误率超过阈值时触发通知，接入钉钉或飞书Webhook实现应急响应。

Filebeat轻量级日志采集配置

Filebeat是Elastic官方推荐的轻量级日志采集器，资源占用远低于Logstash，适合在每台服务器上部署：

filebeat.inputs:
  - type: filestream
    id: nginx-access
    paths:
      - /var/log/nginx/access.log
    fields:
      service: nginx
      env: production
    fields_under_root: true

  - type: filestream
    id: app-log
    paths:
      - /opt/myapp/logs/*.log
    fields:
      service: app
    multiline.pattern: '^\d{4}-\d{2}-\d{2}'
    multiline.negate: true
    multiline.match: after

output.logstash:
  hosts: ["10.0.1.20:5044"]
  loadbalance: true
  bulk_max_size: 2048

processors:
  - drop_fields:
      fields: ["agent.ephemeral_id", "agent.id", "agent.version"]
  - add_host_metadata: ~

multiline配置处理Java异常堆栈等多行日志，以日期开头的行作为新日志的起始行，后续行合并到前一条日志。drop_fields处理器移除不需要的元数据字段，减少网络传输量和索引存储空间。

ELK平台性能调优与容量规划

Elasticsearch JVM堆内存建议设为物理内存的50%，但不超过32GB以利用指针压缩。index.refresh_interval默认1秒，日志场景可调至30秒降低写入压力。批量写入时设置bulk size在5-15MB之间，过大会导致JVM GC压力。集群节点数控制在奇数以避免脑裂，master节点和数据节点分离部署提升稳定性。