ELK Stack日志收集分析平台搭建与Elasticsearch索引优化实战

分布式系统架构下，应用日志分散在数十上百台服务器上，传统的SSH+grep方式无法满足实时排查和关联分析需求。ELK Stack（Elasticsearch + Logstash + Kibana）是目前最主流的日志分析平台方案，配合Beats轻量采集器构成完整的日志收集、存储、分析和可视化链路。这套架构支撑着DevOps实践中从故障应急响应到全局可观测性的核心能力。

ELK Stack架构设计与日志采集链路规划

生产环境推荐使用Filebeat替代Logstash作为日志采集器，避免Logstash在每台节点部署带来的资源开销：

• Filebeat：部署在应用服务器，读取日志文件并转发
• Logstash：集中部署，负责日志解析、过滤和字段转换
• Elasticsearch：存储与全文检索引擎
• Kibana：可视化界面与仪表盘

# 架构示意
应用服务器          Logstash集群       Elasticsearch集群      Kibana
[App + Filebeat] -> [Logstash:5044] -> [ES:9200 x3节点] -> [:5601]
[App + Filebeat] ->
[App + Filebeat] ->

Elasticsearch集群部署与配置

# /etc/elasticsearch/elasticsearch.yml
cluster.name: elk-prod
node.name: es-node-1
node.roles: [data, master, ingest]

network.host: 0.0.0.0
http.port: 9200
transport.port: 9300

discovery.seed_hosts: ["10.0.1.11:9300", "10.0.1.12:9300", "10.0.1.13:9300"]
cluster.initial_master_nodes: ["es-node-1", "es-node-2", "es-node-3"]

path.data: /data/elasticsearch/data
path.logs: /var/log/elasticsearch

# JVM堆内存（jvm.options中配置）
# -Xms16g
# -Xmx16g

# 系统参数调整（每个ES节点执行）
echo "elasticsearch  -  nofile  65535" >> /etc/security/limits.conf
sysctl -w vm.max_map_count=262144
echo "vm.max_map_count=262144" >> /etc/sysctl.conf
swapoff -a

Logstash管道配置与Grok模式解析

# /etc/logstash/conf.d/app-logs.conf
input {
  beats {
    port => 5044
  }
}

filter {
  if [fields][log_type] == "nginx_access" {
    grok {
      match => {
        "message" => '%{IPORHOST:client_ip} - %{DATA:user} \[%{HTTPDATE:timestamp}\] "%{WORD:method} %{URIPATHPARAM:request} HTTP/%{NUMBER:http_version}" %{NUMBER:status} %{NUMBER:bytes} "%{DATA:referrer}" "%{DATA:user_agent}" %{NUMBER:request_time}'
      }
      add_tag => ["nginx_access_parsed"]
    }
    useragent { source => "user_agent", target => "ua_info" }
    geoip { source => "client_ip", target => "geo_info" }
  }
  
  if [fields][log_type] == "app_json" {
    json { source => "message", target => "app_log" }
    date { match => ["[app_log][timestamp]", "ISO8601"], target => "@timestamp" }
  }
  
  mutate {
    remove_field => ["message", "input_type", "offset", "@version"]
    convert => { "status" => "integer", "bytes" => "integer", "request_time" => "float" }
  }
}

output {
  elasticsearch {
    hosts => ["10.0.1.11:9200", "10.0.1.12:9200", "10.0.1.13:9200"]
    index => "%{[fields][index_prefix]}-%{+YYYY.MM.dd}"
  }
}

Filebeat采集器配置与多日志源收集

# /etc/filebeat/filebeat.yml
filebeat.inputs:
- type: filestream
  id: nginx-access
  paths:
    - /var/log/nginx/access.log
  fields:
    log_type: nginx_access
    index_prefix: nginx
  fields_under_root: true

- type: filestream
  id: app-json
  paths:
    - /opt/myapp/logs/*.json
  fields:
    log_type: app_json
    index_prefix: myapp
  fields_under_root: true
  parsers:
    - ndjson:
        target: ""
        overwrite_keys: true

output.logstash:
  hosts: ["10.0.2.10:5044"]
  loadbalance: true
  worker: 4
  bulk_max_size: 2048

queue.mem:
  events: 4096
  flush.min_events: 1024
  flush.timeout: 5s

Elasticsearch索引模板与分片策略优化

# PUT /_index_template/app-logs-template
{
  "index_patterns": ["nginx-*", "myapp-*", "system-*"],
  "template": {
    "settings": {
      "number_of_shards": 3,
      "number_of_replicas": 1,
      "refresh_interval": "30s",
      "index.codec": "best_compression"
    },
    "mappings": {
      "dynamic": false,
      "properties": {
        "@timestamp": { "type": "date" },
        "message": { "type": "text", "analyzer": "ik_max_word" },
        "client_ip": { "type": "ip" },
        "status": { "type": "integer" },
        "request_time": { "type": "float" }
      }
    }
  }
}

分片数量原则：单个分片大小建议控制在30GB-50GB。按日均日志量估算：日100GB日志对应3-4个分片。分片过多导致集群管理开销增大，过少则影响并行查询能力。replicas设为1即可满足高可用要求。

索引生命周期管理ILM与存储成本控制

# 创建ILM策略
PUT /_ilm/policy/logs-policy
{
  "policy": {
    "phases": {
      "hot": {
        "actions": {
          "rollover": { "max_size": "50gb", "max_age": "1d" },
          "set_priority": { "priority": 100 }
        }
      },
      "warm": {
        "min_age": "3d",
        "actions": {
          "forcemerge": { "max_num_segments": 1 },
          "shrink": { "number_of_shards": 1 },
          "set_priority": { "priority": 50 }
        }
      },
      "cold": {
        "min_age": "14d",
        "actions": { "freeze": {}, "set_priority": { "priority": 0 } }
      },
      "delete": {
        "min_age": "30d",
        "actions": { "delete": {} }
      }
    }
  }
}

Kibana仪表盘配置与日志查询实战

# Kibana KQL查询语法示例

# 按状态码过滤HTTP 5xx错误
status >= 500

# 多条件组合查询
log_type: "nginx_access" and status >= 500 and request_time > 2

# 模糊匹配URL路径
request: *api* and status: 404

# 按服务名和时间范围查询
app_log.service: "user-service" and app_log.level: "ERROR"
  and @timestamp >= "2026-08-15T00:00:00"
  and @timestamp < "2026-08-16T00:00:00"

性能监控与集群健康检查

# 集群健康状态
curl -s "localhost:9200/_cluster/health?pretty"

# 节点资源使用情况
curl -s "localhost:9200/_cat/nodes?v&h=name,heap.percent,ram.percent,cpu,load_1m,disk.used_percent"

# 索引大小和文档数
curl -s "localhost:9200/_cat/indices/nginx-*?v&h=index,docs.count,store.size"

# 查询慢日志配置
PUT /_all/_settings
{
  "index.search.slowlog.threshold.query.warn": "2s",
  "index.search.slowlog.threshold.query.info": "1s"
}

# 监控关键指标
# - heap使用率 < 75%
# - CPU使用率 < 80%
# - 磁盘使用率 < 85%
# - 查询延迟P99 < 500ms
# - 索引延迟P99 < 1s