ELK Stack日志分析平台搭建与Filebeat日志采集管道配置实战

小编 • • 网站运维

ELK Stack架构设计与组件职责

ELK Stack由Elasticsearch、Logstash、Kibana三个核心组件构成，配合Filebeat实现轻量级日志采集。Elasticsearch负责日志数据的存储和全文检索，Logstash负责日志数据的解析和转换，Kibana提供可视化界面，Filebeat作为轻量级agent部署在各应用服务器上采集日志。在DevOps实践中，ELK Stack是日志分析体系的基础设施。

ELK Stack的数据流向：Filebeat从应用服务器采集原始日志，发送到Logstash进行解析过滤，Logstash将结构化数据写入Elasticsearch索引，Kibana从Elasticsearch查询数据并渲染可视化面板。整个管道支持水平扩展，Elasticsearch可配置集群模式应对海量日志存储。

Elasticsearch集群部署配置

使用Docker Compose部署三节点Elasticsearch集群：

# docker-compose.yml
version: '3.8'
services:
  es-node1:
    image: docker.elastic.co/elasticsearch/elasticsearch:8.12.0
    container_name: es-node1
    environment:
      - node.name=es-node1
      - cluster.name=elk-cluster
      - discovery.seed_hosts=es-node2,es-node3
      - cluster.initial_master_nodes=es-node1,es-node2,es-node3
      - bootstrap.memory_lock=true
      - xpack.security.enabled=false
      - ES_JAVA_OPTS=-Xms4g -Xmx4g
    ulimits:
      memlock:
        soft: -1
        hard: -1
    volumes:
      - es-data1:/usr/share/elasticsearch/data
    ports:
      - "9200:9200"
    networks:
      - elk

  es-node2:
    image: docker.elastic.co/elasticsearch/elasticsearch:8.12.0
    container_name: es-node2
    environment:
      - node.name=es-node2
      - cluster.name=elk-cluster
      - discovery.seed_hosts=es-node1,es-node3
      - cluster.initial_master_nodes=es-node1,es-node2,es-node3
      - bootstrap.memory_lock=true
      - xpack.security.enabled=false
      - ES_JAVA_OPTS=-Xms4g -Xmx4g
    volumes:
      - es-data2:/usr/share/elasticsearch/data
    networks:
      - elk

  es-node3:
    image: docker.elastic.co/elasticsearch/elasticsearch:8.12.0
    container_name: es-node3
    environment:
      - node.name=es-node3
      - cluster.name=elk-cluster
      - discovery.seed_hosts=es-node1,es-node2
      - cluster.initial_master_nodes=es-node1,es-node2,es-node3
      - bootstrap.memory_lock=true
      - xpack.security.enabled=false
      - ES_JAVA_OPTS=-Xms4g -Xmx4g
    volumes:
      - es-data3:/usr/share/elasticsearch/data
    networks:
      - elk

volumes:
  es-data1:
  es-data2:
  es-data3:

networks:
  elk:
    driver: bridge

验证Elasticsearch集群状态：

# 检查集群健康状态
curl -X GET "localhost:9200/_cluster/health?pretty"
# 检查节点列表
curl -X GET "localhost:9200/_cat/nodes?v"
# 检查索引列表
curl -X GET "localhost:9200/_cat/indices?v"

Logstash日志解析管道配置

Logstash的核心配置在pipeline.conf文件中，分为input、filter、output三个部分。以解析Nginx访问日志为例：

# /usr/share/logstash/pipeline/nginx-log.conf
input {
  beats {
    port => 5044
  }
}

filter {
  if [fileset][module] == "nginx" {
    # 解析Nginx access日志
    if [fileset][name] == "access" {
      grok {
        match => {
          "message" => '%{IPORHOST:remote_ip} - %{DATA:user_name} \[%{HTTPDATE:time_local}\] "%{WORD:method} %{DATA:request} HTTP/%{NUMBER:http_version}" %{NUMBER:status_code} %{NUMBER:body_bytes_sent} "%{DATA:referrer}" "%{DATA:agent}" rt=%{NUMBER:request_time} uct="%{DATA:upstream_connect_time}" uht="%{DATA:upstream_header_time}" urt="%{DATA:upstream_response_time}"'
        }
        remove_field => ["message"]
      }

      # 提取地理位置
      geoip {
        source => "remote_ip"
        target => "geoip"
      }

      # 转换数据类型
      mutate {
        convert => {
          "status_code" => "integer"
          "body_bytes_sent" => "integer"
          "request_time" => "float"
        }
      }

      # 提取时间戳
      date {
        match => ["time_local", "dd/MMM/yyyy:HH:mm:ss Z"]
        target => "@timestamp"
      }
    }

    # 解析Nginx error日志
    if [fileset][name] == "error" {
      grok {
        match => {
          "message" => '%{DATESTAMP:timestamp} \[%{LOGLEVEL:log_level}\] %{GREEDYDATA:error_message}'
        }
      }
    }
  }
}

output {
  elasticsearch {
    hosts => ["es-node1:9200", "es-node2:9200", "es-node3:9200"]
    index => "nginx-logs-%{+YYYY.MM.dd}"
    template => "/usr/share/logstash/config/nginx-template.json"
    template_name => "nginx-logs"
  }
}

Filebeat轻量级日志采集配置

Filebeat部署在应用服务器上，负责采集日志文件并发送到Logstash：

# /etc/filebeat/filebeat.yml
filebeat.inputs:
  # 采集Nginx access日志
  - type: filestream
    id: nginx-access
    enabled: true
    paths:
      - /var/log/nginx/access.log
    fields:
      log_type: nginx-access
    fields_under_root: true

  # 采集Nginx error日志
  - type: filestream
    id: nginx-error
    enabled: true
    paths:
      - /var/log/nginx/error.log
    fields:
      log_type: nginx-error
    fields_under_root: true

  # 采集应用日志（多行合并）
  - type: filestream
    id: app-log
    enabled: true
    paths:
      - /opt/app/logs/*.log
    parsers:
      - multiline:
          type: pattern
          pattern: '^\d{4}-\d{2}-\d{2}'
          negate: true
          match: after
    fields:
      log_type: application
    fields_under_root: true

# 输出到Logstash
output.logstash:
  hosts: ["192.168.1.100:5044"]
  loadbalance: true
  worker: 2

# Filebeat自身日志
logging.level: info
logging.to_files: true
logging.files:
  path: /var/log/filebeat
  keepfiles: 7

# 监控
monitoring.enabled: true
monitoring.elasticsearch:
  hosts: ["http://192.168.1.101:9200"]
# 启动Filebeat
systemctl enable filebeat
systemctl start filebeat

# 验证配置
filebeat test config
filebeat test output

# 检查Filebeat状态
filebeat status
# 查看采集的文件状态
curl -X GET "localhost:5066/stats/filebeat" | python -m json.tool

Kibana可视化面板配置

Kibana提供日志检索和可视化能力，配置Index Pattern后即可创建Dashboard：

# docker-compose.yml 中添加Kibana
  kibana:
    image: docker.elastic.co/kibana/kibana:8.12.0
    container_name: kibana
    environment:
      - ELASTICSEARCH_HOSTS=http://es-node1:9200
      - SERVER_NAME=kibana
      - XPACK_SECURITY_ENABLED=false
    ports:
      - "5601:5601"
    networks:
      - elk
    depends_on:
      - es-node1

在Kibana中创建Index Pattern：进入Stack Management > Index Patterns，创建匹配nginx-logs-*的pattern，设置@timestamp为时间字段。创建Dashboard可视化面板，常用图表包括：状态码分布饼图（status_code字段聚合）、请求时间趋势折线图（request_time字段平均值按时间聚合）、Top IP访问柱状图（remote_ip字段计数排序）、地理位置地图（geoip.location字段）。

日志索引生命周期管理ILM

生产环境中日志数据增长迅速，需要配置ILM策略自动滚动和清理旧索引：

# 创建ILM策略
PUT _ilm/policy/nginx-logs-policy
{
  "policy": {
    "phases": {
      "hot": {
        actions": {
          "rollover": {
            "max_size": "50gb",
            "max_age": "1d"
          }
        }
      },
      "warm": {
        "min_age": "7d",
        "actions": {
          "shrink": { "number_of_shards": 1 },
          "forcemerge": { "max_num_segments": 1 }
        }
      },
      "delete": {
        "min_age": "30d",
        "actions": {
          "delete": {}
        }
      }
    }
  }
}

# 创建索引模板绑定ILM策略
PUT _index_template/nginx-logs
{
  "index_patterns": ["nginx-logs-*"],
  "template": {
    "settings": {
      "index.lifecycle.name": "nginx-logs-policy",
      "number_of_shards": 3,
      "number_of_replicas": 1
    }
  }
}

ELK Stack日志分析平台的搭建需要各组件协同配置。Filebeat的轻量级设计使其适合部署在每台应用服务器上，Logstash的filter插件链可以实现复杂的日志解析和字段提取，Elasticsearch集群提供海量日志的存储和全文检索能力，Kibana则将日志数据转化为可视化面板。配合ILM策略管理索引生命周期，可以控制存储成本并保持查询性能。

原创文章，作者：小编，如若转载，请注明出处：https://www.yunthe.com/elkstack-ri-zhi-fen-xi-ping-tai-da-jian-yu-filebeat-ri-zhi/

赞 (0)
小编小编
0
上一篇 1小时前
下一篇 1小时前

相关推荐