Loki日志分析平台部署实战：Promtail采集与Grafana可视化查询方案

Loki是Grafana Labs开源的日志聚合系统，采用与Prometheus相同的标签索引模型，仅索引日志流的元数据（标签）而非全文内容，存储成本比ELK低一个数量级。配合Promtail采集器和Grafana可视化，构成轻量级日志分析平台。本文给出Loki部署、Promtail配置、日志查询和告警的完整方案。

Loki架构设计与存储后端配置

Loki采用微服务架构，核心组件包括Distributor（日志写入入口）、Ingester（日志写入缓冲）、Querier（日志查询）、Compactor（数据压缩合并）和Ruler（告警规则评估）。存储后端支持本地文件系统、S3、GCS等。小规模部署使用Single Binary模式，所有组件运行在一个进程中。

# docker-compose.yml - Loki单节点部署
version: '3.8'
services:
  loki:
    image: grafana/loki:3.2.0
    ports:
      - "3100:3100"
    volumes:
      - ./loki-config.yml:/etc/loki/local-config.yaml
      - loki-data:/loki
    command: -config.file=/etc/loki/local-config.yaml
    restart: unless-stopped

  grafana:
    image: grafana/grafana:11.3.0
    ports:
      - "3000:3000"
    volumes:
      - grafana-data:/var/lib/grafana
    environment:
      - GF_SECURITY_ADMIN_PASSWORD=admin
    restart: unless-stopped

volumes:
  loki-data:
  grafana-data:

Loki配置文件定义存储、保留周期和写入限制：

# loki-config.yml
auth_enabled: false

server:
  http_listen_port: 3100

common:
  path_prefix: /loki
  storage:
    filesystem:
      chunks_directory: /loki/chunks
      rules_directory: /loki/rules
  replication_factor: 1
  ring:
    kvstore:
      store: inmemory

schema_config:
  configs:
    - from: 2024-01-01
      store: tsdb
      object_store: filesystem
      schema: v13
      index:
        prefix: index_
        period: 24h

limits_config:
  retention_period: 168h          # 日志保留7天
  max_query_series: 5000
  max_query_parallelism: 16
  reject_old_samples: true
  reject_old_samples_max_age: 168h
  ingestion_rate_mb: 15           # 每租户每秒写入上限15MB
  ingestion_burst_size_mb: 30
  max_streams_per_user: 10000

compactor:
  working_directory: /loki/compactor
  retention_enabled: true
  retention_delete_delay: 2h
  delete_request_store: filesystem

Promtail日志采集与标签提取

Promtail是Loki官方日志采集Agent，支持文件采集、systemd日志和Docker容器日志。核心配置包括采集目标（scrape_configs）、管道阶段（pipeline_stages）和写入地址。

# promtail-config.yml
server:
  http_listen_port: 9080
  grpc_listen_port: 0

positions:
  filename: /tmp/positions.yaml

clients:
  - url: http://loki:3100/loki/api/v1/push
    backoff_config:
      min_period: 500ms
      max_period: 5m
      max_retries: 10

scrape_configs:
  # 采集Nginx访问日志
  - job_name: nginx
    static_configs:
      - targets: [localhost]
        labels:
          job: nginx
          host: web-01
          __path__: /var/log/nginx/*.log
    pipeline_stages:
      - regex:
          expression: '(?P\d+\.\d+\.\d+\.\d+) - \S+ \[(?P[^\]]+)\] "(?P\S+) (?P\S+) (?P[^"]+)" (?P\d+) (?P\d+) "(?P[^"]*)" "(?P[^"]*)"'
      - labels:
          status:
          method:
      - timestamp:
          source: timestamp
          format: '02/Jan/2006:15:04:05 -0700'

  # 采集Docker容器日志
  - job_name: docker
    docker_sd_configs:
      - host: unix:///var/run/docker.sock
        refresh_interval: 5s
        filters:
          - name: label
            values: ["logging=loki"]
    pipeline_stages:
      - docker:
      - labels:
          container_name:
      - json:
          expressions:
            level: level
            service: service
            trace_id: trace_id
      - labels:
          level:
          service:
      - output:
          source: message

  # 采集应用多行日志（Java堆栈跟踪）
  - job_name: java-app
    static_configs:
      - targets: [localhost]
        labels:
          job: java-app
          app: order-service
          __path__: /opt/app/logs/*.log
    pipeline_stages:
      - multiline:
          firstline: '^\d{4}-\d{2}-\d{2} \d{2}:\d{2}:\d{2}'
          max_wait_time: 3s
      - regex:
          expression: '(?P\d{4}-\d{2}-\d{2} \d{2}:\d{2}:\d{2}) (?P\w+) \[(?P[^\]]+)\] (?P\S+) - (?P.*)'
      - labels:
          level:
      - timestamp:
          source: timestamp
          format: '2006-01-02 15:04:05'

pipeline_stages的关键作用是将非结构化日志转为结构化标签。regex阶段提取字段，labels阶段将字段转为Loki索引标签，timestamp阶段解析日志时间戳。标签数量直接影响查询效率，建议只将高频查询字段（status、level、service）转为标签，避免标签基数爆炸。

LogQL日志查询语法与聚合分析

LogQL是Loki的查询语言，语法分为日志流选择器和过滤管道两部分。Grafana Explore面板直接执行LogQL查询：

# 查询特定服务的所有日志
{job="nginx", host="web-01"}

# 按HTTP状态码过滤
{job="nginx", status="500"}

# 全文正则搜索
{job="nginx"} |= "timeout"

# 多条件组合过滤
{job="nginx"} |= "error" != "healthcheck" | status="500"

# 提取字段并过滤
{job="java-app", level="ERROR"}
  | regexp "(?P\w+Exception)"
  | exception = "NullPointerException"

# 统计每分钟日志条数
sum by (host) (rate({job="nginx"}[1m]))

# 统计HTTP 5xx错误率
sum by (host) (rate({job="nginx", status=~"5.."}[5m]))
  /
sum by (host) (rate({job="nginx"}[5m]))

# 计算P99响应时间（从日志中提取duration字段）
quantile_over_time(0.99,
  {job="nginx"}
  | regexp "duration=(?P[0-9.]+)"
  | unwrap duration [5m]
) by (host)

# Top 10 请求路径
topk(10, sum by (path) (
  count_over_time({job="nginx"} | regexp "(?P\S+) HTTP" [1h])
))

Loki告警规则与Grafana仪表板集成

Loki Ruler组件支持类似Prometheus的告警规则。规则定义在rules目录下：

# /loki/rules/alerts.yml
groups:
  - name: nginx_alerts
    rules:
      - alert: NginxHighErrorRate
        expr: |
          sum by (host) (rate({job="nginx", status=~"5.."}[5m]))
          /
          sum by (host) (rate({job="nginx"}[5m]))
          > 0.05
        for: 3m
        labels:
          severity: critical
        annotations:
          summary: "{{ $labels.host }} Nginx 5xx错误率超过5%"

      - alert: JavaAppExceptionSpike
        expr: |
          sum by (app) (rate({job="java-app", level="ERROR"}[5m])) > 10
        for: 2m
        labels:
          severity: warning
        annotations:
          summary: "{{ $labels.app }} ERROR日志速率超过10条/秒"

      - alert: NoLogsReceived
        expr: |
          count_over_time({job="nginx"}[10m]) == 0
        for: 5m
        labels:
          severity: warning
        annotations:
          summary: "10分钟内未收到Nginx日志，Promtail可能已停止"

告警通过Alertmanager路由到不同通知渠道。Grafana仪表板中将Loki日志与Prometheus指标关联展示——指标图表点击某时刻数据点，下方面板自动展开对应时段日志，实现从指标异常到日志定位的一键跳转。

# Alertmanager配置
route:
  group_by: ['alertname', 'host']
  group_wait: 30s
  group_interval: 5m
  repeat_interval: 4h
  receiver: 'default'
  routes:
    - match:
        severity: critical
      receiver: 'pagerduty'
    - match:
        severity: warning
      receiver: 'slack'

receivers:
  - name: 'default'
    webhook_configs:
      - url: 'http://dingtalk-webhook/dingtalk/send'
  - name: 'pagerduty'
    pagerduty_configs:
      - service_key: '${PAGERDUTY_KEY}'
  - name: 'slack'
    slack_configs:
      - api_url: '${SLACK_WEBHOOK}'
        channel: '#ops-alerts'

Loki相比ELK的核心优势在于存储成本。同样的日志量，Elasticsearch索引全文内容，存储开销约为原始日志的3-5倍；Loki仅索引标签，存储开销约为原始日志的1-1.5倍，Gzip压缩后更低。查询性能上，Loki在大范围时间扫描（如查看某服务全天的日志流）时延迟与ELK相当，但在全文检索复杂关键词时因无倒排索引而较慢。适用场景判断：日志量巨大且以标签维度查询为主的运维场景选Loki；需要全文检索和复杂聚合分析的业务分析场景选ELK。