告警系统

OpenClaw告警系统为平台提供了全面的监控和通知能力。当Agent出现异常、系统资源超限或关键指标偏离预期时,告警系统会立即通过配置的渠道通知相关人员。本章将详细介绍告警规则的定义、通知渠道的配置、告警路由和升级策略。

告警规则定义

告警规则是告警系统的核心,它定义了何时触发告警以及告警的严重程度。OpenClaw支持基于阈值、趋势和复合条件的告警规则。

# 告警规则配置
alerts:
  rules:
    # 基于阈值的告警
    - name: high_error_rate
      description: "Agent错误率超过阈值"
      condition:
        metric: agent.error_rate
        operator: ">"
        threshold: 0.05
        window: 5m
      severity: critical
      labels:
        team: platform
        component: agent-runtime

    - name: high_latency
      description: "Agent响应延迟过高"
      condition:
        metric: agent.latency_p99
        operator: ">"
        threshold: 10000  # 10秒
        window: 5m
      severity: warning
      labels:
        team: platform

    # 基于趋势的告警
    - name: error_rate_increasing
      description: "错误率持续上升"
      condition:
        metric: agent.error_rate
        trend: increasing
        periods: 3
        comparison: previous_period
      severity: warning

    # 复合条件告警
    - name: system_overload
      description: "系统过载"
      condition:
        operator: AND
        conditions:
          - metric: cpu.usage
            operator: ">"
            threshold: 90
          - metric: memory.usage
            operator: ">"
            threshold: 85
          - metric: agent.active_tasks
            operator: ">"
            threshold: 100
      severity: critical

核心概念:告警严重程度

OpenClaw定义了四个告警严重程度级别:info(信息,无需立即处理)、warning(警告,需要关注)、critical(严重,需要尽快处理)和 fatal(致命,需要立即处理)。不同级别可以路由到不同的通知渠道和响应团队。

通知渠道配置

OpenClaw支持多种通知渠道,您可以根据告警的严重程度和接收者的偏好配置不同的通知方式。

邮件通知

# 邮件通知配置
channels:
  email:
    smtp:
      host: smtp.example.com
      port: 587
      tls: true
      username_env: SMTP_USERNAME
      password_env: SMTP_PASSWORD
    from: openclaw-alerts@example.com
    templates:
      subject: "[{{severity}}] {{rule_name}} - {{agent_id}}"
      body: |
        告警名称: {{rule_name}}
        严重程度: {{severity}}
        Agent: {{agent_id}}
        触发时间: {{timestamp}}
        详情: {{description}}
        当前值: {{current_value}}
        阈值: {{threshold}}

Webhook通知

# Webhook通知配置
channels:
  webhook:
    - name: pagerduty
      url: https://events.pagerduty.com/v2/enqueue
      method: POST
      headers:
        Content-Type: application/json
      body_template: |
        {
          "routing_key": "{{pagerduty_key}}",
          "event_action": "trigger",
          "payload": {
            "summary": "{{rule_name}}: {{description}}",
            "severity": "{{severity}}",
            "source": "openclaw",
            "component": "{{agent_id}}"
          }
        }

    - name: custom_webhook
      url: ${WEBHOOK_URL}
      method: POST
      retry:
        max_attempts: 3
        backoff: exponential

Slack通知

# Slack通知配置
channels:
  slack:
    - name: ops_channel
      webhook_url: ${SLACK_WEBHOOK_URL}
      channel: "#ops-alerts"
      username: "OpenClaw Alert"
      icon_emoji: ":warning:"
      template:
        text: |
          *{{severity_icon}} {{rule_name}}*
          Agent: `{{agent_id}}`
          详情: {{description}}
          当前值: {{current_value}} (阈值: {{threshold}})
          时间: {{timestamp}}
      severity_icons:
        info: ":information_source:"
        warning: ":warning:"
        critical: ":fire:"
        fatal: ":boom:"

企业微信通知

# 企业微信通知配置
channels:
  wechat_work:
    - name: ops_team
      webhook_url: ${WECHAT_WORK_WEBHOOK}
      message_type: markdown
      template: |
        ## {{severity_icon}} {{rule_name}}
        > **Agent**: {{agent_id}}
        > **严重程度**: {{severity}}
        > **详情**: {{description}}
        > **当前值**: {{current_value}}
        > **触发时间**: {{timestamp}}
        [查看详情]({{dashboard_url}})

告警路由与升级

告警路由决定了告警消息发送给谁,而告警升级机制确保未处理的告警会在适当的时候升级到更高级别的响应者。

# 告警路由和升级配置
routing:
  rules:
    # 关键告警立即通知值班人员
    - match:
        severity: critical
      channels:
        - pagerduty
        - slack:ops_channel
        - wechat_work:ops_team
      escalation:
        - after: 5m
          if_not_acknowledged: true
          notify:
            - email:team-lead@example.com
            - slack:tech-lead-channel

        - after: 15m
          if_not_resolved: true
          notify:
            - email:engineering-director@example.com
            - phone:+86-138-xxxx-xxxx

    # 警告级别通知到团队频道
    - match:
        severity: warning
      channels:
        - slack:ops_channel
      escalation:
        - after: 30m
          if_not_acknowledged: true
          channels:
            - email:team@example.com

    # 按Agent路由
    - match:
        agent_id: research-bot-*
      channels:
        - slack:research-team
    - match:
        agent_id: coding-bot-*
      channels:
        - slack:engineering-team

提示:静默时间

您可以配置告警静默时间(maintenance window),在计划维护期间暂时屏蔽特定告警。例如:silence: {rule: high_latency, from: "2024-03-15 02:00", to: "2024-03-15 04:00", timezone: Asia/Shanghai}

告警去重

告警去重机制防止同一问题产生大量重复告警,避免通知疲劳。OpenClaw提供了多种去重策略:

  • 时间窗口去重:在同一时间窗口内,相同规则的告警只发送一次
  • 分组聚合:将相关告警合并为一条通知,例如同一Agent的多个指标异常
  • 抑制规则:当高级别告警触发时,抑制相关的低级别告警
# 告警去重配置
deduplication:
  # 时间窗口去重
  time_window:
    default: 5m
    overrides:
      high_error_rate: 15m
      system_overload: 10m

  # 分组聚合
  grouping:
    group_by:
      - agent_id
      - severity
    wait: 30s  # 等待30秒收集同组告警
    send_resolved: true

  # 抑制规则
  inhibition:
    - source_match:
        severity: critical
      target_match:
        severity: warning
      equal:
        - agent_id

警告:告警疲劳

过多的告警会导致"告警疲劳",使团队对真正的紧急情况反应迟钝。建议定期审查告警规则,调整阈值以减少误报,并确保每条告警都有明确的响应行动。一般来说,每周触发的告警数量应该控制在可处理的范围内。

通过合理配置告警规则、通知渠道和升级策略,OpenClaw告警系统可以帮助您的团队快速发现和响应平台运行中的各种问题,确保AI智能体服务的稳定性和可靠性。