告警系统
OpenClaw告警系统为平台提供了全面的监控和通知能力。当Agent出现异常、系统资源超限或关键指标偏离预期时,告警系统会立即通过配置的渠道通知相关人员。本章将详细介绍告警规则的定义、通知渠道的配置、告警路由和升级策略。
告警规则定义
告警规则是告警系统的核心,它定义了何时触发告警以及告警的严重程度。OpenClaw支持基于阈值、趋势和复合条件的告警规则。
# 告警规则配置
alerts:
rules:
# 基于阈值的告警
- name: high_error_rate
description: "Agent错误率超过阈值"
condition:
metric: agent.error_rate
operator: ">"
threshold: 0.05
window: 5m
severity: critical
labels:
team: platform
component: agent-runtime
- name: high_latency
description: "Agent响应延迟过高"
condition:
metric: agent.latency_p99
operator: ">"
threshold: 10000 # 10秒
window: 5m
severity: warning
labels:
team: platform
# 基于趋势的告警
- name: error_rate_increasing
description: "错误率持续上升"
condition:
metric: agent.error_rate
trend: increasing
periods: 3
comparison: previous_period
severity: warning
# 复合条件告警
- name: system_overload
description: "系统过载"
condition:
operator: AND
conditions:
- metric: cpu.usage
operator: ">"
threshold: 90
- metric: memory.usage
operator: ">"
threshold: 85
- metric: agent.active_tasks
operator: ">"
threshold: 100
severity: critical
核心概念:告警严重程度
OpenClaw定义了四个告警严重程度级别:info(信息,无需立即处理)、warning(警告,需要关注)、critical(严重,需要尽快处理)和 fatal(致命,需要立即处理)。不同级别可以路由到不同的通知渠道和响应团队。
通知渠道配置
OpenClaw支持多种通知渠道,您可以根据告警的严重程度和接收者的偏好配置不同的通知方式。
邮件通知
# 邮件通知配置
channels:
email:
smtp:
host: smtp.example.com
port: 587
tls: true
username_env: SMTP_USERNAME
password_env: SMTP_PASSWORD
from: openclaw-alerts@example.com
templates:
subject: "[{{severity}}] {{rule_name}} - {{agent_id}}"
body: |
告警名称: {{rule_name}}
严重程度: {{severity}}
Agent: {{agent_id}}
触发时间: {{timestamp}}
详情: {{description}}
当前值: {{current_value}}
阈值: {{threshold}}
Webhook通知
# Webhook通知配置
channels:
webhook:
- name: pagerduty
url: https://events.pagerduty.com/v2/enqueue
method: POST
headers:
Content-Type: application/json
body_template: |
{
"routing_key": "{{pagerduty_key}}",
"event_action": "trigger",
"payload": {
"summary": "{{rule_name}}: {{description}}",
"severity": "{{severity}}",
"source": "openclaw",
"component": "{{agent_id}}"
}
}
- name: custom_webhook
url: ${WEBHOOK_URL}
method: POST
retry:
max_attempts: 3
backoff: exponential
Slack通知
# Slack通知配置
channels:
slack:
- name: ops_channel
webhook_url: ${SLACK_WEBHOOK_URL}
channel: "#ops-alerts"
username: "OpenClaw Alert"
icon_emoji: ":warning:"
template:
text: |
*{{severity_icon}} {{rule_name}}*
Agent: `{{agent_id}}`
详情: {{description}}
当前值: {{current_value}} (阈值: {{threshold}})
时间: {{timestamp}}
severity_icons:
info: ":information_source:"
warning: ":warning:"
critical: ":fire:"
fatal: ":boom:"
企业微信通知
# 企业微信通知配置
channels:
wechat_work:
- name: ops_team
webhook_url: ${WECHAT_WORK_WEBHOOK}
message_type: markdown
template: |
## {{severity_icon}} {{rule_name}}
> **Agent**: {{agent_id}}
> **严重程度**: {{severity}}
> **详情**: {{description}}
> **当前值**: {{current_value}}
> **触发时间**: {{timestamp}}
[查看详情]({{dashboard_url}})
告警路由与升级
告警路由决定了告警消息发送给谁,而告警升级机制确保未处理的告警会在适当的时候升级到更高级别的响应者。
# 告警路由和升级配置
routing:
rules:
# 关键告警立即通知值班人员
- match:
severity: critical
channels:
- pagerduty
- slack:ops_channel
- wechat_work:ops_team
escalation:
- after: 5m
if_not_acknowledged: true
notify:
- email:team-lead@example.com
- slack:tech-lead-channel
- after: 15m
if_not_resolved: true
notify:
- email:engineering-director@example.com
- phone:+86-138-xxxx-xxxx
# 警告级别通知到团队频道
- match:
severity: warning
channels:
- slack:ops_channel
escalation:
- after: 30m
if_not_acknowledged: true
channels:
- email:team@example.com
# 按Agent路由
- match:
agent_id: research-bot-*
channels:
- slack:research-team
- match:
agent_id: coding-bot-*
channels:
- slack:engineering-team
提示:静默时间
您可以配置告警静默时间(maintenance window),在计划维护期间暂时屏蔽特定告警。例如:silence: {rule: high_latency, from: "2024-03-15 02:00", to: "2024-03-15 04:00", timezone: Asia/Shanghai}
告警去重
告警去重机制防止同一问题产生大量重复告警,避免通知疲劳。OpenClaw提供了多种去重策略:
- 时间窗口去重:在同一时间窗口内,相同规则的告警只发送一次
- 分组聚合:将相关告警合并为一条通知,例如同一Agent的多个指标异常
- 抑制规则:当高级别告警触发时,抑制相关的低级别告警
# 告警去重配置
deduplication:
# 时间窗口去重
time_window:
default: 5m
overrides:
high_error_rate: 15m
system_overload: 10m
# 分组聚合
grouping:
group_by:
- agent_id
- severity
wait: 30s # 等待30秒收集同组告警
send_resolved: true
# 抑制规则
inhibition:
- source_match:
severity: critical
target_match:
severity: warning
equal:
- agent_id
警告:告警疲劳
过多的告警会导致"告警疲劳",使团队对真正的紧急情况反应迟钝。建议定期审查告警规则,调整阈值以减少误报,并确保每条告警都有明确的响应行动。一般来说,每周触发的告警数量应该控制在可处理的范围内。
通过合理配置告警规则、通知渠道和升级策略,OpenClaw告警系统可以帮助您的团队快速发现和响应平台运行中的各种问题,确保AI智能体服务的稳定性和可靠性。