Prometheus告警规则与SLO落地实践

概述SLO 将可用性与延迟目标转化为可度量与告警。本文提供 Recording Rule 与多窗口 Burn Rate 告警示例，并给出路由与抑制策略与验证方法。录制规则（已验证）groups: - name: recording rules: - record: job:http_request_total:rate1m expr: rate(http_requests_total[1m]) - record: job:http_error_ratio:rate1m expr: rate(http_requests_errors_total[1m]) / rate(http_requests_total[1m]) 多窗口Burn Rate告警groups: - name: alerts rules: - alert: SLOErrorBudgetBurnFast expr: (rate(http_requests_errors_total[5m]) / rate(http_requests_total[5m])) > 0.14 for: 5m labels: { severity: critical, slo: availability } annotations: summary: 快速燃尽错误预算（5m窗口） - alert: SLOErrorBudgetBurnSlow expr: (rate(http_requests_errors_total[1h]) / rate(http_requests_total[1h])) > 0.02 for: 2h labels: { severity: warning, slo: availability } annotations: summary: 缓慢燃尽错误预算（1h窗口） 阈值示例基于 99.9% 可用性目标（需结合业务验证）。Alertmanager 路由与抑制route: group_by: [ alertname, job ] group_wait: 30s group_interval: 5m repeat_interval: 4h receiver: oncall routes: - matchers: - severity="critical" receiver: pager - matchers: - slo="availability" receiver: slo-channel receivers: - name: oncall slack_configs: - channel: '#oncall' - name: pager pagerduty_configs: - routing_key: '<redacted>' - name: slo-channel slack_configs: - channel: '#slo' inhibit_rules: - source_matchers: [ severity="critical" ] target_matchers: [ severity="warning" ] equal: [ alertname, job ] 验证与指标使用 `promtool check rules` 校验语法与录制规则；观测触发与抑制行为、重复间隔与路由命中；SLO 仪表盘：错误率、Burn Rate、剩余错误预算与趋势；常见误区仅使用单窗口告警，易漏报或误报；未区分严重等级与路由，造成噪音过高；无录制规则导致查询复杂与成本上升。结语以录制规则与多窗口 Burn Rate 为核心，配合 Alertmanager 的分组与抑制策略，实现可靠的 SLO 告警与可执行运维闭环。