Prometheus 监控与告警规则实战

概览与核心价值Prometheus 提供强大的时序监控与规则计算能力。通过 Recording/Alert 规则与 Alertmanager，可实现服务可用性、延时与错误率的可靠监控与告警。关键规则示例Recording Rules（聚合加速）groups: - name: recording.rules rules: - record: job:http_request_duration_seconds:p95 expr: histogram_quantile(0.95, sum(rate(http_request_duration_seconds_bucket[5m])) by (le, job)) - record: job:http_error_rate expr: sum(rate(http_requests_total{status=~"5.."}[5m])) by (job) / sum(rate(http_requests_total[5m])) by (job) Alert 规则（错误率与可用性）groups: - name: alerts.rules rules: - alert: HighErrorRate expr: job:http_error_rate > 0.05 for: 10m labels: severity: critical annotations: summary: 高错误率告警 description: "{{ $labels.job }} 错误率超过 5% 持续 10 分钟" - alert: ServiceUnavailable expr: sum(rate(http_requests_total{status=~"5..|4(0[13])"}[5m])) by (job) > 0 for: 5m labels: severity: warning annotations: summary: 服务可用性告警 description: "{{ $labels.job }} 存在不可用行为，需排查" Alertmanager 路由route: group_by: ['alertname'] group_wait: 30s group_interval: 5m repeat_interval: 2h receivers: - name: default webhook_configs: - url: http://ops.example.com/alerts 参数与验证环境：`Prometheus v2.47+`、`Alertmanager v0.27+`。验证点：录制规则减少查询开销，仪表盘响应更快错误率 > 5% 持续 10m 触发告警，路由正常送达P95 时延曲线可用，趋势与业务观察一致最佳实践使用 Recording Rules 为复杂聚合提供缓存告警设置 `for` 避免瞬时波动误报与服务指标定义（SLI/SLO）对齐阈值与窗口结论通过 Recording 与 Alert 规则，结合 Alertmanager 路由，可构建稳定可靠的监控与告警系统，指标与阈值可验证与可审计。