Cheat SheetSOP Step 21 — Reference Card
Forward-Deployed Engineer — Telemetry, Observability & Diagnostic Metrics
Production monitoring thresholds, error budgets, telemetry signals, and evaluation formulas.
Golden Signals for Forward-Deployed Engineer Production systems require real-time observability across four foundational telemetry vectors:
- Latency (p50, p95, p99): Measure end-to-end request durations including queue wait time and external dependencies.
- Error Rate: Ratio of non-2xx responses or invalid output generations relative to total throughput.
- Saturation: Utilization percentages of CPU, memory, connection pools, and upstream rate limit quotas.
- Fidelity / Quality Drift: Automated scoring drift against baseline evaluation golden sets.
---
Threshold Table
| Metric | Warning Threshold | Critical Incident Threshold | Recommended Remediation |
|---|---|---|---|
| **p99 Latency** | > 1.5x Baseline | > 3.0x Baseline | Enable cache fallbacks; degrade non-critical features |
| **Upstream Error Ratio** | > 2% over 5m | > 5% over 1m | Circuit break upstream calls; switch to secondary provider |
| **Output Schema Violation** | > 0.5% | > 2% | Force stricter output constraints; freeze prompt updates |