Как обрабатывать incidents в production LLM: model / provider degradation, hallucination spike, fallback, runbook?

Provider outage / 5xx / 429 → polling https://status.openai.com / https://status.anthropic.com + auto-fallback на secondary provider. Latency spike P95 > SLO 5 min → PagerDuty. Hallucination spike (online eval faithfulness drop) → check recent prompt deploys / model swap / retrieval change. Cost spike 10× → retry storm / loop bug. Runbook templates + Sentry + post-mortem обязательны.