Observability and Evaluation
Connect traces, prompts, tool calls, quality scores, latency, and cost.
Written and reviewed by Eric Han · Co-founder & CTO, reopt
Key takeaways
- AI observability must answer both operational and product questions: did it work, was it useful, what did it cost, and where did it fail.
- The signal map connects request traces, prompt and model metadata, tool calls, retrieval results, user feedback, and evaluation scores.
- Run a review loop: inspect failures and cost outliers, compare quality with feedback, find the cause, ship one controlled change, then re-run evaluation and watch for drift.
- Show quality, latency, cost, and error rate together, since optimizing one metric in isolation often damages another.
Register AI SDK 7 telemetry
pnpm add @ai-sdk/otelOpenTelemetry collection moved from ai to @ai-sdk/otel. Register the integration once at application startup, alongside the existing provider/exporter:
import { registerTelemetry } from 'ai'
import { OpenTelemetry } from '@ai-sdk/otel'
export function register() {
registerTelemetry(new OpenTelemetry())
// Connect the application's existing OTel provider and exporter here too.
}Calls emit telemetry by default after an integration is registered. Apply per-call policy with telemetry, review captured inputs and outputs, and verify actual span receipt at the exporter. Renaming an option alone does not verify observability.
AI observability must answer both operational and product questions: did it work, was it useful, how much did it cost, and where did it fail?
Signal Map
| Signal | Use |
|---|---|
| Request trace | Debug latency and failures |
| Prompt and model metadata | Explain behavior and cost |
| Tool calls | Audit side effects and data access |
| Retrieval results | Diagnose grounding problems |
| User feedback | Measure usefulness |
| Evaluation scores | Track quality over time |
Review Loop
- Inspect failures and high-cost outliers.
- Compare quality metrics with user feedback.
- Identify prompt, retrieval, model, or tool cause.
- Ship one controlled improvement.
- Re-run evaluation and monitor drift.
Dashboard Rule
Show quality, latency, cost, and error rate together. Optimizing one metric in isolation often damages another.