Ch13. Observability and Deployment
Operate Eve with OpenTelemetry, Workflow tags, deployment checklists, health checks, and production runbooks.
Written and reviewed by Eric Han · Co-founder & CTO, reopt
Key takeaways
- Eve observability needs session, turn, step, tool, sandbox, subagent, and model-usage views.
- OpenTelemetry, Workflow tags, and runtime hooks are complementary surfaces.
- Vercel and self-host deployments both need artifact, health, session, and stream verification.
Run-data retention since 0.80
A session trace policy allowing both inputs and outputs uses the Workflow World's default retention. Dropping the trace or excluding either direction requests retention 0 for payloads and streams after the run ends. Outside development, private/unknown conversations request this deletion by default even without an OTel destination.
Set experimental.workflow.retention to 0 or "default" to choose retention independently. Actual deletion depends on World support. This does not erase active payloads, remaining IDs/timestamps/attributes, or external OTel/provider/tool/channel data. Handle 410 session_expired when a deleted session is read.
New Vercel projects created by eve deploy get 100% sampling; existing projects need their sampling rules checked separately.
Run-data retention · Agent Runs.
Durable agents cannot be operated with HTTP request logs alone. You need to see where a run is parked, what step failed, which tool ran, which sandbox backend was used, and how much model usage accumulated.
Three Observability Surfaces
| Surface | Location | Use |
|---|---|---|
| Workflow run tags | framework emitted | Agent Runs and session tree |
| OpenTelemetry | agent/instrumentation/ | export spans to observability backend |
| Runtime hooks | agent/hooks/** | audit, metrics, warehouse ingestion |
instrumentation.ts
The single instrumentation.ts API was removed. Replace it with this file and delete the old one.
import { otel } from "eve/instrumentation/otel";
export default otel({
functionId: "support-agent",
traceChannelRequests: true,
tracePolicy: () => ({
emit: true,
recordInputs: false,
recordOutputs: false,
}),
});Move each exporter to a destination file using otelIntegration() and remove startup registerOTel() calls. New defaults can capture content in development and public conversations when an OTel destination exists. Set capture policy explicitly and review each destination's exportPolicy, retention, and access controls.
Runtime Context Enrichment
Move runtimeContext from events["step.started"] into the destination's otelIntegration({ runtimeContext }). The eve. prefix is reserved. Narrow channel metadata types before reading them, and exclude secrets and personal data. Content excluded by tracePolicy is also unavailable to the resolver.
Workflow Tags
Do not treat the $eve.* tag list from the 0.11 source analysis as the current observability schema. Current OTel uses invoke_agent → agent.step → chat/execute_tool topology and structural session/turn identifiers. Correlate tools and approvals with gen_ai.tool.call.id; inspect error.type for failures.
Vercel Agent Runs is a separate destination whose exportPolicy can further restrict content. Manage mandatory audit ledgers and exported traces with distinct access and retention policies.
Vercel Deployment
eve build
vercel deployOn Vercel, Eve emits Vercel Build Output, Workflow runs on Vercel Workflow, and defaultBackend() selects Vercel Sandbox.
Smoke checks:
curl https://your-agent.example.com/eve/v1/health
eve remote connect --url https://your-agent.example.comSelf-host Deployment
eve build
PORT=3000 eve start --host 0.0.0.0| Concern | Standard |
|---|---|
| workflow state | persistent .eve/.workflow-data |
| model auth | AI Gateway key or direct provider key |
| route auth | replace Vercel OIDC with host-valid auth |
| sandbox | Docker, microsandbox, or custom backend |
| schedules | ensure Nitro scheduled tasks run |
| logs | process manager and log collector |
| TLS/routing | reverse proxy or platform |
Build Artifact Review
| Artifact | Check |
|---|---|
.eve/diagnostics.json | no unexpected warnings/errors |
agent-discovery-manifest.json | expected files only |
compiled-agent-manifest.json | tools, connections, channels, schedules, subagents |
module-map.mjs | compiled module resolution |
Runtime Runbook
| Symptom | Check |
|---|---|
| production 401 | route auth and placeholder removal |
stuck in turn.waiting | approval, question, or OAuth pending |
| tool not visible | eve info, dynamic resolver event, disabled default |
| missing subagent result | child stream and parent proxy input request |
| sandbox command failed | backend, network policy, bootstrap |
| cost spike | token tags, tool count, compaction |
| trace missing | instrumentation setup and exporter |
Production SLO Candidates
| SLO | Measurement |
|---|---|
| session start success | POST /session 2xx ratio |
| turn completion | turn.completed / started |
| no failed step | step.failed rate |
| approval latency | input.requested to answer |
| model latency | step.started to step.completed |
| cost per task | tokens + tool infrastructure |
| sandbox setup latency | first sandbox use |
| eval pass rate | CI and scheduled evals |
Deployment Checklist
| Gate | Standard |
|---|---|
| build | eve build succeeds |
| diagnostics | .eve/diagnostics.json clean |
| auth | production route auth fail-closed |
| secrets | not present in artifacts or workspace |
| sandbox | backend and network policy explicit |
| eval | eve eval --strict passes |
| smoke | health, session, and stream checks pass |
| observability | OTel or hook audit works |
| rollback | model/prompt/tool rollback documented |
Additional self-host checks
The default local World persists under .eve/.workflow-data. Forward both /eve/ and /.well-known/workflow/ through the proxy so callbacks arrive. Pin a World package compatible with Eve's Workflow 5 beta line and read credentials from the runtime environment. A successful health response does not replace authenticated session, stream, and recovery checks.