Observability — execution tracing

Every handler run is recorded as a tree of steps with real request/response bodies, and is visible in the admin at /executions. This page is the contract for keeping it that way.

The one rule

Everything you await must either be an auto-instrumented client or sit inside step.run.

That is the whole rule. pnpm check:instrumentation enforces it in CI.

Why the rule exists

Cloudflare shows that a request happened. It does not show that the CRM lookup returned zero contacts, that the order payload was missing total, or which of five steps was the slow one. The execution tree does — but only for work that announces itself.

Untraced awaits are invisible. A run made of five untraced calls looks identical to a run that did nothing, and you find out during an incident.

What is already covered

createHandlerContext wraps these clients with instrument(), so every method call becomes a step automatically, with its arguments and return value:

Client	Example step
db	db.insert, db.query
cache	cache.get, cache.set
config	config.get
creds	creds.get

// Already a step named `db.insert` — no wrapping needed for visibility.
const order = await db.insert("orders", payload);

Wrap one of these only when a business name says more than the client call does:

// Worth it: "db.findActiveCall" reads better than "db.query" in an incident.
const call = await step.run("db.findActiveCall", () => db.query(sql, [phone]), { input: { phone } });

Wrapping db.insert in a step also called db.insert is pure noise — don't.

What you must wrap

Anything else that leaves the Worker or takes real time: HTTP calls, CRM clients, LLM calls, the call queue, vector search.

// Wrong — invisible in the execution tree
const contact = await crm.search(phone);

// Right
const contact = await step.run("crm.findContact", () => crm.search(phone), {
	input: { phone },
});

Naming steps

<area>.<action>, lowercase, dot-separated: crm.findContact, llm.classifyIntent, horoshop.fetchCatalog. The name is what someone reads at 3am — make it say what it did, not which function it called.

input is not optional in practice

Without input, the step shows a name and a duration. With it, you can see what was sent — which is usually the whole reason you opened the run. Pass it.

Optional steps

const enriched = await step.run("crm.enrich", () => crm.enrich(id), {
	optional: true,
	default: null,
});

Marked skipped in the tree instead of failing the run.

Escape hatch

If a step genuinely adds nothing, say so on the line:

const now = await getServerTime(); // no-step: clock read, no external effect

Use it sparingly. It is a claim that the call cannot fail in a way anyone would investigate.

Enforcement

Layer	What it catches
pnpm check:instrumentation (CI)	A new await that would not appear in the tree
scripts/instrumentation-baseline.json	Existing debt — the list may shrink, never grow
Runtime warning handler_not_instrumented	A run that produced zero steps
Admin /executions	A run with no steps renders as such

The baseline

141 pre-existing uninstrumented awaits are recorded in the baseline and accepted as debt. They are mostly call-queue, raw d1 and bare fetch. CI does not fail on them; it fails on anything new.

When you touch a handler that has entries in the baseline, wrap them and regenerate:

pnpm check:instrumentation --update-baseline

A PR that grows the baseline should be rejected in review.

Retention and personal data

Executions store real request and response bodies, which for most integrations means personal data from a CRM. They are deleted 14 days after the run by an hourly sweep in integ-api. Bodies over 256 KB are replaced by a marker. Keys matching password, token, secret are redacted before storage.

Do not extend retention without a reason that survives a privacy question.

Turning it off

EXECUTION_RECORDING=off in the integration's Doppler config stops the shipping without a redeploy. Logs and console output are unaffected.

How it works

1. createExecutionRecorder() per request collects step events.
2. The recorder is fed before the log-level filter — LOG_LEVEL controls what is printed, never what is recorded.
3. step.run and instrument() send the raw value to the recorder and a summarized copy to the console.
4. After the response is sent, shipExecution() POSTs the run once from ctx.waitUntil() to integ-api. One subrequest, off the hot path, never fails the handler.

See packages/trace/src/execution-recorder.ts and packages/core/src/execution-sink.ts.