Dispatch observability

How Syncropel turns every dispatched task into a queryable record stream — what records get emitted, how to read them, and what the completion codepath and failure reason enums mean.

The philosophy — records are spans

Every dispatched task in Syncropel — every spl task dispatch, every fanout subtask, every agent invocation that spawns a subprocess — emits a stream of observation records as it runs. When the task finishes (or doesn't), you don't need to read logs to know what happened. You query records.

This follows the same "state = fold(records)" design as the rest of the protocol. Logs are a side-effect-ful stream that can be dropped, corrupted, or silently buffered. Records are content-addressed, immutable, and fold deterministically into a reconstructable state. If the dispatch completed at all, the records are there — even if the subprocess crashed, the log process died, or the log file rotated out from under you.

This page documents what gets emitted, how to query it, and what the enum values mean.

The four record kinds

Every CLI-adapter dispatch emits up to four observation record kinds on a private sub-thread so tool calls don't pollute the parent task thread. Only the final completion summary flows back to the parent.

`syncropel.dispatch.stream_event.v1`

Emitted once per Assistant event the subprocess writes. Carries delta + running totals for input/output tokens.

{
  "kind": "syncropel.dispatch.stream_event.v1",
  "topic": "stream_event_seen",
  "event_type": "Assistant",
  "tokens_in_delta": 420,
  "tokens_out_delta": 117,
  "tokens_in_accum": 8452,
  "tokens_out_accum": 2203,
  "cost_usd_delta": null,
  "turn_id": "t_01"
}

Why it exists: lets you reconstruct cost and token progress even when the subprocess dies before the terminal Result event. Before this record, a subprocess that crashed at 80% through a long task showed cost_usd: 0.0 — as if nothing had happened. With it, you can sum *_accum on the latest record and know what actually got paid for.

`syncropel.dispatch.session.v1`

Emitted once, when the subprocess's first system init event arrives. Captures the agent CLI's internal session_id plus cwd/model.

{
  "kind": "syncropel.dispatch.session.v1",
  "topic": "agent_session_captured",
  "session_id": "a3f29c4e-0b1d-4f8a-...",
  "project_cwd": "/home/you/repos/project",
  "project_slug": "home-you-repos-project",
  "model": "<model-id>"
}

Why it exists: bridges Syncropel records to the agent CLI's own per-project state on disk. When an agent-side bug or unusual tool-use pattern needs diagnosis, you need the raw agent-side log. See Correlating records with agent CLI session state for the full workflow.

`syncropel.dispatch.subprocess_exit.v1`

Emitted always after child.wait(). Records the objective facts that previously only existed as log lines.

{
  "kind": "syncropel.dispatch.subprocess_exit.v1",
  "topic": "subprocess_exited",
  "pid": 284123,
  "exit_code": 0,
  "signal": null,
  "duration_ms": 12534,
  "last_stream_event_type": "Result",
  "last_stream_event_clock": 47
}

Why it exists: the subprocess may close its stdout, emit a final Result event, and exit with code 0 — or it may die mid-stream with signal 9 from OOM. Either way, child.wait() returned something, and this record captures it verbatim. Absence of this record means the dispatch task itself crashed before child.wait() returned (engine-level bug — file a ticket).

`syncropel.dispatch.complete.v1`

Emitted on every dispatch conclusion — success or any failure path. Carries the completion metadata that makes the full dispatch diagnosable.

{
  "kind": "syncropel.dispatch.complete.v1",
  "topic": "dispatch_complete",
  "fulfills": "trigger_id_of_the_INTEND",
  "success": true,
  "cost_usd": 0.0423,
  "cost_source": "result_event",
  "completion_codepath": "result_event",
  "failure_reason": null,
  "latency_ms": 12534,
  "tokens_in": 8452,
  "tokens_out": 2203,
  "domain": "code"
}

The completion_codepath, failure_reason, and cost_source fields answer three questions that previously required grepping logs:

completion_codepath — which of the four exit paths fired:

Value	When
`result_event`	Subprocess emitted a terminal `Result` stream event. Normal completion.
`line_timeout`	`per_line_timeout` fired — the subprocess went silent longer than the configured interval.
`budget_deadline`	Absolute wall-clock `budget_deadline` reached. The subprocess was killed.
`stream_eof_fallback`	stdout closed without a `result` event — the child exited or stdout broke. This is the "silent loss" codepath.

failure_reason — the human-queryable reason the dispatch failed (None on success):

Value	Meaning
`result_missing`	stdout closed before a `Result` event arrived.
`subprocess_exited_nonzero`	Subprocess exited non-zero after closing stdout.
`line_timeout`	Per-line timeout fired.
`budget_exceeded`	Absolute wall-clock deadline reached.
`read_error`	stdout read error mid-dispatch.
`result_reported_error`	The `Result` event carried `is_error: true` — the CLI itself reported failure.

cost_source — how we got the cost figure:

Value	Meaning
`result_event`	Figure came from the terminal `Result` stream event. Most accurate.
`accumulated_from_events`	Figure was summed from per-event usage deltas. Used when Result was never emitted.
`unknown`	No figure available — stream ended before any cost signal (rare; indicates very early failure).

Reading this with `spl task diagnose`

spl task diagnose <alias> reads these records and produces a human-readable summary with recovery guidance.

spl task diagnose TASK-0042

Task: TASK-0042 — your task title here
Sub-thread: th_8c51e9c8...

Dispatch pipeline
  ├─ INTEND on parent thread      clock 24
  ├─ sub-thread created            th_8c51e9c8...
  ├─ subprocess spawned            pid 284123 at 18:02:11
  ├─ session captured              session_id=a3f29c4e...
  │                                project: home-you-projects-myproject
  ├─ 17 stream events              last at 18:15:44 (type=Assistant)
  ├─ subprocess exited             code=0 signal=None dur=821s
  └─ completion                    codepath=result_event
                                    failure=None
                                    cost=$0.0423 (from result_event)
                                    tokens=8452/2203
                                    latency=821.3s

Recovery: no action needed — dispatch completed normally.

Or when a dispatch went sideways:

Task: TASK-0042 — your task title here
Sub-thread: th_8c51e9c8...

Dispatch pipeline
  ├─ INTEND on parent thread      clock 24
  ├─ sub-thread created            th_8c51e9c8...
  ├─ subprocess spawned            pid 284123 at 18:02:11
  ├─ session captured              session_id=a3f29c4e...
  ├─ 127 stream events              last at 18:19:46 (type=Assistant)
  ├─ subprocess exited             code=None signal=9 dur=1119s
  └─ completion                    codepath=stream_eof_fallback
                                    failure=subprocess_exited_nonzero
                                    cost=$0.1847 (from accumulated_from_events)
                                    tokens=34127/8821
                                    latency=1119.2s

Recovery: OOM-killed (signal=9). Worktree preserved at
.../myproject-TASK-0042/. To resume with prior commits replayed
into the agent briefing:

    spl task dispatch --resume TASK-0042

The key win: cost=$0.1847 with a definitive failure reason. Work did happen, we know what it cost, and we know why it failed — all surfaced by spl task diagnose without touching a log file.

Querying the raw log lines with `spl logs --trace-id`

Records give you the shape of what happened. Logs give you the line-by-line narrative. The two are linked by the sub-thread id that wraps every dispatch as a tracing span.

spl logs --trace-id th_8c51e9c8...

This reads both the JSON-structured daemon log (~/.syncro/logs/spl.log) and the rotated siblings (spl.log.1, .2), plus the legacy foreground ~/.syncro/serve.log if present, filters to lines whose span context includes sub_thread=th_8c51e9c8..., merges them chronologically, and prints them pretty-formatted.

Other selectors:

spl logs --dispatch-id dispatch_th_abc_12345     # tracking_id (PID-scoped)
spl logs --sub-thread th_8c51e9c8...              # explicit form
spl logs --turn-id t_01                           # single turn
spl logs --trace-id th_abc --since 1h             # rolling window
spl logs --trace-id th_abc --level warn           # level floor
spl logs --trace-id th_abc --sources tracing     # only JSON log
spl logs --trace-id th_abc -o json                # raw JSONL pass-through

Example output:

2026-04-23T18:02:11.000Z [tracing] INFO  syncropel_engine::dispatch::cli_adapter cli_adapter: dispatched with private sub-thread binary=claude pid=284123 sub_thread=th_8c51e9c8...
2026-04-23T18:15:44.218Z [tracing] WARN  syncropel_engine::dispatch::stream_monitor cli_adapter: stdout read error error=early eof sub_thread=th_8c51e9c8...
2026-04-23T18:15:44.901Z [tracing] INFO  syncropel_engine::dispatch::stream_monitor cli_adapter: dispatch complete latency_ms=821301 cost_usd=0.0423 codepath=stream_eof_fallback failure_reason=subprocess_exited_nonzero sub_thread=th_8c51e9c8...

At least one filter is required — unfiltered tailing is spl serve --logs. The log query deliberately refuses empty filters so you don't drown in every line of the daemon's output.

Log retention

Daemon logs rotate at 10 MB per file with three generations (spl.log, spl.log.1, spl.log.2). Rotated siblings and the legacy foreground serve.log are time-pruned at daemon startup with a 7-day default. Configure via:

The SPL_LOG_RETENTION_DAYS env var on the daemon process — runtime override.
A syncropel.config.log_retention.v1 LEARN record on th_engine_config (format below) — persisted via the record log; the daemon surfaces its days value as SPL_LOG_RETENTION_DAYS on subsequent restarts.

{
  "kind": "syncropel.config.log_retention.v1",
  "topic": "log_retention",
  "days": 14
}

Set days: 0 to disable retention (files will only be pruned by size rotation).

What to do when a dispatch goes wrong

spl task diagnose <alias> — human-readable summary. 80% of incidents are fully explained by the output.
spl logs --trace-id <sub_thread> — pull the narrative lines for the dispatch.
spl task dispatch --resume <alias> — re-run on the preserved worktree, with prior commits surfaced in the agent briefing.
Recovery cookbook — the full salvage workflow for a partially-completed task.
Claude session correlation — when the Syncropel-side records aren't enough and you need the raw Claude tool-use history.

On this page