Instance Lifecycle

Provision an instance, get past the first-run auth chicken-and-egg, run a backup discipline that actually saves you, watch the right health probes, and tear down cleanly. The end-to-end recipe for self-hosted operators.

Audience

This page is for the operator standing up an spl serve instance for real work — laptop, server, container, or a Syncropic-hosted instance. It covers the lifecycle from spl init to teardown without re-deriving the architecture from first principles.

If something is broken right now, jump to Recovery on the runbook page. The lifecycle below assumes you're starting clean.

How the engine actually runs

The instance implements four cooperating loops over an immutable record log. Every operational decision below — when to back up, how to read a probe, what to drain before stopping — makes more sense once the loops are visible.

The four engine loops. State is computed from records; none of the loops mutate anything outside the record log.

The internal layers follow the same structure. Nothing above depends on anything below it; the engine is portable along any of the dashed boundaries (Records / Algebra and Trust are portable to the browser).

spl instance internal layering. Strict layering, no upward dependencies.

1. Provisioning

Self-hosted (laptop, server, VPS)

Install the binary, then run the setup wizard:

curl -sSf https://get.syncropic.com/spl | sh
spl init

spl init creates ~/.syncro/, generates an instance identity (or surfaces the existing one), and prints the next-step commands. It does not start the instance.

Start the instance as a foreground process the first time so you can see what it's doing:

spl serve --foreground

You should see a startup banner naming the bind address, the SQLite path, and the backup destination. Ctrl-C to stop. Once you're satisfied, run it as an instance:

spl serve
spl status

For an instance that survives reboots, see Keeping Your Instance Running — systemd user units on Linux, launchd plists on macOS, and Windows Service wrappers are all documented there.

Hosted

Hosted instances are provisioned for you: sign in at syncropel.com and the instance comes up at <label>.syncropel.app. You connect with the code shown at sign-up, and every recipe on this page that uses the CLI works against it from there.

2. First-run bootstrap

A fresh database has zero service accounts. The auth middleware rejects every unauthenticated request, including the request you'd use to mint your first SA. Two paths through the chicken-and-egg:

Path A — `--insecure-localhost` (laptop dev)

Restart the instance with --insecure-localhost. It binds 127.0.0.1 only and disables auth. Mint an SA via the bootstrap endpoint:

spl stop
spl serve --foreground --insecure-localhost &

curl -fsS -X POST http://127.0.0.1:9100/v1/bootstrap/service-account \
  -H 'content-type: application/json' \
  -d '{"sa_id":"sa-admin","scopes":["admin"]}'
# → 201 + bearer token; copy the token (it is shown once).

spl token save "$BEARER"
spl stop
spl serve

This is the path documented in Authentication & Service Accounts and is fine for a single-user laptop. Avoid it on multi-tenant hosts: any local user can curl 127.0.0.1.

Path B — hosted (no shell access)

Sign in at syncropel.com and provision. The instance is bootstrapped for you and you receive a working https://<label>.syncropel.app URL; connect with the code shown at sign-up.

3. Auth posture for production

Once bootstrapped, treat auth.required = true as load-bearing. It is the default and every shipping route enforces it (including the federation surface). Specifics live in Authentication & Service Accounts; the operational rules are:

One service account per tool/agent/integration. Closed scope per SA — records:read,records:write for an emitter, admin only when minting tokens or setting policy.
Tokens carry a snapshot of the SA's scopes at issuance time. Live-edit the SA's scopes for new tokens; revoke + re-mint the existing token only when scopes shrink.
Save the token once via spl token save <bearer>; the CLI auto-injects it for every subsequent invocation.
Permission CEL is enforced fail-closed. Before turning on a permission rule, run spl config permissions-enable — its preflight refuses unless allow-rules cover record_write, thread_read, and config_read for the instance's own actor. Skipping the preflight is how operators lock themselves out.

If the instance is ever exposed beyond loopback, stand up a CORS allowlist before the first cross-origin browser hit:

spl config auth-set-cors-origins https://syncropel.com https://app.example.com

4. Backup discipline

CRITICAL — read this section twice. Syncropel's startup backup is a safety net, not a backup system. It will not save you if you don't supplement it with off-host copies. The recovery drill that ships with the instance source tree was written specifically because operators conflated the two.

What the instance does for you

On every startup, spl serve snapshots ~/.syncro/hub.db to ~/.local/share/syncropel/backups/<instance-key>/hub.db.bak. The destination lives outside ~/.syncro/, so a rm -rf ~/.syncro/ does not nuke the backup.

The <instance-key> is normally instance-<did-tail> — the 24-hex tail of the instance DID. To find your directory:

curl -s localhost:9100/health | jq -r .instance_did
ls ~/.local/share/syncropel/backups/instance-<tail>/

The daily backup job writes timestamped hub.db.<YYYYMMDD-HHMMSS> copies into the same per-instance directory.

What the instance does NOT do

The startup backup is destructive on every restart. If hub.db is empty, corrupt, or wrong when the instance starts, that empty/corrupt/wrong file overwrites the backup. By the time you notice, the good copy is gone.

What you should do instead

Schedule an off-host snapshot of the rolling backup. Daily is enough for most workloads:

DEST=$HOME/backups/syncropel
mkdir -p "$DEST"
cp ~/.local/share/syncropel/backups/*/hub.db.bak \
   "$DEST/hub.db.$(date +%Y%m%d-%H%M%S).bak"
find "$DEST" -name 'hub.db.*.bak' -mtime +14 -delete

For containerised deployments, mount a host directory into the backup path so the rolling backup survives the container's ephemeral filesystem.

Manual snapshot (before risky operations)

Right before an upgrade, a permission rule rollout, or a destructive migration:

cp ~/.syncro/hub.db ~/.local/share/syncropel/backups/hub.db.before-$(date +%Y%m%d-%H%M%S)

The instance does not need to be stopped — SQLite's WAL makes this safe — but it does need to not be in the middle of a heavy write burst. Watch spl status first.

Restore

spl stop
cp ~/.local/share/syncropel/backups/<instance-key>/hub.db.bak ~/.syncro/hub.db
spl serve

On startup, trust scores and engine config rebuild from KNOW/DO and LEARN records respectively. Task content files and alias mappings live in ~/.syncro-data/ and are unaffected by hub.db operations — they survive on their own.

WSL2 / UNC paths. If ~/.syncro/ resolves to a UNC path (\\wsl.localhost\...) when accessed from Windows tooling, SQLite's file locks behave erratically. Keep hub.db on a native Linux filesystem (the WSL home directory itself) and copy backups out via cp rather than letting Windows Explorer touch them.

5. Health probes

Three probe surfaces, three intended consumers.

Path	Returns	Use for
`GET /health`	`200 ok` if process is alive and bound	Load-balancer liveness / platform health checks
`GET /v1/engine/health`	JSON with reconcile counters, queue depth, decision pending count	Operator readiness, alerting
`GET /v1/engine/health` (with `details=true`)	Full per-loop breakdown including expression-cache stats	Capacity planning, debugging

Liveness:

curl -fsS http://localhost:9100/health
# ok

Readiness (sample):

curl -fsS http://localhost:9100/v1/engine/health | jq
# {
#   "ingested_total": 18432,
#   "reconciled_total": 18391,
#   "reconcile_queue_depth": 2,
#   "decisions_pending": 0,
#   "intelligence_enabled": true,
#   "uptime_secs": 86400
# }

reconcile_queue_depth should be near zero in steady state. A growing queue with stable ingest is the signal that an adapter has stalled; check /v1/engine/health?details=true for which thread is backed up, then spl debug replay <thread> to walk the records.

CEL hot-path observability:

curl -fsS http://localhost:9100/v1/engine/expression_cache/stats | jq
# Healthy: hit rate > 99%, avg compile < 100μs, size << capacity (1024).

A compile-error spike in this endpoint usually means a bad CEL config record landed; check spl config list-rules and the most recent LEARN on th_engine_config.

6. Teardown

Tearing down for real (decommissioning the host, moving to another machine, retiring an instance):

Drain

Refuse new dispatches but let in-flight ones finish:

spl drain start
spl drain status   # waits until in-flight = 0

Stop the instance gracefully

spl stop

This sends SIGTERM. The instance flushes the SQLite WAL, closes the socket, and removes its PID file. If --stop reports "not running" but the port is still bound, see the orphan-recovery section on the runbook.

Reclaim space (optional)

The record store reclaims freed pages lazily, so the store file does not shrink on its own after an upgrade migration or a large deletion. With the instance stopped, compact the store file with your database tooling to reclaim it. Budget roughly twice the file size in free disk and expect an exclusive lock for the duration (minutes on multi-gigabyte ledgers). Do not compact a running instance.

Delete

Only after the archive is verified somewhere durable:

rm -rf ~/.syncro ~/.syncro-data ~/.local/share/syncropel

If the instance had a federation pair, run spl federation revoke <pair-id> against the peer first — see Federation Pairing for the full pair-revocation procedure.

What's next

Day-2 ops, log fields, recovery drills: Operator Runbook.
First-run auth in detail: Authentication & Service Accounts.
Connection problems from a browser: Troubleshooting Connection.
Cross-instance sync: Federation Pairing.