Jobs, retry, cancellation and recovery

Horologium runs collection and reconciliation as durable, PostgreSQL-backed jobs (ADR-0017, HORO-1395) — not Kafka, not Temporal, no new infrastructure. This page documents the operator-visible contract: what a job's states mean, how retry/cancel/crash-recovery behave, and the default bounded resource use horologium-runner ships with.

Starting a job

curl -s -X POST http://127.0.0.1:8080/v1/jobs \
  -H "Authorization: Bearer $TOKEN" -H "Content-Type: application/json" \
  -d '{"job_kind": "collect_and_reconcile"}'

job_kind is one of collect, reconcile, or collect_and_reconcile (candidate_analysis is refused — it needs the candidate namespace, not yet built). The response is a durable run identity (job_id) you can poll or cancel:

curl -s http://127.0.0.1:8080/v1/jobs/$JOB_ID -H "Authorization: Bearer $TOKEN"
curl -s -X POST http://127.0.0.1:8080/v1/jobs/$JOB_ID/cancel -H "Authorization: Bearer $TOKEN"

Creating a job requires the capability its job_kind needs (run:collect, run:reconcile, or both for collect_and_reconcile); cancelling requires any run:* capability. See crates/horologium-api/README.md's "Enforcement" section for the exact rule.

Nothing runs a job by itself — POST /v1/jobs only enqueues it. A separate horologium-runner process (below) claims and executes queued jobs on a schedule. If no runner process is running, a job simply stays queued forever — this is expected, not a hang.

Job states

State	Meaning
queued	Enqueued, not yet claimed by a runner.
running	Claimed by a runner, actively collecting/reconciling/publishing.
partial	Published successfully, but at least one in-scope source failed to collect this run — see Partial source failure below.
succeeded	Published successfully, every in-scope source collected cleanly.
failed	Did not publish. Either every in-scope source failed to collect (ADR-0017 J6 — an empty-coverage snapshot would falsely assert "nothing exists"), or a non-retryable error occurred.
cancelled	Cancelled by an operator before it reached a terminal state.

partial, succeeded, failed, and cancelled are terminal — a job never leaves one of these states. Retrying means submitting a new job, not resuming the old one (see Retry below).

Partial source failure

A source that fails to collect on a given run (unreachable, unauthorized, misconfigured) never has its prior evidence or claims silently deleted, and never causes an existing finding to auto-resolve. The job publishes partial, with coverage_complete: false on its snapshot, and the failing source's access_state (GET /v1/sources/{id}) reflects why. Only claims from a source that did collect successfully this run are ever superseded.

Retry

POST /v1/jobs takes an idempotency_key. Submitting the same key twice returns the same job_id — it does not enqueue a second run. Retrying a job that failed or was cancelled means submitting a new request (a new idempotency key, or no key), which the runner treats identically to any other queued job. Because every write on the way to a published snapshot is idempotent (evidence is content-addressed, claims/entities/relationships dedupe by fingerprint/natural key), retrying against unchanged inputs produces zero duplicate evidence, claims, or findings — it is always safe to retry.

Crash recovery

Every job is claimed under a time-bounded lease. If the runner process holding a job's lease dies — killed, crashed, host rebooted — the lease simply expires; a separate runner process (or the same one, restarted) reaps it and completes the job through a fresh attempt. This is real, process-level tested behavior, not just a database-row simulation:

• A runner killed with SIGKILL mid-reconcile recovers cleanly, with exactly one attempt marked lease_expired and the reclaiming attempt publishing exactly one snapshot (crates/horologium-jobs/tests/crash_recovery.rs::a_hard_killed_runner_process_does_not_lose_the_job).
• The same guarantee holds specifically during collection — killed while job.phase = 'collecting' — with no duplicate evidence from the source the dead attempt had already (partially) collected (...::a_hard_killed_runner_is_recovered_mid_collection).

A concurrent Web/MCP reader never sees a torn or mixed snapshot while a new run is staged — publication is one atomic transaction; a reader sees either the previous consistent snapshot or the new one, never a partial mix (crates/horologium-jobs/tests/publish_snapshot.rs).

Graceful shutdown (SIGTERM/SIGINT) stops the runner's poll/reap loop cleanly. horologium-runner never spawns or owns a subprocess in production — nothing can be orphaned by either kind of shutdown (crates/horologium-jobs/tests/no_orphaned_subprocess.rs).

Default bounded resource use

horologium-runner is one process with a background poll/reap loop — no cluster, no autoscaling, no indefinite polling required. Defaults, all operator-overridable via environment variable, none of them correctness inputs (a malformed value falls back to the default rather than failing startup):

Env var	Default	Meaning
HOROLOGIUM_RUNNER_LEASE_SECS	30	How long a claimed job's lease is held before it's eligible for reclaim if the holder goes silent.
HOROLOGIUM_RUNNER_POLL_INTERVAL_SECS	5	How often the runner checks for a new queued job.
HOROLOGIUM_RUNNER_REAP_INTERVAL_SECS	60	How often the runner checks for jobs whose lease has expired.
HOROLOGIUM_RUNNER_REAP_BACKOFF_SECS	30	Delay before retrying a reap sweep that found nothing.
HOROLOGIUM_RUNNER_ID	generated	Distinguishes this runner instance in job.lease_owner; set explicitly if you run more than one.

HOROLOGIUM_DATABASE_URL is required and has no default — the runner exits loudly on startup rather than silently doing nothing against a missing database.

Two further bounds are not environment-configurable — they're fixed in the schema/code, not tuning knobs:

Bound	Value	Where
Max retry attempts per job	3	job.max_attempts default
Per-job deadline	1 hour	job.deadline_at, set at enqueue time

Queue depth itself is intentionally unbounded (any number of distinct jobs may sit queued) — the bound that matters operationally is on concurrency, attempts and deadline, all four of which are listed above.

A caveat worth stating plainly: the PostgreSQL row locks this queue uses to prevent overlapping runs are a queue-consistency mechanism, not a Product Truth query — SELECT ... FOR UPDATE SKIP LOCKED tells you nothing about the shape or health of your product. Use GET /v1/jobs/{id} for job status and GET /v1/entities, GET /v1/findings, etc. for Product Truth itself; never infer one from the other.