Operate and recover Chronicle observers

Operations

Operate a running Chronicle

Use Cratis Chronicle’s observer state, failed-partition records and jobs to find why processing stopped, choose a recovery and check the resulting output. Start here when a read model goes stale, a reactor stops doing its work or an observer alert fires.

The operator journey

An observer processes events for a projection, reducer or reactor. A failed partition holds back part of that work; it does not necessarily stop the whole observer. Diagnose the affected scope before choosing how much history to process again.

Monitor the health endpoint and configure alerts on observer failures so you can investigate before a user reports stale output.
Follow Diagnostics to find the observer and partition in the correct event store and namespace. Read the exception and failing event sequence number, then check whether a recovery job is already running.
Use Recovering from failures to choose between preserving the existing state with a retry and rebuilding it with a replay. Clearing quarantine can start processing again, so fix the cause and account for repeated side effects before releasing it.
Follow Recover a failed partition to carry out the recovery through the Cratis CLI runbook and verify both processing progress and application output.

Chronicle automatically retries failed partitions with backoff, waiting longer between attempts. Intervene when the retry budget is exhausted or when the cause needs a code, data or dependency fix. A successful recovery request is only the start: the job and resulting output still need checking.