Get alerts for Chronicle observer failures
Use Chronicle’s observer metrics or a scheduled CLI check to get notified when event processing fails. A reactor can stop processing one event source while other partitions continue, so an application health check alone can miss the failure.
Chronicle records incidents and exports metrics; your monitoring backend or scheduler delivers the notification. Set up an alert rule with OpenTelemetry, a scheduled check with the Cratis CLI, or both.
What to watch
Section titled “What to watch”An observer is a reactor, reducer or projection that processes events. It tracks each event source in a separate partition, so a handler failure initially stops only that partition.
Chronicle retries the failed partition with exponential backoff: waits of 2 seconds, doubling each time, capped at 10 minutes. Processing can recover on its own if, for example, an unavailable dependency comes back. After MaxRetryAttempts (10 by default), Chronicle quarantines the partition and stops retrying it. With the default settings, that happens roughly half an hour after the first failure. An operator must then resume it; see My reactor throws and the stream seems stuck.
Failures can also stop the whole observer. If you configure QuarantineOnFailedPartitionCount or QuarantineOnFailedPartitionPercentage and cross that threshold, Chronicle quarantines the observer.
A quarantined partition is the signal that needs a person. A partition that keeps failing is an early warning. The retry and quarantine settings are described in observer configuration.
Understand recorded incidents
Section titled “Understand recorded incidents”Chronicle records observer incidents in its system event sequence. These incidents describe current operational state, not a complete audit of every failure episode. Open incidents converge to the observer’s committed state; an episode that starts and ends before reconciliation may leave no incident history. Distinct quarantine episodes have separate identities, even if reconciliation misses the revival between them.
After a restart, allow approximately one reminder period (one minute), plus reconciliation time, once the cluster, storage, and reminder service are functioning. This is not a one-minute guarantee during an outage. Healthy observers reconcile once after activation, then stop calling the incident tracker until their state changes.
Ending reasons are best-effort across crashes. If an episode’s ending reason was not recorded before activation memory was lost, a partition incident clears as Recovered and an observer quarantine clears as Cleared. The original operator-clear or revival provenance cannot be reconstructed. Retirement and removal commit a durable Retired level and clear incidents as Removed. This level means the observer no longer desires processing or incidents, not that cleanup has finished.
Removal and retirement can fail when incident reconciliation or resource cleanup fails. Retry the management operation to continue forward: committed retirement is not rolled back. Removal clears incidents before deleting resources in each namespace, but processes namespaces sequentially without a store-wide transaction. Earlier namespaces may already be removed when a later one fails. Avoid concurrent explicit registration while removing an observer; shared-definition cleanup is not coordinated with it.
Automatic recovery, including projection-manager startup, leaves committed retirement intact. Explicit registration can reactivate the observer with a fresh lifecycle after discarding retained failed partitions and retry reminders. It does not reopen ended incident IDs. Probing or retiring an absent observer creates no observer record or alert reminder.
Current recorded incidents are available through scoped incident queries, with bounded pages, lookups, counts and sampled materialization health. Check that health before treating an empty result as no incidents. The query surface does not deliver email or chat notifications; use the monitoring options below for delivery. Open incidents are also published as the chronicle_alerts_open_incidents gauge.
Alert from metrics
Section titled “Alert from metrics”The examples below use the default Legacy series. To move to convention names, set Telemetry:MetricNames to Convention (or Both while migrating) and migrate names and dimensions together using the telemetry name reference. Convention series use keys such as cratis.event_store.name and cratis.observer.id; with Both, do not add the values of the two sets together, because each measurement is recorded twice. The default is planned to switch in the next major version.
Chronicle exports its metrics over OpenTelemetry from the Cratis.Chronicle meter. Three counters describe failing observers. Each has one series per observer, tagged EventStore, Namespace, ObserverId and EventSequenceId. An observer has a series on them from its first failure, not before:
| Instrument | Counts |
|---|---|
chronicle-observer-partitions-failed |
Every failed attempt, including each retry. It is not a count of partitions. |
chronicle-observer-partitions-quarantined |
Partitions that ran out of retries |
chronicle-observer-quarantined |
Times the whole observer was quarantined |
With the default translation of the OpenTelemetry Collector’s Prometheus exporter, or Prometheus’s own OTLP receiver, the counters appear as chronicle_observer_partitions_failed_total, chronicle_observer_partitions_quarantined_total and chronicle_observer_quarantined_total, and the tags keep their names. Look the series up in your backend before relying on the names below.
These failure counters are hand-written rather than generated by Cratis.Metrics.Roslyn, to keep their initialization and tagging unchanged. Use the names exported by your deployed kernel when configuring these rules. Convention names are opt-in through Telemetry:MetricNames; if you switch, update dashboards and alert rules together so a naming change does not leave failures unmonitored.
A series is created by the first failure and starts at 1; Chronicle does not record a 0 beforehand, so healthy observers cost no series. The first value your backend receives for an observer is therefore already 1. A rule built only on increase() needs two samples in its window and reports nothing for that series. Each rule below therefore has two halves, joined with or: the first catches series that went up inside the window, the second catches series that did not exist at the start of the window and are already above the threshold. Both aggregate to the same labels, so an observer produces one series, and so one alert, whichever half caught it.
The rules select each counter by its name prefix, for example {__name__=~"chronicle_observer_partitions_quarantined.*"}, so the same rules work for the long names that earlier Kernels export. These Prometheus alerting rules report, per observer, partitions that ran out of retries and observer quarantines in the last 30 minutes, and repeated failed attempts in the last 15 minutes:
groups: - name: chronicle-observers rules: - alert: ChroniclePartitionsQuarantined expr: | sum by (EventStore, Namespace, ObserverId) ( increase({__name__=~"chronicle_observer_partitions_quarantined.*"}[30m]) ) > 0 or count by (EventStore, Namespace, ObserverId) ( {__name__=~"chronicle_observer_partitions_quarantined.*"} > 0 unless {__name__=~"chronicle_observer_partitions_quarantined.*"} offset 30m ) labels: severity: critical annotations: summary: "Partitions of {{ $labels.ObserverId }} in {{ $labels.EventStore }}/{{ $labels.Namespace }} ran out of retries" description: "Chronicle will not retry them again. Inspect with: cratis chronicle failed-partitions list -e {{ $labels.EventStore }} -n {{ $labels.Namespace }}"
- alert: ChronicleObserverQuarantined expr: | sum by (EventStore, Namespace, ObserverId) ( increase({__name__=~"chronicle_observer_quarantined.*"}[30m]) ) > 0 or count by (EventStore, Namespace, ObserverId) ( {__name__=~"chronicle_observer_quarantined.*"} > 0 unless {__name__=~"chronicle_observer_quarantined.*"} offset 30m ) labels: severity: critical annotations: summary: "{{ $labels.ObserverId }} in {{ $labels.EventStore }}/{{ $labels.Namespace }} was quarantined and processes no events"
- alert: ChroniclePartitionsFailing expr: | sum by (EventStore, Namespace, ObserverId) ( increase({__name__=~"chronicle_observer_partitions_failed.*"}[15m]) ) >= 3 or count by (EventStore, Namespace, ObserverId) ( {__name__=~"chronicle_observer_partitions_failed.*"} >= 3 unless {__name__=~"chronicle_observer_partitions_failed.*"} offset 15m ) labels: severity: warning annotations: summary: "{{ $labels.ObserverId }} in {{ $labels.EventStore }}/{{ $labels.Namespace }} has failed repeatedly in the last 15 minutes"Route the alerts with Alertmanager or Grafana to wherever your team works — email, Slack, Microsoft Teams, PagerDuty. Other backends that receive OTLP, such as Azure Monitor, Datadog or Grafana Cloud, can express the same conditions in their own alert languages. The two halves of each rule are what to carry over: a series that appears already above zero is as much a failure as one that climbs.
Know the limits of counters before you depend on them:
- They say “recently”, not “still”. The quarantine alerts fire when something is quarantined, then resolve after 30 minutes even if it is still stuck. The failing alert cannot tell a partition that is still failing from one that recovered after its latest failure. Keep the scheduled check below for the current state.
- They reset when the Kernel restarts.
increase()handles resets, and a partition that fails again after a restart is counted again. - A gap in your own data raises false alerts. The second half of each rule fires for every series that is above the threshold now and did not exist 30 minutes ago (15 for the failing alert). After Prometheus or the collector restarts, a scrape outage, or a series dropped by retention, every observer that has ever failed qualifies, and the alerts resolve once the window has passed. A
for:clause does not help, because the condition stays true for the whole window. Inhibit these alerts in Alertmanager while your monitoring itself is recovering, for example with an alert ontime() - process_start_time_seconds < 1800for the Prometheus job.
Alert on open incidents
Section titled “Alert on open incidents”The counters above describe failures as they happen. The chronicle_alerts_open_incidents gauge describes what is open now, per observer, condition and severity, and drops back to 0 once an incident clears. Its Severity tag (Warning or Critical) is the incident’s severity and is not the severity label of a Prometheus rule. Aggregate with max, never sum, because more than one instance can briefly report the same series. See open alert incident metrics for the sampling and failover behavior.
groups: - name: chronicle-incidents rules: - alert: ChronicleCriticalIncidentOpen expr: | max by (EventStore, Namespace, ObserverId, EventSequenceId, Condition) ( chronicle_alerts_open_incidents{Severity="Critical"} ) > 0 for: 2m labels: severity: critical annotations: summary: "Critical incident open for {{ $labels.ObserverId }} in {{ $labels.EventStore }}/{{ $labels.Namespace }}" - alert: ChronicleIncidentGaugeUnavailable expr: | max(chronicle_alerts_open_incidents_available) == 0 or absent(chronicle_alerts_open_incidents_available) for: 5m labels: severity: warning annotations: summary: "Open incident metrics are unavailable; an empty gauge does not mean no incidents"The second rule fires when the owner’s snapshot is stale or when no instance reports the gauge at all. Without it, a stalled refresh looks the same as a quiet cluster.
Alert from a scheduled health check
Section titled “Alert from a scheduled health check”cratis chronicle diagnose checks the selected event store and namespace. It exits with a non-zero status for failed partitions, quarantined observers, an unreachable server or checks it could not complete. Schedule it and configure the scheduler’s failure notifications if you do not have a metrics pipeline.
Authenticate with client credentials in the connection string: chronicle://<client-id>:<client-secret>@<host>:35000. See server connection strings for the format. The CLI reads the connection string from CHRONICLE_CONNECTION_STRING; see connecting the CLI.
This GitHub Actions workflow checks every 15 minutes. Replace my-store and the namespaces with your own, store the connection string in the repository secret, and enable notifications for failed workflow runs:
name: Chronicle healthon: schedule: - cron: "*/15 * * * *" workflow_dispatch:
jobs: diagnose: runs-on: ubuntu-latest strategy: fail-fast: false matrix: namespace: [Default] steps: - uses: actions/setup-dotnet@v4 with: dotnet-version: "10.0.x" - name: Install the Cratis CLI run: | dotnet tool install -g Cratis.Cli echo "$HOME/.dotnet/tools" >> "$GITHUB_PATH" - name: Check for failed partitions env: CHRONICLE_CONNECTION_STRING: ${{ secrets.CHRONICLE_CONNECTION_STRING }} run: cratis chronicle diagnose -e my-store -n ${{ matrix.namespace }} -o jsonThe same command works from a Kubernetes CronJob or any cron-style scheduler with the CLI installed. Add one run per namespace you care about; the matrix above does that.
Abandoned client deliveries
Section titled “Abandoned client deliveries”A subscriber timeout is still a Timeout failed-partition attempt, not a successful recovery when a late answer
arrives. These kernel log messages identify what happened after the caller stopped waiting:
Delivery expired before dispatch: the batch waited past its deadline and was never sent to the client mediator.Late result for abandoned delivery: the client answered after the deadline; the result was discarded. The log includes the observer, partition, result state and lateness.Abandoned delivery hold expired: the subscriber released its turn at the configured maximum hold without an answer.Fence released by maximum hold: recovery reached that deadline without proving the original target drained. Remote user code may still be running. A replacement subscription is not evidence that it stopped.
The Cratis.Chronicle meter also exports these instruments:
| Instrument | Kind | Tags / unit | Meaning |
|---|---|---|---|
abandoned-delivery-late-results |
Counter | state |
Results discarded after the caller’s deadline, by result state |
abandoned-delivery-hold-expirations |
Counter | None | Subscriber turns released by the maximum hold |
abandoned-delivery-fence-duration |
Histogram | Seconds | Time the subscriber remained held after the delivery deadline |
Sustained late results suggest a slow client or a timeout that is too short. Hold expirations warrant checking client connectivity and stuck handlers before retrying. These instruments deliberately omit connection and partition tags to avoid unbounded metric cardinality. See configuration for hold limits, disabled behavior and restart limitations.
Configure alerts
Section titled “Configure alerts”The alerts section at the root of chronicle.json controls which observers raise alerts and when. Environment variables use the Cratis__Chronicle__Alerts__ prefix.
{ "alerts": { "enabled": true, "incidentMetricsEnabled": true, "excludeObservers": ["my-noisy-reactor-*"], "conditions": { "partition-failing": { "raiseAfter": "00:15:00" } } }}| Setting | Default | Meaning |
|---|---|---|
enabled |
true |
Turns alerting on or off. |
incidentMetricsEnabled |
true |
Reports incident metrics. |
excludeObservers |
[] |
Observer identifiers to leave out of alerting, for known-noisy or development observers. An entry is an identifier or a glob pattern where * matches any number of characters and ? matches one. The kernel’s own alert observers are always excluded. |
conditions.<name>.enabled |
per condition | Turns one condition on or off. |
conditions.<name>.severity |
per condition | Severity to raise the condition with. Leave it out to keep the built-in severity. |
conditions.<name>.raiseAfter |
00:05:00 |
How long a partition has to keep failing, while still being retried, before partition-failing raises. Zero raises immediately and a negative value is rejected at startup. It has no effect on the other conditions. |
Condition names are the hyphenated names, for example partition-failing, partition-retries-exhausted and observer-quarantined, and they are not case sensitive. A condition without an entry, or a setting an entry leaves out, keeps its built-in default. An unknown condition is ignored with a warning the first time it is evaluated. Raise raiseAfter when observers retry through short outages and the default five minutes pages you for failures that recover on their own.
When an alert fires
Section titled “When an alert fires”- See which partitions failed and why:
cratis chronicle failed-partitions list -e <store> -n <namespace>, thenfailed-partitions showfor the error and the failing sequence number. The Workbench shows the same under the event store’s failed partitions. - Fix the cause and deploy.
- Resume what did not recover on its own. A partition that ran out of retries is not retried by an ordinary retry — follow My reactor throws and the stream seems stuck.