Troubleshooting

Most “it’s not working” moments with Chronicle come down to a handful of causes. This page collects them. If your issue isn’t here, the Glossary and the feature guides go deeper.

My read model is empty or out of date

A read model is built by a projection that runs after events are appended — it is eventually consistent, not instant. A few things to check, in order:

• Did the events actually get appended? Look at the event sequence. No events in, no read model out.
• Did you give the projection time? Immediately after appending, the projection may not have processed yet. For tests, wait for the read model rather than asserting instantly.
• Does the projection map the events you appended? If the read model only handles BookAdded but you appended BookBorrowed, that event won’t change it.
• Is the property mapping right? AutoMap matches by name. A Title on the event maps to Title on the read model; a mismatch means the value silently doesn’t flow.

My read model’s child collection is null

A child collection with no children is stored as an absent field rather than as [] — deliberately, so that a parallel replay cannot erase a child a sibling event already added. Chronicle fills that absence in for you when the property is declared non-nullable, and leaves it as null when you declared it nullable to keep “no children yet” distinguishable. Two things do not close the gap: initial values are dropped for a children collection, and reading the document with the MongoDB driver instead of through Chronicle bypasses the rule entirely. See Empty child collections.

My projection isn’t picking up a change I made

After you change a projection, the existing read model still reflects the old logic. Rebuild it by replaying — re-running the projection over historical events. Because events are the source of truth, read models are disposable and safe to rebuild at any time. See Rebuild a read model for the workflow.

To inspect or replay observers, use the Cratis CLI:

• cratis chronicle observers list shows all registered observers.
• cratis chronicle observers show shows detailed state for a specific observer.
• cratis chronicle observers replay rebuilds the entire observer from scratch.

My reactor ran twice (or sent a duplicate notification)

That’s expected — reactors can run more than once for the same event during replay or recovery. The fix is not to prevent it but to make the reactor idempotent: record that the side effect happened and skip it if it already did. Chronicle gives you the key to record it under — see Delivery identity, and Reacting to events for the wider pattern.

My reactor re-ran all its side effects after a rename

The default reactor id is its full type name. A class rename or namespace move creates a new observer, which catches up from the beginning as ordinary observation. OnceOnly does not skip that catch-up.

Preserve the previous stored id explicitly to retain its processing position. Inspect any abandoned observer before removing it; removal does not transfer a position or undo duplicate side effects. See Reactor identity and renaming. The .NET discovery report exposes likely identity changes before registration, but a still-subscribed old instance during a rolling deployment can hide them.

My reactor throws and the stream seems stuck

When a reactor throws, the failing event source partition pauses — by design, so it doesn’t silently skip events. Chronicle retries it on its own, with an exponential backoff, and tries again whenever the observer subscribes, for example after you redeploy. A partition that keeps failing is quarantined once it has used up its retry attempts (10 by default), and a quarantined partition is not retried automatically. The limits are set in observer configuration.

So fix the underlying error (and make the reactor resilient), and redeploy. Then use the Cratis CLI to see where each partition stands and resume what did not recover on its own:

• cratis chronicle failed-partitions list shows all failed partitions.
• cratis chronicle failed-partitions show shows the error message and failing sequence number for a specific partition.
• cratis chronicle observers retry-partition retries the partition after you’ve fixed the bug.
• cratis chronicle observers replay-partition replays the partition from the beginning if the state is corrupt.

Retry one reactor partition from a client

After fixing the cause of a failure, an application can inspect a typed reactor’s failed partitions and retry one without supplying the observer or event-sequence identifiers:

C#

using Cratis.Chronicle.Reactors;

var failedPartitions = await eventStore.Reactors.GetFailedPartitionsFor<InvitationMailReactor>();
var failedPartition = failedPartitions.FirstOrDefault();
if (failedPartition is not null)
{
    var outcome = await eventStore.Reactors.RetryFailedPartitionFor<InvitationMailReactor>(failedPartition.Partition);
    // Handle outcome before reporting recovery as successful.
}

In .NET, ReactorPartitionRetryOutcome.Started means a recovery job started or resumed. PartitionNotFound means the failure was already cleared (or never existed), not that the call threw. ObserverQuarantined and PartitionQuarantined mean retry was refused until the corresponding quarantine is cleared. Unknown means the server returned an outcome this client version does not recognize; do not treat it as success.

View C# snippet source on GitHub

Kotlin

import io.cratis.chronicle.IEventStore
import kotlin.reflect.KClass

suspend fun retryOneReactorPartition(store: IEventStore, reactor: KClass<*>) {
    val failedPartition = store.failedPartitions.getFor(reactor).firstOrNull() ?: return
    store.failedPartitions.retry(reactor, failedPartition.partition)
    // retry returns Unit, so a refused retry of a quarantined partition is not signalled here.
    // Inspect partition state before reporting recovery.
}

View Kotlin snippet source on GitHub

Java

import io.cratis.chronicle.IEventStore;
import io.cratis.chronicle.java.FailedPartitionsJavaBridge;
import io.cratis.chronicle.observation.FailedPartition;

class ReactorPartitionRecovery {
    static void retryOneReactorPartition(IEventStore store, Class<?> reactor) {
        var failedPartitions = store.getFailedPartitions();
        var failures = FailedPartitionsJavaBridge.getFor(failedPartitions, reactor);
        if (failures.isEmpty()) return;

        FailedPartition failed = failures.get(0);
        FailedPartitionsJavaBridge.retry(failedPartitions, reactor, failed.getPartition());
        // The bridge returns void, so a refused retry of a quarantined partition is not signalled here.
        // Inspect partition state before reporting recovery.
    }
}

View Java snippet source on GitHub

Elixir

defmodule MyApp.Events.InvitationSent do
  use Chronicle.Events.EventType, id: "troubleshooting-invitation-sent"

  defstruct [:email]
end

defmodule MyApp.Reactors.InvitationMailReactor do
  use Chronicle.Reactors.Reactor

  @handles MyApp.Events.InvitationSent

  @impl true
  def handle(%MyApp.Events.InvitationSent{}, _context), do: :ok
end

defmodule MyApp.Troubleshooting.RetryReactorPartition do
  alias MyApp.Reactors.InvitationMailReactor

  def retry do
    observer_id = InvitationMailReactor.__chronicle_reactor__(:id)
    {:ok, failed_partitions} = Chronicle.FailedPartitions.get_for(observer_id)

    case failed_partitions do
      [failed_partition | _] ->
        {:ok, outcome} =
          Chronicle.Observers.retry_partition(InvitationMailReactor, failed_partition.partition)

        # Handle outcome before reporting recovery as successful.
        outcome

      [] ->
        :nothing_failed
    end
  end
end

In Elixir, {:ok, :started} means a recovery job started or resumed. {:ok, :partition_not_found} means the failure was already cleared (or never existed), not that the call failed. {:ok, :observer_quarantined} and {:ok, :partition_quarantined} mean retry was refused until the corresponding quarantine is cleared. {:ok, :unknown} means the server returned an outcome this client version does not recognize; do not treat it as success.

View Elixir snippet source on GitHub

TypeScript

TypeScript does not support this workflow yet.

View TypeScript snippet source on GitHub

A retry request is not proof that recovery finished. Inspect the partition before reporting recovery as successful; retrying does not guarantee exactly-once execution of reactor side effects.

To find out about a stuck partition when it happens rather than when someone notices, see Get alerted when observers stop processing.

A partition that ran out of retries before your fix was deployed is quarantined, and does not resume on its own. Inspect it with failed-partitions show first. An ordinary retry is refused for a quarantined partition, so recover it by clearing the quarantine of that one partition. Clearing starts a new retry budget for the partition, keeps its attempt history, and — when you ask for it — retries it straight away. A quarantined observer is a different state: it needs observers clear-quarantine first, and clearing a partition while the observer is quarantined reports that the retry was not started.

replay-partition is not a way out for a reactor handler marked [OnceOnly]: the client skips those handlers for replayed events, so a replay never runs the handler that failed. Remember that every retry runs the handler again, so a side effect the handler completed before it failed happens again unless you guard it; see Delivery identity.

Clear the quarantine of one reactor partition from a client

C#

using Cratis.Chronicle.Observation;
using Cratis.Chronicle.Reactors;

var failedPartitions = await eventStore.Reactors.GetFailedPartitionsFor<InvitationMailReactor>();
var quarantined = failedPartitions.FirstOrDefault(partition => partition.IsQuarantined == true);
if (quarantined is not null)
{
    var result = await eventStore.Reactors.ClearFailedPartitionQuarantineFor<InvitationMailReactor>(
        quarantined.Partition,
        retryImmediately: true);
    // Handle result.Outcome and result.RetryOutcome before reporting recovery as successful.
}

In .NET, ReactorPartitionQuarantineClearOutcome.Cleared means the quarantine was cleared and the partition has a new retry budget; its attempt history is kept. NotFound means the partition is not among the failed partitions, and NotQuarantined means it is failed but was not quarantined; neither changed anything. Unknown means the server returned an outcome this client version does not recognize. RetryOutcome is only meaningful when retryImmediately is true and the quarantine was cleared: ObserverQuarantined there means the partition was cleared but the retry did not start because the observer itself is quarantined.

View C# snippet source on GitHub

Every refusal has its own outcome, so nothing reports success when it changed nothing. The same operation is available on the Observers service as ClearPartitionQuarantine.

An observer may have skipped events

Chronicle counts and logs a suspected skip when an observer’s position moves past a live event and no delivery path the observer saw covers it. It is a warning to investigate, not proof that the event was lost: a catch-up job may have delivered it without reporting it. The counters chronicle-observer-events-suspected-skipped and chronicle-appended-events-queue-suspected-undelivered-events are described in Open Telemetry.

The observer writes this warning, inside the scope of the observer it concerns:

Observer may have skipped 1 event(s) of partition <partition>, first at sequence number <number> (<reason>); its position is <position>. This is a suspicion, not proof: investigate and consider replaying the partition

The reason says how the position moved past the event:

Reason	What happened
HeldBackWhilePreparingCatchUp	The event arrived while the observer was preparing a catch-up, was held back, and the position later moved past it.
HeldBackWhileReplaying	The event arrived while its partition was replaying, was held back, and the position later moved past it.
HeldBackWhileCatchingUp	The event arrived while its partition was catching up, was held back, and the position later moved past it.
PassedByPartitionJob	The event arrived after another partition’s job had already moved the observer’s position beyond it, so the observer treated it as handled.

The appended-events queue writes a different warning when handing events to an observer fails: Dispatching <count> event(s) <first>-<last> of partition <partition> to observer <observer> failed; they may not have been delivered. The queue cannot tell whether the observer got to handle them.

Warnings are rate-limited to 10 per observer, or per queue, per minute. When more were suspected, a further warning says how many were not logged. The counters are never limited.

A suspicion is also raised for an event that a catch-up round delivered when the live copy of that event only reached the observer after the round had finished, because the appended-events queue lagged behind a whole round. The event was handled, so there is nothing to replay.

Investigate whether the partition’s read model or side effects reflect the event. If they do not, consider replaying the partition with cratis chronicle observers replay-partition. For a reactor, remember that a replay skips handlers marked [OnceOnly]. To retry a partition that failed, see My reactor throws and the stream seems stuck.

A constraint is rejecting an append I expected to succeed

Constraints are enforced as events are appended — a uniqueness constraint, for example, rejects a second event that would violate it. Check which constraint fired and whether the value really is a duplicate within its scope (remember namespaces isolate data per tenant).

A projection stalls with “the schema does not declare it”

The full message names the property, the subject, and what the schema does declare:

Could not apply compliance metadata for property 'address.postCode' of 'person-42'
because the schema does not declare it. The schema declares: 'street', 'city'.

Compliance handling walks the whole document — one [PII] marker anywhere switches it on for every value — and looks each property up in the schema to find the classification that applies. This message means the document and its schema have drifted apart: the stored payload carries a property the schema has never heard of. The usual cause is an event type that gained or renamed a property without an event type migration, so old events no longer match the shape the schema now describes. Add the migration and replay.

On the write side the append fails outright; on the read side the failing partition pauses and retries, so the read model stops advancing at that event.

A geospatial Point, LineString or Polygon is never the cause — the walk stops at those by design and never reads their GeoJSON members as properties.

Two shapes are a cause, deliberately: a polymorphic base type and a dictionary both hold members the schema does not declare, and Chronicle would rather fail here than write a value it was told is personal into the store unprotected. Model the data as a value object with declared properties. See Geospatial values are not looked inside.

My traces have no Chronicle spans

The Chronicle client emits its spans on an activity source named Cratis.Chronicle.Client, and OpenTelemetry records a source only after it has been registered. Without the registration nothing fails, the spans are just missing. Add AddCratisChronicleInstrumentation() to your tracer provider configuration. See Trace Chronicle client operations with OpenTelemetry.

I can’t connect to the Chronicle kernel

• Confirm the kernel is running — curl --insecure https://localhost:35000/health prints Healthy when a local kernel is up. For local development, the quickest path is docker run -d --name chronicle -p 127.0.0.1:35000:35000 -p 127.0.0.1:27017:27017 cratis/chronicle:latest-development, which publishes the ports on this machine only. For other setups (Docker Compose, Aspire, or production), see Choose an application host model.
• Confirm the client URL matches the kernel’s address and port — the default is chronicle://localhost:35000.
• Confirm your storage is reachable — the development image bundles MongoDB on port 27017; if you’re using separate storage, verify the kernel can reach it.

See Connection strings and Get started.

Events look wrong after a schema change

When an event type’s shape changes over time, use event type migrations to evolve old events to the new shape rather than mutating history. Events are immutable — you migrate how they’re read, you don’t edit what was written.