Open Telemetry

Chronicle exports telemetry signals through the OpenTelemetry Protocol (OTLP). Telemetry is always enabled and configured via standard OpenTelemetry environment variables.

Signals

Chronicle exports the following signals:

Signal	Description
Metrics	Application and infrastructure metrics (event sequences, observers, .NET runtime)
Traces	Distributed traces for Chronicle operations (gRPC, HTTP, Orleans)
Logs	Structured application logs

Configuration

Telemetry is configured using standard OpenTelemetry environment variables:

Variable	Default	Description
OTEL_EXPORTER_OTLP_ENDPOINT	none	OTLP receiver endpoint (e.g. http://localhost:4317)
OTEL_EXPORTER_OTLP_PROTOCOL	grpc	Export protocol: grpc or http/protobuf
OTEL_EXPORTER_OTLP_HEADERS	none	Additional headers (e.g. API keys for cloud backends)
OTEL_SERVICE_NAME	Chronicle	Service name reported to the telemetry backend

When OTEL_EXPORTER_OTLP_ENDPOINT is not set, Chronicle does not export telemetry but the instrumentation is still active.

Local development with .NET Aspire Dashboard

The Kernel ships with a docker-compose.yml that starts the .NET Aspire Dashboard as a lightweight all-in-one OTLP receiver and telemetry viewer.

aspire-dashboard:
  image: mcr.microsoft.com/dotnet/aspire-dashboard:latest
  environment:
    - DOTNET_DASHBOARD_UNSECURED_ALLOW_ANONYMOUS=true
    - ALLOW_UNSECURED_TRANSPORT=true
  ports:
    - 18888:18888   # dashboard UI
    - 4317:18889    # OTLP/gRPC receiver

Start the services:

docker compose up -d

Then set the endpoint environment variable before starting the Kernel:

OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:4317 dotnet run

Open the Aspire Dashboard at http://localhost:18888 to view metrics, traces, and logs.

Example: Prometheus and Grafana

For production-grade metric collection, connect Chronicle to a Prometheus-compatible OTLP receiver (for example the OpenTelemetry Collector):

OTEL_EXPORTER_OTLP_ENDPOINT=http://otel-collector:4317

Example: cloud observability backends

Most cloud observability backends (Datadog, Honeycomb, Grafana Cloud, Azure Monitor) accept OTLP. Set the endpoint and any required authentication headers:

OTEL_EXPORTER_OTLP_ENDPOINT=https://otlp.example.com
OTEL_EXPORTER_OTLP_HEADERS=x-api-key=your-api-key

Metrics instrumented

Chronicle instruments the following meters:

Meter	Description
Cratis.Chronicle	Chronicle application metrics (events appended, observers, etc.)
Microsoft.Orleans	Orleans distributed actor framework metrics
Grpc.AspNetCore.Server	gRPC server request metrics
.NET runtime	GC, thread pool, and memory metrics from the .NET runtime

Observer failure metrics

Chronicle counts observer failures on the Cratis.Chronicle meter. Every one of these instruments carries the tags EventStore, Namespace, ObserverId and EventSequenceId, and no others. Neither the partition nor the event source id is a tag, so the number of series depends on the number of observers and never on how many partitions fail.

Instrument	Prometheus name	Counts
chronicle-observer-partitions-failed	chronicle_observer_partitions_failed_total	Every failed handling attempt of a partition, including the failure of each retry. It is not a count of failed partitions.
chronicle-observer-partition-retry-attempts	chronicle_observer_partition_retry_attempts_total	Every time a failed partition was evaluated for retry. It is recorded together with each failed attempt, whether or not a retry ends up being scheduled, so it does not count retries that ran.
chronicle-observer-partitions-quarantined	chronicle_observer_partitions_quarantined_total	Partitions that ran out of retry attempts and were quarantined. It does not count an observer being quarantined.
chronicle-observer-quarantined	chronicle_observer_quarantined_total	Times an observer was quarantined.

The Prometheus names are what the OpenTelemetry Collector’s Prometheus exporter and Prometheus’s own OTLP receiver produce by default. These four instruments have no unit and carry their description, so a Prometheus HELP line explains each of them. Earlier Kernels created them with the description in the unit position, which gave them long names such as chronicle_observer_partitions_failed_Number_of_failed_partitions_per_observer_in_a_given_event_store_and_namespace_total and an empty HELP.

An observer has a series on these instruments from its first failure, not before: Chronicle records nothing when an observer starts, so healthy observers add no series and the number of series follows the number of observers that fail. The first value a backend receives for an observer is therefore already 1, and alerts on these counters should not depend on seeing a 0 first. The alerting guide shows how.

The SDK limits each instrument to 500 series. Beyond that it folds the excess into one series without the tags, which can no longer be attributed to an observer. That takes more than 500 observer instances in one process, counted across observers, namespaces and event stores.

Suspected skipped events

Two counters on the Cratis.Chronicle meter show how often an observer’s position moved past a live event without any delivery path covering it. The first has the observer tags EventStore, Namespace, ObserverId and EventSequenceId, and no others. The second has the appended-events queue tags EventStore, Namespace, EventSequenceId and QueueId, plus ObserverId.

Instrument	Prometheus name	Counts
chronicle-observer-events-suspected-skipped	chronicle_observer_events_suspected_skipped_total	Live events an observer’s position moved past without the observer seeing any delivery path cover them.
chronicle-appended-events-queue-suspected-undelivered-events	chronicle_appended_events_queue_suspected_undelivered_events_total	Event deliveries to an observer whose dispatch failed. One event counts once for each observer it was dispatched to.

These counters show suspicions, not proof. A catch-up job may have delivered an event without reporting it, and a dispatch that failed may still have reached the observer before it failed. Counting changes nothing about delivery. The counters count every suspicion, while the matching warnings are rate-limited to 10 per observer, or per queue, per minute, with a summary of how many were not logged. Like the failure counters, they only have a series once something is suspected.

When one of them increases, see An observer may have skipped events.

Open alert incident metrics

Open alert incident metrics are enabled by default. To opt out, set Cratis__Chronicle__Alerts__IncidentMetricsEnabled=false on every silo and restart all silos. Alternatively, set Alerts.IncidentMetricsEnabled to false in chronicle.json:

{
  "Alerts": {
    "IncidentMetricsEnabled": false
  }
}

This startup setting prevents registration of both incident gauges, owner keepalive pings, periodic sampling, and incident-storage reads for metrics. It does not merely filter exported data. It leaves alert raising, incident materialization, incident queries, observer failure counters, and other telemetry unchanged. Setting Alerts.Enabled=false controls alert raising, not incident metrics. To restore incident metrics, set IncidentMetricsEnabled to true and restart every silo. A mixed cluster can still collect and publish metrics from enabled silos; complete the restart before relying on the opt-out. Disable or adjust alerts that depend on these gauges, including availability alerts, because disabled gauges are absent, not zero.

Chronicle publishes the number of open alert incidents on the Cratis.Chronicle meter. The chronicle-alerts-open-incidents gauge is exported to Prometheus as chronicle_alerts_open_incidents (a gauge has no _total suffix). It has one series per bucket, tagged EventStore, Namespace, ObserverId, EventSequenceId, Condition and Severity, and no others. The Severity tag holds the incident’s severity (Warning or Critical).

The value is sampled from incident storage, not counted from live events. One owner grain in the cluster aggregates the open incident rows every 30 seconds and publishes the result; a scrape reads that snapshot and does no storage I/O. An escalation moves an incident between Severity buckets without changing the total.

• A snapshot older than 90 seconds publishes nothing, rather than zero. The same applies to a refresh that cannot run: while incident materialization is catching up or degraded, or storage fails, Chronicle keeps the previous snapshot until it ages out. Absence of the series therefore means “unknown”, not “no incidents”.
• A bucket that empties reports 0 for 5 minutes and is then retired.
• chronicle-alerts-open-incidents-available (chronicle_alerts_open_incidents_available, no tags) is 1 while the instance that owns the snapshot has a fresh one and 0 while its snapshot is missing or stale. Instances that do not own the snapshot report no series.

Aggregate across instances with max, never sum. A silo that is partitioned or hung can keep publishing its last snapshot for up to 90 seconds after another instance has taken ownership, so more than one instance can report a series for a short time. Summing would double count; max by (EventStore, Namespace, ObserverId, EventSequenceId, Condition, Severity) does not.

When the silo that owns the snapshot dies, another silo re-activates the owner within about 30 seconds. The new owner starts with no memory of the buckets that were retiring, so a series can disappear without ever reporting 0, and a backend’s lookback can keep returning the last value briefly. Do not depend on seeing a 0 before a series ends.

Like the other instruments, this gauge is limited to 500 series per process. The series count is the number of observers with incidents, times the conditions and severities they hold.

To be alerted when an observer’s partitions keep failing or run out of retries, see Get alerted when observers stop processing.

Traces instrumented

These are the server’s sources. The Chronicle client in your application emits its own spans on the Cratis.Chronicle.Client source, which you register yourself; see Trace Chronicle client operations with OpenTelemetry.

Chronicle traces the following activity sources:

Source	Description
Cratis.Chronicle	Chronicle internal operations
Microsoft.Orleans.Runtime	Orleans runtime activities
Microsoft.Orleans.Application	Orleans application activities
HTTP client	Outgoing HTTP requests
ASP.NET Core	Incoming HTTP and gRPC requests
gRPC client	Outgoing gRPC calls