Trace Chronicle .NET client operations with OpenTelemetry
Register AddCratisChronicleInstrumentation() to trace Chronicle .NET client operations inside your application with OpenTelemetry. The client creates spans, timed records of operations, for event appends, unit-of-work commits and event handling in reactors and reducers.
These spans come from the activity source Cratis.Chronicle.Client. OpenTelemetry only records sources registered with its tracer provider; omitting this source produces no warning, just missing client spans.
Configure this instrumentation in your application host. To collect the kernel’s telemetry as well, configure server OpenTelemetry separately.
Register the instrumentation
Section titled “Register the instrumentation”AddCratisChronicleInstrumentation() is an extension on TracerProviderBuilder in the Cratis.Chronicle.AspNetCore package, in the Cratis.Chronicle.AspNetCore.OpenTelemetry namespace. Add it next to the other instrumentation when you configure tracing:
using Cratis.Chronicle.AspNetCore.OpenTelemetry;using Microsoft.AspNetCore.Builder;using Microsoft.Extensions.DependencyInjection;using OpenTelemetry.Resources;using OpenTelemetry.Trace;
public static class ClientTracingRegistration{ public static void Configure(string[] args) { var builder = WebApplication.CreateBuilder(args);
builder.AddCratisChronicle();
builder.Services .AddOpenTelemetry() .ConfigureResource(resource => resource.AddService("orders-api")) .WithTracing(tracing => tracing .AddCratisChronicleInstrumentation() .AddOtlpExporter());
var app = builder.Build(); }}The snippet also needs the OpenTelemetry.Extensions.Hosting and OpenTelemetry.Exporter.OpenTelemetryProtocol packages, which provide AddOpenTelemetry() and AddOtlpExporter().
If your host does not reference Cratis.Chronicle.AspNetCore, such as a worker service, register the source by name instead. ClientActivity.SourceName is the constant Cratis.Chronicle.Client:
using Cratis.Chronicle.Diagnostics.OpenTelemetry.Tracing;using Microsoft.Extensions.DependencyInjection;using Microsoft.Extensions.Hosting;using OpenTelemetry.Trace;
public static class ClientTracingAddSourceRegistration{ public static void Configure(string[] args) { var builder = Host.CreateApplicationBuilder(args);
builder.AddCratisChronicle();
builder.Services .AddOpenTelemetry() .WithTracing(tracing => tracing .AddSource(ClientActivity.SourceName) .AddOtlpExporter()); }}What you will see
Section titled “What you will see”By default, span operation and display names start with client. exactly as before. Set ChronicleTelemetryOptions.SpanNames to ClientSpanNames.Convention through WithTelemetry, or configure Telemetry:SpanNames as Convention, to opt in to cratis.chronicle.client.* names. Each operation still emits one span. The telemetry name reference lists all mappings and the legacy keys deprecated for removal in the next major version.
The tags below hold identifiers, not event content. Both naming modes also carry shared keys such as cratis.event_store.name, cratis.event_sequence.id, and cratis.correlation_id alongside these legacy keys. Both copies of an event source id follow the same identifier exposure setting.
| Span | Kind | Tags |
|---|---|---|
client.event_sequence.append |
Client | event_store_name, namespace_name, event_sequence_id, event_source_type, event_source_id |
client.event_sequence.append_many |
Client | event_store_name, namespace_name, event_sequence_id |
client.unit_of_work.commit |
Client | correlation_id |
client.unit_of_work.rollback |
Client | correlation_id |
client.reactor.handle |
Consumer | event_store_name, namespace_name, event_sequence_id, reactor_id |
client.reducer.handle |
Consumer | event_store_name, namespace_name, event_sequence_id, reducer_id |
A reactor or reducer span covers one delivery of events to that observer. Replay notifications do not produce one.
Failed append and handle operations set Error status and error.type to an exception type or a low-cardinality failure code, without recording exception messages. Append spans also carry cratis.chronicle.append.outcome: success, error, concurrency_violation, or constraint_violation.
Send the spans to a backend
Section titled “Send the spans to a backend”AddOtlpExporter() follows the standard OpenTelemetry environment variables, so the same code works against any backend that accepts OTLP. Set the endpoint, and the headers when your backend needs them, in the environment of the application:
OTEL_EXPORTER_OTLP_ENDPOINT=https://otlp.example.comOTEL_EXPORTER_OTLP_HEADERS=x-api-key=<your-api-key>OTEL_SERVICE_NAME=orders-apiRead the key from your secret store or deployment configuration. Do not commit it, and do not put it in a tag or a log line.
Try it locally with the Aspire Dashboard
Section titled “Try it locally with the Aspire Dashboard”For a local check, run the .NET Aspire Dashboard. This container receives OTLP traces and lets you inspect them in a browser:
docker run --rm -d --name aspire-dashboard \ -p 18888:18888 -p 4317:18889 \ -e DOTNET_DASHBOARD_UNSECURED_ALLOW_ANONYMOUS=true \ mcr.microsoft.com/dotnet/aspire-dashboard:latestStart your application with the dashboard as the endpoint, then append an event:
OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:4317 dotnet runOpen http://localhost:18888, go to Traces, and look for your service. You should see client.event_sequence.append for the append, and, if a reactor observes that event, a client.reactor.handle span. Stop the dashboard with docker stop aspire-dashboard.
The dashboard allows anonymous access in this setup. Use it on your own machine only.
When no spans appear
Section titled “When no spans appear”If client spans are missing, check the path from registration to export in this order:
- Confirm that
AddCratisChronicleInstrumentation()orAddSource("Cratis.Chronicle.Client")is in the sameWithTracingconfiguration as the exporter. The server’sCratis.Chroniclesource does not enable client spans. - Append an event so there is an operation to record, then look again.
- Check the exporter endpoint. Its default is
http://localhost:4317, which only works when a collector or backend is listening there. - Check sampling. A sampler that records nothing, or a parent span that was not sampled, can exclude these spans along with the rest.
Keep sensitive data out of traces
Section titled “Keep sensitive data out of traces”The spans Chronicle creates identify what happened and where, not what the event said. Two tags can still hold values you care about:
event_source_idis the client event source id exactly as you appended it by default (Raw). Kernel spans usecratis.eventsource.id. If you use an email address, a national id or another personal value, that value reaches your tracing backend unless you opt intoHashedorOmitted.correlation_idis the correlation id of the unit of work. It is meant to be searched for, so do not derive it from anything private.
Apply the same checks when adding your own tags: include only identifiers and counts that your tracing backend is allowed to hold. Keep credentials, tokens, event content and read model content out of traces.
Choose how event source ids appear
Section titled “Choose how event source ids appear”The client defaults to EventSourceIdTracing.Raw. Opt into Hashed to send a keyed pseudonym, or Omitted to leave out event_source_id. Both modes keep the original id in append requests and storage. Client spans use event_source_id; kernel spans use cratis.eventsource.id.
Supply a key of at least 32 characters from your secret store before starting the host. This worker example reads it from deployment configuration; the same WithTelemetry callback works with the ASP.NET Core Chronicle builder:
using Cratis.Chronicle;using Cratis.Chronicle.Diagnostics;using Microsoft.Extensions.Configuration;using Microsoft.Extensions.Hosting;
var builder = Host.CreateApplicationBuilder(args);var hashKey = builder.Configuration["ChronicleSecrets:EventSourceIdHashKey"];
builder.AddCratisChronicle(configure: chronicle => chronicle.WithTelemetry(options =>{ options.EventSourceIdTracing = EventSourceIdTracing.Hashed; options.EventSourceIdHashKey = hashKey;}));
await builder.Build().RunAsync();Hashed fails validation without a sufficiently long key. Raw and Omitted do not require one. The hash is HMAC-SHA256 truncated to 128 bits, encoded as 32 lowercase hexadecimal characters. Use the same key in clients and the kernel to correlate ids across processes. Rotating the key changes the hashes and breaks correlation with earlier traces.
For local development you can explicitly select EventSourceIdTracing.Raw. It exposes the id to exporters; do not use it with personal ids unless your tracing backend is allowed to hold that data. These options do not redact logs. See event source ids in telemetry for kernel configuration, metric exclusions, and log levels.
Correlating spans
Section titled “Correlating spans”client.unit_of_work.commit and client.unit_of_work.rollback carry the unit of work’s correlation_id. The append, reactor and reducer spans do not carry it, so searching on a correlation id finds the commit or rollback span only.
See also
Section titled “See also”- Open Telemetry for the server’s own telemetry
- Get alerted when observers stop processing
- Troubleshooting