C# observability in Cratis Arc

Cratis Arc instruments its command and query pipelines so you can locate time spent in validation, authorization and handlers, and distinguish rejected operations from errors. Subscribe to its activity source and meters through OpenTelemetry to collect that telemetry without adding instrumentation to your handlers.

Arc supplies the instrumentation; your host configures collection and export. The listeners and tracing sampler determine which spans are recorded.

Set up OpenTelemetry with one call

The quickest way to collect Arc’s telemetry is the shared Cratis.OpenTelemetry package, which is separate from Cratis.Fundamentals. Add the package to your host and call AddCratisOpenTelemetry:

using Cratis.OpenTelemetry;

builder.AddCratisOpenTelemetry(options => options.ServiceName = "my-app");

The call subscribes to every Cratis.* activity source and meter, so Arc’s Cratis.Arc source and meter are included alongside the Chronicle client. It also adds ASP.NET Core, HttpClient and runtime instrumentation. It registers an OTLP exporter only when an endpoint such as OTEL_EXPORTER_OTLP_ENDPOINT is configured. To add Cratis to an existing AddOpenTelemetry() call, use services.AddOpenTelemetry().WithCratis(). It brings its own exporter, so do not also call UseOtlpExporter(). If you own the whole pipeline, call tracing.AddCratisInstrumentation() and metrics.AddCratisInstrumentation() instead. The OpenTelemetry setup guide covers configuration, resource identity and optional instrumentation.

Subscribe to the source and the meter yourself

Without the shared package, subscribe to Arc directly. Arc publishes its spans on an activity source named Cratis.Arc and its metrics on a meter of the same name. Both names are available as constants on WellKnownDiagnostics, so you do not have to repeat the string:

using Cratis.Arc;

builder.Services.AddOpenTelemetry()
    .WithTracing(tracing => tracing
        .AddSource(WellKnownDiagnostics.ActivitySourceName)
        .AddAspNetCoreInstrumentation())
    .WithMetrics(metrics => metrics
        .AddMeter(WellKnownDiagnostics.MeterName, "Cratis.Arc.MongoDB")
        .AddAspNetCoreInstrumentation());

This configures subscriptions and ASP.NET Core instrumentation, not an exporter. Add an exporter to send the collected telemetry to your backend. There is no Arc-side switch to turn tracing on; the tracing sampler still controls which spans are recorded.

The spans

Pipeline activities are Internal; the controller action spans cratis.arc.command.action and cratis.arc.query.action are Server activities. Each span keeps one stable name, which is what tracing backends show as the span name and what you filter and group on. The command or query it covers is an attribute, cratis.arc.command.type or cratis.arc.query.name, so a dashboard keyed on a span name keeps working as your application adds commands and queries.

Span	Raised when
cratis.arc.command.execute	a command runs through the pipeline
cratis.arc.command.validate	a command is validated without being run
cratis.arc.command.filter	the command filters run
cratis.arc.command.authorize	an authorization filter decides on the command
cratis.arc.validator.invoke	a validator runs, for a command, a query or a nested model
cratis.arc.command.provide	a model-bound command’s Provide() method runs
cratis.arc.command.handle	the command handler runs
cratis.arc.query.push	a changed result of an observable query is pushed to a subscriber
cratis.arc.command.action	a controller-based command action is invoked
cratis.arc.query.perform	a query runs through the pipeline
cratis.arc.query.filter	a query filter runs
cratis.arc.query.action	a controller-based query action is invoked
cratis.arc.identity.resolve	identity details are resolved for a request

The child spans nest inside the command span, which itself nests inside the request span your ASP.NET Core instrumentation already creates. For a RegisterAuthor command, the execute, authorize, provide and handle spans carry cratis.arc.command.type = MyApp.Authors.RegisterAuthor, and the validator span carries the validator type:

POST /api/authors/register
└── cratis.arc.command.execute
    ├── cratis.arc.command.filter
    │   ├── cratis.arc.command.authorize
    │   └── cratis.arc.validator.invoke
    ├── cratis.arc.command.provide
    └── cratis.arc.command.handle

A validator runs once for every instance of the type it validates, so a command carrying a list of 500 items would otherwise add 500 spans. A validator gets a span the first time it runs in a validation, not for each item after that, and one validation adds at most 16 validator spans.

For a request over HTTP, the command or query span starts before Arc prepares authorization, so a request turned away at that point still has a span and is still counted in the metrics.

Attributes

Attribute	On	Value
cratis.arc.command.type	execute, validate, authorize, provide, handle	the full name of the command type
cratis.arc.command.outcome	execute, validate	success, validation, authorization, append_rejected, cancelled or error
cratis.arc.command.key.type	execute, validate	the type the command declares its key as, such as its event source id with the Chronicle integration
cratis.arc.query.name	perform	the fully qualified name of a known query; absent for a name that matches no query
cratis.arc.query.transport	perform	snapshot, observable, or unknown when the query failed before it ran
cratis.arc.query.name	push	the fully qualified name of a known query; _other for a name that matches no query
cratis.arc.query.transport	push	websocket or sse, the transport the result is pushed over
cratis.arc.query.outcome	perform	success, validation, authorization, cancelled or error
cratis.arc.validator.type	cratis.arc.validator.invoke	the full name of the validator type
cratis.arc.validation.result_count	cratis.arc.validator.invoke	how many results the validator produced
cratis.correlation_id	execute, validate, perform	the correlation id of the operation
cratis.tenant	execute, validate, perform	the tenant the operation runs for, when one was resolved while it ran; a query records it only when it was authorized for a principal
cratis.tenant.id	execute, validate, perform	the same resolved tenant; emitted alongside the legacy cratis.tenant tag
http.route	controller command/query action spans	the route template, alongside legacy route_template; never a concrete request path or URL

The execute, validate, filter and perform spans also keep the command_type and query_name attributes they carried in earlier versions. query_name is _other for a name that matches no query. Prefer the cratis.arc.* attributes for anything new.

Arc records the tenant once the pipeline has established who the operation runs as, without resolving a tenant itself, since that would change what the operation sees. The tenant is held per asynchronous flow, so one that a filter or validator resolves inside its own call is not visible to the pipeline afterwards. An operation turned away before that point, for example a command with no handler, carries no tenant.

Outcomes

The outcome attribute says how an operation ended. Following OpenTelemetry, only an error outcome sets the span status to Error: a failure that is not a validation, authorization or event store rejection and not a cancellation. A command rejected by validation, by authorization or by the event store is the application working as intended, so its span status stays unset; filter on the outcome attribute to find those. cancelled means the operation’s cancellation token was cancelled, normally because the caller went away, and it leaves the status unset too.

The child spans follow the same rule. An exception thrown from Handle(), Provide() or a validator adds an exception event to that span, but sets its status to Error only when it is an error: not when it is a validation failure (an exception implementing IValidationFailure, which Arc reports as a validation outcome) and not once the operation was cancelled.

Depending on how an operation ended, events on its span say why:

Event	Raised when	Attributes
cratis.arc.validation.failed	once for each validation result that blocked the operation, up to 16	cratis.arc.validation.severity, cratis.arc.validation.members, cratis.arc.validation.reason and, when set, cratis.arc.validation.reason_detail
cratis.arc.authorization.denied	authorization denied the operation, on the operation span and the authorize span	—
exception	an exception was thrown, on the span it was thrown in and on the command or query span that turned it into a result	exception.type

append_rejected is the outcome when the rejection came from the event store rather than from a rule: a constraint violation or a concurrency conflict. The reason says which, and for a constraint the reason detail is the constraint name.

What the spans never carry

Arc’s pipeline instrumentation records types, names, correlation and tenant identifiers, member names and outcomes, not command, query or event payload values. A validation event names the member and rule but omits the message, which can quote an input value. An exception event carries the exception type but omits its message and stack trace. The event source id is recorded by its type, never by its value.

This applies to the telemetry Arc adds, not to application-added tags, other instrumentation or logs. Review those outputs separately for sensitive data, including values marked [PII] or [NotAudited].

Names as constants

Every span, attribute, event and metric name in this page is a constant on WellKnownTelemetryNames, and every outcome is a constant on WellKnownOperationOutcomes, so dashboards, alerts and tests built in C# can use them instead of repeating the strings.

What this is useful for

The split between validate and execute is the one worth watching. Validation that reaches a database — a uniqueness rule, a state-dependent check — is easy to write and easy to forget, and it shows up here as time spent before the handler ever ran.

cratis.arc.identity.resolve is the other common surprise. The span covers each identity lookup, such as the frontend’s /.cratis/me request. Every lookup calls your identity-details provider, so one that queries a store puts that query on the critical path of every lookup.

Metrics

Arc records these on the Cratis.Arc meter:

Metric	Type	Unit	Attributes
cratis.arc.command.duration	histogram	s	cratis.arc.command.type, cratis.arc.command.outcome
cratis.arc.command.outcomes	counter	{command}	cratis.arc.command.type, cratis.arc.command.outcome
cratis.arc.query.duration	histogram	s	cratis.arc.query.name, cratis.arc.query.transport, cratis.arc.query.outcome
cratis.arc.query.subscriptions	up-down counter	{subscription}	cratis.arc.query.name, cratis.arc.query.transport
cratis.arc.query.pushes	counter	{push}	cratis.arc.query.name, cratis.arc.query.transport

A command validated without being run is traced but not counted. For an observable query the duration covers setting up the subscription, not how long it stays open. The subscriptions metric goes up when a subscription is established and down exactly once when it ends, whichever way it ends, so its sum is the number of subscriptions open now. The pushes metric counts results the query’s stream emitted; a snapshot or a validation rejection that answers the subscribe is not counted. A push span is a child of the activity that was current when the subscription was established, and links to the activity that caused the change when one is current. The pushes and subscriptions metrics use the websocket and sse transports, where the duration histogram reports observable. Observable queries answered over a single HTTP request are not counted.

Each metric records at most 1,000 distinct command types or query names, the subscription and push metrics included. The limit is fixed; there is no option to change it. Past that, further ones are recorded as _other, so a metric backend never has to hold an unbounded number of series. The limit holds exactly, however many requests arrive at once; only right at the limit can a new name be recorded as _other once before it gets the last free place. A query name that matches no known query is always recorded as _other.

When nothing listens to the meter or the activity source, Arc skips most of the work of recording: it starts no child spans, builds no attributes or events, and does not look up whether a query name is known. It still checks whether anything listens on every operation.

Duration measurements are recorded while the operation activity is still current, so an OpenTelemetry exporter configured for exemplars can associate them with that trace. Arc does not force an exporter or exemplar policy. Query outcomes remain an attribute on cratis.arc.query.duration; there is no additional query-outcomes counter.

MongoDB client metrics

The MongoDB integration keeps its legacy hyphenated instruments on Cratis.Arc. It also emits these dotted instruments on the Cratis.Arc.MongoDB meter (WellKnownMongoDBDiagnostics.MeterName in Cratis.Arc.MongoDB):

Dotted instrument	Type	Legacy instrument on Cratis.Arc
mongodb.open.connections	gauge	mongodb-open-connections
mongodb.checked.out.connections	gauge	mongodb-open-connections (legacy duplicate name)
mongodb.connections.in.pool	gauge	mongodb-connections-in-pool
mongodb.commands	gauge	mongodb-commands
mongodb.failed.connections	counter	mongodb-failed-connections
mongodb.aggregated.commands	counter	mongodb-aggregated-commands

The dotted checked-out and open-connection gauges have distinct names. The legacy duplicate is retained for compatibility; existing dashboards continue to receive the old instruments. Manual OpenTelemetry configuration must subscribe to both meters as shown above. The shared Cratis.* subscription already includes both. Both sets retain the existing Server attribute.

On the JVM

The Kotlin and Java backend uses Micrometer rather than ActivitySource, with its own observation names and setup requirements. See Observability for its recorded operations and correlation context.