Run Chronicle with a Production
A Production is Chronicle for one or more applications. Stage Manager turns it into a running Chronicle: a StatefulSet behind a headless Service, its TLS certificate, its database, an optional Neo4j graph, and a connection Secret that applications use and nothing else. Each application on it keeps its own event stores.
Before you start
Section titled “Before you start”A Production needs these in the cluster, besides the operator:
- cert-manager, which issues Chronicle’s TLS certificate. Without it the Production is
Blockedwith the conditionCertManagerInstalledfalse. - The Percona Operator for MongoDB, only for
database.mode: Managed. Without it a managed Production isBlockedwithDatabaseReadyfalse and the reasonPerconaOperatorNotInstalled. - The Secrets the spec names, in the namespace of the Production.
Create the Production in a namespace listed in the chart’s managedNamespaces, and the Performances that use it in the same namespace: a Performance mounts the connection Secret from its own namespace.
A Production on an external MongoDB
Section titled “A Production on an external MongoDB”apiVersion: v1kind: Secretmetadata: name: ledger-mongo namespace: appsstringData: connectionString: mongodb://chronicle:<password>@mongo.example.internal:27017/?replicaSet=rs0---apiVersion: showrunner.cratis.io/v1alpha1kind: Productionmetadata: name: ledger namespace: appsspec: version: 19.148.0 # the tag of the cratis/chronicle production image replicas: 1 database: kind: MongoDB mode: External connectionSecretName: ledger-mongoversion is a tag of the production image, cratis/chronicle, not of the -development images. Changing it rolls the StatefulSet one replica at a time.
What Stage Manager creates
Section titled “What Stage Manager creates”Every resource is named after the Production and owned by it, so deleting the Production removes them. The volumes of a managed database or graph are kept.
| Resource | Name | What it is |
|---|---|---|
| StatefulSet | <name>-chronicle |
Chronicle, replicas of it, from cratis/chronicle:<version> |
| Service | <name>-chronicle |
Headless. Its port 35000 is named chronicle, so DNS answers _chronicle._tcp.<name>-chronicle.<namespace>.svc.cluster.local and chronicle+srv:// finds every ready replica |
| Issuer, Certificate | <name>-selfsigned, <name>-ca |
A self-signed certificate authority for the Production |
| Certificate | <name>-chronicle-tls |
Chronicle’s certificate, written as a PKCS#12 keystore for Chronicle and with ca.crt for clients |
| Secret | <name>-chronicle-keystore |
The generated keystore password |
| Secret | <name>-chronicle-credentials |
The generated administrator password and the client applications connect as |
| Secret | <name>-chronicle-connection |
The connection Secret, reported in status.connectionSecretName |
| PerconaServerMongoDB, Secret | <name>-mongodb, <name>-mongodb-connection |
A managed database and the connection Chronicle reads from it |
| StatefulSet, Service, Secret | <name>-neo4j |
A managed graph, its address and its generated password |
Secrets that hold a generated value are written once and kept: the administrator, the client and the graph keep their credentials across reconciliations and restarts of the operator.
Chronicle is applied only once its certificate is issued and its database is ready, so it never starts without them.
Connect an application
Section titled “Connect an application”The connection Secret holds everything an application needs, and an application needs nothing else:
| Key | Holds |
|---|---|
connectionString |
chronicle+srv://<client>:<secret>@<name>-chronicle.<namespace>.svc.cluster.local |
clientId, clientSecret |
The client Chronicle bootstraps for applications |
ca.crt |
The certificate authority Chronicle’s certificate chains to |
A Performance uses it through productionRef; see Run an application with a Performance. Any other application mounts the Secret and reads connectionString. To validate Chronicle’s certificate, mount ca.crt as well, set Cratis__Chronicle__Tls__CaCertificatePath to its path and add skipTlsValidation=false to the connection string.
status.chronicleAddress is the same address without credentials.
Replicas and versions
Section titled “Replicas and versions”One replica runs with Chronicle’s default single-node clustering. Above one, the replicas form one Orleans cluster through the database: MongoDB clustering on MongoDB storage and Sql clustering on PostgreSQL or SQL Server, with the Production’s name as the cluster and service id and each pod advertising its own address. Scaling from 1 to 3 restarts the replica it has and starts two more in the same cluster.
Database
Section titled “Database”database.kind is the engine Chronicle stores its events in:
| Kind | Mode | Chronicle storage type |
|---|---|---|
MongoDB (default) |
Managed or External |
MongoDB |
PostgreSql |
External only |
PostgreSql |
SqlServer |
External only |
MsSql |
External
Section titled “External”An External database takes its connection string from the Secret named by database.connectionSecretName, under the key connectionString. Stage Manager creates no database resource for it.
Managed
Section titled “Managed”A Managed database is a PerconaServerMongoDB run by the Percona operator, with these defaults:
| Setting | Default | Meaning |
|---|---|---|
members |
1 |
1 for development, 3 or 5 for production. One member is accepted through the Percona operator’s replica-set-size flag |
oneMemberPerNode |
true |
Run each member on its own node. Set false on a single-node cluster, such as kind |
memoryLimit |
8Gi |
The memory limit of each member |
storageSize, storageClassName |
10Gi, the cluster’s default |
The volume of each member |
nodeSelector |
none | The node labels the members must match |
image |
percona/percona-server-mongodb:7.0.14-8 |
Pinned. Automatic upgrades are off |
TLS between the members is disabled: when cert-manager rotates the certificate authority, the Percona operator otherwise restarts the replica set in a loop. The members are only reachable inside the cluster.
Stage Manager waits for the Percona operator to report the database ready, then composes Chronicle’s connection from the database administrator the Percona operator created into <name>-mongodb-connection.
backup.enabled runs the Percona Backup for MongoDB agent (backup.image, pinned by default) next to every member. It only makes the database ready to be backed up; the storage the backups go to is attached by an Understudy.
vault.mode says where Chronicle keeps the keys of the personal data it encrypts:
| Mode | Meaning |
|---|---|
None (default) |
In Chronicle’s database. A backup of the database then brings back keys that were erased, so use a Vault when erasure must survive restores |
Managed |
A HashiCorp Vault that Stage Manager runs, named <name>-vault |
External |
A Vault you run: vault.address and a token in the Secret vault.tokenSecretRef, under the key token unless key is named |
Chronicle is configured for Vault compliance storage automatically, and always gets its token from a Secret (VAULT_TOKEN), never as a value. A Production does not start Chronicle until its Vault is ready (VaultReady).
Managed Vault
Section titled “Managed Vault”vault.managed.tier chooses the shape:
| Tier | Nodes | Storage | Token of Chronicle |
|---|---|---|---|
Development (default) |
1 | file | the root token |
Production |
3 | Raft, so one node can be lost | a periodic token whose policy reaches only Chronicle’s key paths (secret/data/k1/* and the few paths the Chronicle documentation lists), renewed hourly |
storageSize (default 1Gi), storageClassName and image (pinned by default) are settings too. KV v2 is mounted at secret/.
A helper container next to every node initializes Vault once (the first node), keeps the unseal or recovery key and the root token in the Secret <name>-vault-keys, keeps the node unsealed, and writes the token Chronicle uses to <name>-vault-chronicle as its last step. The helper runs as a ServiceAccount of its own whose Role only reads and creates Secrets. Protect <name>-vault-keys like the Vault itself: whoever can read it can read every key.
vault.managed.unseal.mode chooses how Vault is unsealed whenever it starts:
| Mode | Meaning |
|---|---|
KubernetesSecret (default) |
The helper unseals each node with the key in <name>-vault-keys |
CloudKeyService |
A cloud key service unseals Vault: cloud.provider is Aws (region, kmsKeyId), Azure (vaultName, keyName, optionally tenantId) or Google (project, region, keyRing, cryptoKey), in cloud.settings. cloud.credentialsSecretName is a Secret whose keys become the provider’s environment variables; leave it empty to use workload identity |
TransitVault |
A second Vault outside the cluster unseals it: transit.address, transit.keyName, transit.mountPath (default transit) and transit.tokenSecretRef |
vault: mode: Managed managed: tier: Production unseal: mode: CloudKeyService cloud: provider: Aws settings: region: eu-west-1 kmsKeyId: arn:aws:kms:eu-west-1:111122223333:key/example credentialsSecretName: vault-awsWith a cloud key service or a transit Vault, Vault is initialized with a recovery key, which the helper keeps in <name>-vault-keys. Settings that end up in the Vault configuration may only hold letters, digits and . _ : / @ = , + - and spaces.
Only a token is supported for an external Vault, because Chronicle authenticates with VAULT_TOKEN.
Federation
Section titled “Federation”federation makes the Chronicle of a Production take part in federation. Nothing is on by default.
| Field | Meaning |
|---|---|
enabled, port |
Opens the federation listener (Federation:Enabled), on port 35100 by default, in the container and in a federation port of Chronicle’s Service. The listener presents Chronicle’s certificate |
hosted.enabled, hosted.publicEndpoint, hosted.trustDomain |
Makes Chronicle the certificate authority customers enroll with. Needs enabled. The host of publicEndpoint is added to Chronicle’s certificate |
dial.enabled, dial.eventStore, dial.namespace, dial.logicalSource |
Dials the authority this Chronicle enrolled with (Federation:Dial). It opens no listener and no port |
dial.serve, dial.serveEventTypes |
Also produces this Chronicle’s public events to the authority over the same connection: a Production with only dial set is the sending side of a one-way link. Empty serveEventTypes serves every public event type |
dial.enrollmentCodeSecretRef |
The Secret holding the enrollment code, under the key code unless key is named. Mounted as a file |
federation: dial: enabled: true eventStore: ledger logicalSource: direct serve: true enrollmentCodeSecretRef: name: direct-enrollmentThe authority still needs a policy that lets this Chronicle in; that is configured on the authority, not here.
graph gives the Production a Neo4j that graph projections land in.
graph.mode |
What happens |
|---|---|
None (default) |
No graph. The kernel’s Neo4j settings are left unset |
Managed |
Neo4j Community in a StatefulSet, with a persistent volume and a generated password |
External |
A Neo4j server elsewhere. graph.credentialsSecretName names a Secret with its bolt URI under server and its password under password; graph.userName is the user, neo4j by default. No Neo4j resource is created |
graph.edition is Community (default) or Enterprise:
- Community has one user database, so the kernel stores every graph of every event store in it (
databaseMode: Single). A managed graph is always Community. - Enterprise gives each event store and namespace its own database (
databaseMode: PerNamespace), prefixed bygraph.databasePrefix, the name of the Production by default, so several Productions can share one server.
A managed graph takes storageSize (10Gi), storageClassName, nodeSelector and image (neo4j:5.26.0-community, pinned), as a managed database does.
The Neo4j password is never written into the spec, the status, a condition or an environment value: it is mounted from its Secret. The condition GraphReady says whether the graph is ready.
Named sinks
Section titled “Named sinks”Applications can name the sink a read model or graph is stored in. The model only names the sink; the Production maps each name to storage, the way an environment maps its keys.
spec: database: kind: MongoDB mode: Managed graph: mode: Managed sinks: readModels: reporting # optional; empty keeps the database graphs: people # optional; empty keeps the graph of the Production configurations: - name: reporting kind: PostgreSql # MongoDB | PostgreSql | SqlServer | Neo4j connectionSecretRef: name: reporting-pg # key defaults to connectionString - name: people kind: Neo4j graphRef: managed # the Production's managed graph, or a connectionSecretRef| Field | Meaning |
|---|---|
readModels |
The sink read models are stored in when they name none. Must be MongoDB, PostgreSql or SqlServer |
graphs |
The sink graphs are stored in when they name none. Must be Neo4j |
configurations[].name |
The name applications use: letters and digits, separated by single hyphens or underscores. Unique without regard to case |
configurations[].kind |
MongoDB, PostgreSql or SqlServer for read models, Neo4j for graphs |
configurations[].connectionSecretRef |
The Secret, in the namespace of the Production, holding the connection string, or the bolt URI of a Neo4j server |
configurations[].graphRef |
A Neo4j sink only: managed, the Production’s own graph, which needs graph.mode: Managed |
Each configured sink has a condition, SinkConfigured.<name>. A duplicate name, an unknown kind, a SQL or MongoDB sink without connectionSecretRef, a Neo4j sink without exactly one of connectionSecretRef and graphRef, and a Secret that does not exist or lacks its key each make that condition false with a message naming the sink, and the Production Blocked.
The kernel configuration
Section titled “The kernel configuration”From a valid spec, Stage Manager derives Chronicle’s configuration:
| Setting | From |
|---|---|
Cratis__Chronicle__Storage__Type |
database.kind |
Cratis__Chronicle__Storage__ConnectionDetailsFile |
The database’s connection, mounted as a file |
Cratis__Chronicle__Neo4j__ServerFile, …__PasswordFile |
The graph’s bolt URI and password, mounted as files |
Cratis__Chronicle__Neo4j__UserName, …__DatabaseMode, …__DatabasePrefix |
The graph’s user, Single or PerNamespace, and the prefix on Enterprise |
Cratis__Chronicle__Sinks__ReadModels, Cratis__Chronicle__Sinks__Graphs |
sinks.readModels, sinks.graphs |
Cratis__Chronicle__Sinks__Configurations__<name>__Type |
MongoDB, SQL for PostgreSql and SqlServer, or Neo4j |
Cratis__Chronicle__Sinks__Configurations__<name>__ConnectionDetailsFile |
A MongoDB or SQL sink’s connection, mounted as a file |
Cratis__Chronicle__Sinks__Configurations__<name>__ServerFile |
A Neo4j sink’s bolt URI, mounted as a file. A sink on the managed graph also gets its PasswordFile, UserName and DatabaseMode |
A Production with no sinks and no graph sets only the storage type and its connection file.
Connections are never written into the spec, the status or an environment value. Each is mounted from its Secret into /run/secrets/chronicle, as a file named after the setting it holds, such as /run/secrets/chronicle/Cratis__Chronicle__Storage__ConnectionDetails. The kernel reads the file through the matching …File setting, once, at startup. Stage Manager sets only the …File form, never the value itself: Chronicle refuses to start when both are set.
The Neo4j ServerFile settings, and PasswordFile on the top-level neo4j section, need the Chronicle release that contains Cratis/Chronicle#5771. An older kernel ignores them and finds no graph.
The keystore password, the administrator password and the client secret reach Chronicle as environment variables that reference their Secrets, so their values never stand in the StatefulSet.
Status
Section titled “Status”| Field | Meaning |
|---|---|
phase |
Blocked when nothing is applied, Progressing while something is not ready, Ready when Chronicle, its database and its graph are |
chronicleAddress |
The chronicle+srv:// address of Chronicle |
connectionSecretName |
The connection Secret, once the certificate authority is issued |
database, graph, vault |
The database kind and mode, the graph mode and edition, and the vault mode and tier, the Production runs with |
conditions |
SpecValid, VaultSupported, DatabaseModeUnchanged, GraphModeUnchanged, VaultModeUnchanged, CertManagerInstalled, SinkConfigured.<name> per sink, FederationConfigured when the Production dials out and its enrollment code is missing, CertificatesReady, DatabaseReady, GraphReady, VaultReady with a Vault, ConnectionAvailable and ChronicleReady |
A Production is never half configured. An invalid spec, a missing Secret, a missing operator or a refused change makes it Blocked, and nothing is applied until it is resolved.
Changes that are refused
Section titled “Changes that are refused”Changing database.kind, database.mode, graph.mode, graph.edition, vault.mode or vault.managed.tier on a Production that runs would leave its data or its keys behind, so it is refused: the Production is Blocked with DatabaseModeUnchanged, GraphModeUnchanged or VaultModeUnchanged false, naming the field and what it runs with. Set the field back, or create a new Production.
Performances check and read their sinks
Section titled “Performances check and read their sinks”A Performance on a Production lists the sinks its application uses in application.sinks. It is Blocked with the condition SinksMapped false, naming each sink the Production does not configure, before Chronicle would refuse the registration at run time. See Run an application with a Performance.
The Production also gives the application the read side of what it uses, so queries read where the kernel writes:
| Sink or graph | What the application gets |
|---|---|
A MongoDB sink |
Its connectionSecretRef key, mounted as /run/secrets/sinks/<name>, and Cratis__MongoDB__Sinks__<name>__ServerFile naming it |
| The graph of the Production | The Cratis__Neo4j__… settings of Cratis.Arc.Neo4j, when the application sets application.graph or uses a Neo4j sink with graphRef: managed |
A PostgreSql or SqlServer sink, or a Neo4j sink on another server |
Nothing: Cratis Arc cannot read it. The condition SinksReadable is false as a warning, and the Performance still runs |
The Performance mounts these Secrets from its own namespace, so create it in the namespace of the Production. When one is not there, it is Blocked with SinksBound or GraphBound false, naming the Secret; nothing is copied between namespaces. Reading a named MongoDB sink needs the Cratis Arc release with Cratis/Arc#3770.