---
title: Test AI conformance
editUrl: https://github.com/Cratis/cli/edit/main/Documentation/ai/test.md
description: Run opt-in, deterministic AI conformance fixtures in temporary workspaces without changing the invoking repository.
lastUpdated: 2026-10-25T00:00:00.000Z
---

`cratis ai test` runs deterministic fixture checks against isolated harness sessions. It is opt-in: invoking this command can spend model quota. Ordinary installation, builds and specs do not run it.

```bash
cratis ai test --source ../AI --harnesses claude,codex --output json
cratis ai test --fixture example --timeout 60 --max-budget-usd 0.25
```

The command reads the current repository's `.cratis/ai.json` and installation manifest. Missing installations and modified managed files are rejected. `--source` (or `CRATIS_AI_SOURCE`) selects a corpus checkout and permits corpus-CI mode without a clean installation. Without a repository configuration, corpus-CI mode selects the corpus's available profiles and languages. The default source otherwise follows `cratis ai install`.

For every matching fixture and harness, fixture input is copied into a fresh temporary workspace and the fixture's profiles and languages are installed through the normal installer, with only the harness under test. Normal mode intersects declared selections with the invoking configuration; source mode uses the fixture's declared selections. Empty declarations inherit the invoking selection. The invoking repository is not changed. This is filesystem separation, **not a security boundary**: fixtures, their command checks and harnesses must be trusted. Harness authentication and host tools remain available. User-memory isolation is reported as `unknown`.

## Options

| Option | Meaning |
| --- | --- |
| `--harnesses claude,codex` | Select adapters; defaults to supported adapters in the configuration. |
| `--fixture <name>` | Select a fixture by name; repeat to select several. |
| `--fixtures <directory>` | Override the default `Evaluations/conformance` root in the corpus. |
| `--source <directory>` | Use a local corpus checkout. |
| `--output text\|json` | Human-readable summary or structured report. |
| `--timeout <seconds>` | Lower, never raise, a fixture's timeout. |
| `--max-budget-usd <amount>` | Lower, never raise, a fixture's requested model budget. |
| `--install-only` | Run install-phase checks only, without any harness. See [Run without a harness (CI)](#run-without-a-harness-ci). |
| `--keep-transcripts` | Retain redacted transcripts under the run's temporary output directory. |

Claude uses project settings, stream JSON, no session persistence and `acceptEdits`; it does not use `--bare`. Budget and turn flags are supplied when requested. Codex uses `exec --json --ephemeral --ignore-user-config --sandbox workspace-write`; its budget and turn limits are reported as `not_enforced`, not guaranteed. Timeout enforcement applies to both. Missing executables, credentials and unavailable modes are never passes.

## Run without a harness (CI)

`--install-only` prepares each fixture workspace, installs its slice and evaluates only the install-phase checks. No harness is resolved or invoked, so no harness executable, credential or model quota is needed:

```bash
cratis ai test --source <corpus> --install-only --output json
```

The slice is installed once per fixture and harness id. Without `--harnesses`, the ids come from the fixture's `harnesses`, or the configured harnesses when it declares none; the requested id is reported unchanged and that harness's integration files are installed, because install checks may target them. Statuses: `passed` (at least one install check and every non-optional one passed), `failed`, or `skipped` with reason `no install-phase checks`. Every result carries `result_phase: "not_run"`, and the text summary says result-phase checks were not run; nothing is claimed about behavior. Exit codes are unchanged. `--timeout` still bounds each fixture; `--max-budget-usd` is accepted, and the budget and turn bounds report `not_applicable`; `--keep-transcripts` is accepted but there are no transcripts.

In normal mode, when a harness is unavailable after installation, the result is `skipped` and still lists the install-check results.

## Fixture format

Each `<fixtures-root>/<name>/fixture.json` can have a sibling `input/` tree. For example:

```json
{
  "name": "example",
  "description": "Produce a file using installed guidance",
  "prompt": "Create result.txt containing ready.",
  "profiles": [],
  "languages": [],
  "harnesses": ["claude", "codex"],
  "expected_instruction_paths": [".cratis/ai/rules/general.md"],
  "checks": [
    {"type": "file_exists", "path": ".cratis/ai/rules/general.md", "phase": "install"},
    {"type": "file_matches", "glob": "**/result.txt", "pattern": "ready", "phase": "result"}
  ],
  "bounds": {"timeout_seconds": 120, "max_budget_usd": 0.5, "max_turns": 5}
}
```

Empty profile/language/harness restrictions mean any selection. `expected_output`, if supplied, is ignored: there is no model judge. Every fixture must have at least one non-optional result-phase check. Invalid fixtures produce errors, not passes.

Checks support `file_exists`, `file_absent`, `file_matches`, `file_not_matches` and `command`. File checks require exactly one `path` or `glob`. Globs are relative to the workspace root: `*` stays within a directory and `**` crosses directories. Absolute paths, `..` and escaping links are rejected. Result-phase checks exclude `.git/`, `.cratis/` and `node_modules/`; install-phase checks can explicitly target `.cratis/`. Directory links are not traversed by globs.

`phase` is `install` or `result` (default). Install-phase checks run before the harness, and cannot be commands. Patterns use .NET regex per file, multiline mode off by default, with a one-second match timeout. Every selected file must satisfy the regex check. No matching files fails a regex check; `file_matches` reports `zero files matched` distinctly.

Command checks use `run: ["executable", "argument", "..."]`, without an implicit shell, and an expected `exit_code` (default zero). An `optional` check never fails the fixture: failed/error outcomes remain in the report but do not determine fixture success. Missing executables for optional command checks are skipped. Invalid fixture declarations still produce fixture errors. Process deadlines and cancellation include output draining. After exit, draining has a 250-millisecond grace; expiry cancels the reads and attempts process-tree termination. Captured output is capped at 65,536 characters per stream; excess output is drained and discarded.

## Results and retention

Each report separates installed instruction paths/hashes from `observed_load`. Only an explicit tool-read event naming a path establishes `observed`; other paths are `unknown`. Passing checks and installed hashes do not prove that instructions were loaded. Each bound includes its value and enforcement status: `enforced`, `not_enforced` or `not_requested`.

Exit codes are 0 for all passed/skipped or no selected runs, 1 for check, harness or fixture failures/errors, and 2 for usage or installation errors. When there is no passed result, the report warns `no fixtures ran successfully`; skipped results never count as passes.

Transcripts are not retained by default. With `--keep-transcripts`, the report identifies a file under a temporary output directory; the workspace is still removed. Environment values of at least eight characters whose variable names contain KEY, TOKEN or SECRET, and recognizable token formats, are redacted. Redaction is best-effort, not a promise that arbitrary sensitive text can be detected; inspect retained files before sharing them.