CLI
fh test
Discover and run eval suites.
fabric-harness test [path] [options]fh test discovers *.eval.ts files, imports them as eval suites, runs each case through the suite's runner, scores the outputs, and prints a summary. It exits 0 when every suite passes and 1 when any suite fails.
Options
| Flag | Description |
|---|---|
--pattern <glob> | Glob pattern for eval files (default **/*.eval.ts). |
--suite <name> | Run only a specific suite by name. |
--scorer <name> | Filter reporting to a specific scorer. |
--json | Output results as JSON. |
--threshold <number> | Override the pass threshold for all suites. |
Example eval file
import { defineEvalSuite, exactMatchScorer } from '@fabric-harness/evals';
export default defineEvalSuite({
name: 'hello-world',
cases: [
{ id: 'greeting', input: 'hello', expected: 'HELLO' },
],
runner: ({ case: evalCase }) => String(evalCase.input).toUpperCase(),
scorers: [exactMatchScorer()],
});Examples
Run all eval suites in the current directory:
fh testRun eval suites in a specific directory:
fh test examples/hello-worldRun only a specific suite:
fh test --suite hello-worldOutput results as JSON:
fh test --jsonOverride the pass threshold:
fh test --threshold 0.9Exit codes
| Code | Meaning |
|---|---|
0 | All suites passed. |
1 | One or more suites failed, or an error occurred. |