Evaluations

Fabric eval suites are TypeScript modules named *.eval.ts. Each suite supplies cases, a runner, one or more scorers, and an optional pass threshold. fh test discovers and runs them locally or in CI.

Install

pnpm add -D @fabric-harness/evals @fabric-harness/node

Evaluate a job

evals/hello.eval.ts

import { containsTextScorer, defineEvalSuite } from '@fabric-harness/evals';
import { runAgent } from '@fabric-harness/node';

export default defineEvalSuite({
  name: 'hello-quality',
  cases: [
    { id: 'named-user', input: { name: 'Ada' }, expected: 'Ada' },
    { id: 'default-user', input: { name: 'World' }, expected: 'World' },
  ],
  runner: async ({ case: evalCase }) => {
    const run = await runAgent({
      agent: 'hello',
      payload: evalCase.input,
      mock: true,
    });
    return run.result;
  },
  scorers: [containsTextScorer()],
  passThreshold: 1,
});

Run every suite:

fh test

Run a subset or produce machine-readable output:

fh test evals --suite hello-quality
fh test --threshold 0.9 --json

The command exits nonzero when any suite fails, so it can be used directly as a CI quality gate.

Choose scorers

Scorer	Use it for
`exactMatchScorer()`	Deterministic structured or scalar output.
`containsTextScorer()`	Required text in an answer.
`regexMatchScorer()`	IDs, formats, and constrained strings.
`jsonShapeMatchScorer()`	Required JSON fields and primitive types.
`llmAsJudgeScorer()`	Semantic criteria that deterministic assertions cannot express.

Prefer deterministic scorers first. For an LLM judge, use a separate provider/model, write a narrow rubric, and set passThreshold explicitly. Never pass provider secrets through eval case input.

For long-lived comparisons and stored grading trajectories, see Jetty. For every exported type and scorer, see the eval library reference.

Evaluations

Install

Evaluate a job

Choose scorers

On this page