Eval Library

Install @fabric-harness/evals and import all APIs from the package root.

Suite API

import { defineEvalSuite, runEvalSuite } from '@fabric-harness/evals';

• defineEvalSuite(suite) preserves generic inference and returns the suite unchanged.
• runEvalSuite(suite) runs cases sequentially and returns EvalSuiteResult.
• EvalSuite contains name, cases, runner, scorers, optional passThreshold, and optional metadata.
• EvalCase contains id, input, optional expected, tags, and metadata.
• EvalScore contains name, a 0..1 score, optional passed, reason, and metadata.

A case passes when no scorer explicitly returns passed: false and its average score meets passThreshold (default 0.8). A runner or scorer error records a failed case instead of aborting the remaining suite.

Built-in scorers

exactMatchScorer(name?)
containsTextScorer(expectedText?, name?)
regexMatchScorer(pattern, { name?, flags? })
jsonShapeMatchScorer(shape, { name? })
llmAsJudgeScorer({ judge, model, rubric, name?, passThreshold? })

containsTextScorer() uses the case's string expected value when no fixed text is provided. jsonShapeMatchScorer() checks required keys and primitive types while allowing extra keys. llmAsJudgeScorer() expects a model provider with generate() and treats malformed judge output as a zero score.

Custom scorer

import type { EvalScorer } from '@fabric-harness/evals';

const concise: EvalScorer<unknown, string> = ({ output }) => ({
  name: 'concise',
  score: output.length <= 200 ? 1 : 0,
  passed: output.length <= 200,
  reason: `${output.length} characters`,
});

See Evaluations for job execution and CLI examples.