Vitest Evals

Fabric ships @fabric-harness/evals for case/scorer execution and uses Vitest for repository-level regression tests. Keep behavioral eval suites in *.eval.ts; use ordinary Vitest tests for deterministic runtime contracts.

fh add vitest-evals

The managed recipe creates a credential-free defineEvalSuite() fixture and Vitest contract. Add @fabric-harness/node when the suite invokes a project agent rather than a pure runner function.

import { describe, expect, it } from 'vitest';
import { runAgent } from '@fabric-harness/node';

describe('hello job', () => {
  it('runs without provider credentials', async () => {
    const run = await runAgent({
      agent: 'hello',
      payload: { name: 'Ada' },
      mock: true,
    });
    expect(run.result).toContain('Ada');
  });
});

For multi-case scoring, use defineEvalSuite() and run fh test. CI can run both gates:

pnpm vitest run
fh test --json > eval-results.json

Eval artifacts can include prompts, outputs, tool results, and model judgments. Review them before uploading to CI or a hosted evaluation service. See Evaluations and the eval API.