Testing Locally

The framework is designed so that "test the agent" doesn't mean "spin up a full LLM." Use the mock provider for fast, deterministic checks and reach for live models only when behavior depends on the model.

Mock model

fh run ask --question "hi" --mock

--mock injects the deterministic mock provider and still exercises input/output validation. Combine it with snapshot testing or schema-only assertions in your test suite.

Vitest example

import { describe, expect, it } from 'vitest';
import { runAgent } from '@fabric-harness/node';

describe('ask agent', () => {
  it('returns a string', async () => {
    const { result } = await runAgent({
      agent: 'ask',
      payload: { question: 'hello' },
      mock: true,
    });
    expect(typeof result).toBe('string');
  });
});

Doctor

fh doctor --tools           # binary checks
fh doctor --live --model openai/gpt-5.5  # one real provider round-trip

Dev server

fh dev --mock starts the same routes the deployed Node target uses without provider credentials. Invoke a finite job at /jobs/:name:

fh dev --mock --port 4000
curl -X POST -H 'Content-Type: application/json' \
  -d '{"question":"What is Temporal?"}' \
  http://localhost:4000/jobs/ask

Live integration tests

Live tests are opt-in and skipped by default. Use them when validating real provider credentials and hosted resources:

pnpm --filter @fabric-harness/connectors test   # Daytona / E2B / Modal live suites skip unless enabled
pnpm --filter @fabric-harness/azure test        # Azure OpenAI / Foundry / ARM live suites skip unless enabled
pnpm --filter @fabric-harness/databricks test   # Databricks live suites skip unless enabled

See Live tests for the full environment variable matrix.

Harness child-process tests start from a sanitized environment. Provider keys, cloud credentials, database URLs, and agent markers are removed unless a test explicitly supplies an override. This keeps offline suites repeatable on developer machines that happen to be logged into cloud services. HTTP integration tests bind port 0 and use the operating system's assigned port, avoiding parallel-suite collisions.

Maintainers should also run pnpm check:unused. The Knip configuration treats package entry points as public API, checks internal exports and dependency declarations, excludes generated .fabricharness/build output, and is included in pnpm lint.

Maintainer workspace concurrency

The root pnpm test intentionally runs package suites with --workspace-concurrency=1. A measured concurrency-2 run on 2026-07-30 caused the Node persistent-prompt admission test and the Databricks capability-registry import test to exceed their bounded timeouts under shared CPU pressure. Keep the workspace layer serial until those integration suites have isolated worker/resource budgets; Vitest still parallelizes safely within each package. CI avoids the larger duplicate cost by running typecheck, documentation, example, parity, release, coverage, and lint gates only in the Node 22 matrix lane.

Recipes from examples/

The repo's examples/ directory is the canonical reference for how to test each capability:

• examples/hello-world — basic metadata agents and real model invocation.
• examples/with-tools — built-in tools.
• examples/with-skill — skill loading and typed results.
• examples/with-task — durable task lifecycle and artifacts.
• examples/with-approval — approval-gated commands.
• examples/with-docker — Docker sandbox basics.
• examples/with-temporal — Temporal worker integration.
• examples/with-config — central config including SQLite session storage.
• examples/with-postgres-store — Postgres session/artifact storage.
• examples/data-analyst — Docker-backed CSV analysis with artifacts.
• examples/issue-triage-ci — controlled CI pilot for read-only GitHub issue triage.

What to assert in tests

For metadata agents, the most useful assertions are:

1. Schema shape. Output validates against the declared output schema.
2. Tool/command scope. No unexpected commands ran.
3. Artifacts. Expected artifacts were published with the right content type.
4. Metrics. Token / call counts stay within bounds for a given task.
5. Idempotence. Re-running the same prompt produces compatible output (when using the mock model or fixed seed).