Jetty

Jetty can grade output from a Fabric Harness job and store the grading task as a trajectory. Use it when you need a persistent, comparable record of quality across prompts, models, or releases. Use Fabric evals for fast local and CI assertions.

Install

fh add jetty

The managed recipe pins the published @jetty/sdk 0.2.x contract and creates a typed gradeOutputWithJetty() helper and verification test.

Create and deploy a Jetty grading runbook before invoking it from Fabric Harness. The runbook must produce the grade.json expected by gradeWithJetty().

Grade a job response

import { defineJob, schema } from '@fabric-harness/sdk';
import { gradeWithJetty, JettyClient } from '@jetty/sdk';

interface TriageGrade {
  total: number;
  pass: boolean;
}

const jetty = new JettyClient();

export default defineJob({
  name: 'evaluate-triage',
  input: schema.object({ ticket: schema.string() }),
  triggers: { manual: true },
  async run({ input, prompt, run }) {
    const response = await prompt(`Triage this support ticket:\n\n${input.ticket}`);
    const { grade, trajectoryId } = await gradeWithJetty<TriageGrade>(
      jetty,
      process.env.JETTY_COLLECTION!,
      process.env.JETTY_GRADE_TASK!,
      {
        files: [{
          filename: 'case.json',
          data: JSON.stringify({ ticket: input.ticket, response }),
        }],
        useTrialKeys: process.env.JETTY_USE_TRIAL_KEYS === 'true',
        labels: (result) => ({
          'eval.grade': String(result.total),
          'eval.pass': String(result.pass),
          'fabric.run_id': run?.runId ?? 'local',
        }),
      },
    );
    return { response, grade, trajectoryId };
  },
});

Run it on the Node target:

fh run evaluate-triage --ticket "Customers cannot reset passwords"

@jetty/sdk requires Node.js. Keep the grader separate from the job being evaluated so a prompt change cannot silently change its own rubric.

Environment

The Fabric job still needs its model-provider credentials. Jetty credentials configure only the grading operation.

Sensitive data

Jetty trajectories may persist uploaded files, inputs, and outputs. Redact credentials, personal data, and regulated content before grading. Put grader credentials in Jetty secret parameters rather than initialization parameters or uploaded files, and review retention and access controls before grading production content.

For a credential-free automated fixture using the same correlation and redaction contract, see examples/with-observability.