Operational SLOs
Stable runtime metrics, SLO evaluation, alerts, dashboards, and cross-target conformance.
Fabric Harness defines low-cardinality metrics for the operational signals that matter across Node, Temporal, Cloudflare, and Databricks Apps:
| Metric | Meaning |
|---|---|
fabric_harness_request_latency_ms | End-to-end HTTP or application request latency |
fabric_harness_submission_duration_ms | Admission-to-settlement duration |
fabric_harness_queue_age_ms | Admission-to-worker-start delay |
fabric_harness_errors_total | Failed requests, submissions, events, and recovery attempts |
fabric_harness_approval_wait_ms | Approval request-to-terminal-decision delay |
fabric_harness_recovery_duration_ms | Worker or persistence recovery duration |
fabric_harness_cost_usd_total | Attributed model and serving cost |
Collect and export
import { createOperationalMetricsCollector } from '@fabric-harness/sdk';
const metrics = createOperationalMetricsCollector();
const config = {
onEvent: (event) => metrics.recordEvent(event),
submissionTelemetry: metrics.submissionSink,
};
application.get('/metrics', () => new Response(metrics.openMetrics(), {
headers: { 'content-type': 'application/openmetrics-text; version=1.0.0' },
}));Call recordRequest(durationMs, outcome) at the HTTP or application boundary
and recordRecovery(durationMs, recovered) around recovery drills. The
collector keeps bounded samples and exports count, sum, p50, p95, and p99.
Evaluate a service objective
import { evaluateOperationalSlos } from '@fabric-harness/sdk';
const evaluation = evaluateOperationalSlos(metrics.snapshot(), {
requestLatencyP95Ms: 2_000,
queueAgeP95Ms: 5_000,
errorRateMax: 0.01,
approvalWaitP95Ms: 4 * 60 * 60_000,
recoveryP95Ms: 60_000,
costUsdMax: 1_000,
});The runnable examples/operational-slos workspace includes an importable
Grafana dashboard and Prometheus alert rules for every metric above. Adjust the
reference thresholds to the user journey and recovery tier of each deployment.
Deployment conformance
runDeploymentConformance() is the same black-box client suite for every
deployment target. It checks health, dependency readiness, asynchronous finite
job settlement and event offsets, plus persistent-agent submission settlement
and conversation offsets.
import { assertDeploymentConformance } from '@fabric-harness/client';
await assertDeploymentConformance({
target: 'databricks-app',
baseUrl: process.env.DATABRICKS_APP_URL!,
headers: { authorization: `Bearer ${process.env.APP_TOKEN}` },
job: { name: 'analyst', input: { question: 'SELECT 1' } },
});The live workflow runs this contract against a local Node host, a real Temporal cluster behind the Node HTTP API, local and deployed Cloudflare Workers, and the reference Databricks App. Each run writes a redacted JSON evidence artifact.