Backup and Disaster Recovery

Fabric Harness uses the same migration and recovery path for PostgreSQL and Databricks Lakebase. Migrations are ordered, recorded in fabric_harness_schema_migrations, and serialized with a PostgreSQL advisory lock. Each version runs in its own transaction.

import { postgresPersistence } from '@fabric-harness/node';

const persistence = postgresPersistence({ client: pool });
await persistence.migrate();

Calling migrate() from several replicas is safe. Exactly one connection owns the migration lock, while the other replicas wait and then read the completed ledger. Rollback is available only for versions with an explicit down migration; Fabric Harness refuses a destructive implicit rollback.

Object backup

The logical backup runs in a REPEATABLE READ, read-only transaction. Binary attachments and artifacts are encoded without losing bytes. The complete payload receives a SHA-256 digest before it is sent to object storage.

import {
  backupPostgresPersistence,
  httpBackupObjectStore,
} from '@fabric-harness/node';

const objectStore = httpBackupObjectStore({
  // Return a short-lived S3/R2 presigned URL, Azure Blob SAS URL, or an
  // authenticated internal object-gateway URL for this key.
  urlForKey: (key) => backupUrlFor(key),
  headers: async (method, key) => backupHeadersFor(method, key),
});

const backup = await backupPostgresPersistence({
  client: pool,
  objectStore,
  objectKey: `production/${new Date().toISOString()}.json`,
});

console.log(backup.sha256, backup.tables, backup.bytes);

Lakebase uses the same call after exchanging the Databricks workspace identity for a short-lived database credential. Workspace tokens and database passwords are never written into the backup object.

Restore

Restore verifies the digest and exact Fabric Harness table set before changing the database. It then takes an exclusive restore lock, truncates the managed tables, restores every row in one transaction, and resets identity sequences.

import { restorePostgresPersistence } from '@fabric-harness/node';

await restorePostgresPersistence({
  client: pool,
  objectStore,
  objectKey: 'production/2026-07-09T08:00:00.000Z.json',
});

Stop application writes before restore. Keep the API unready until restore and post-restore verification finish.

Readiness

/ready checks the session and submission stores. A configured persistence bundle also contributes its health result. Add worker and provider dependencies with readiness:

await startDevServer({
  readiness: async () => ({
    temporalWorker: { ok: await temporalWorkerIsPolling() },
    modelServing: { ok: await servingEndpointIsReady() },
  }),
});

Any failed or rejected dependency returns HTTP 503, so Kubernetes, Databricks Apps, and load balancers stop routing work to an incomplete replica.

Recovery targets

A practical production starting point is:

Use provider-native WAL or point-in-time recovery in addition to Fabric Harness logical backups when the required RPO is below the backup interval.

Exercise the application-level path with runPostgresRecoveryDrill() in an isolated database. It creates an object backup, invokes your failure simulation, restores, runs your verification callback, and fails when maxRpoSeconds or maxRtoMs is exceeded.

const result = await runPostgresRecoveryDrill({
  client: drillPool,
  objectStore,
  objectKey: `drills/${Date.now()}.json`,
  maxRpoSeconds: 300,
  maxRtoMs: 30 * 60_000,
  simulateFailure: () => destroyDrillDatabaseState(),
  verify: () => verifyReferenceSessionsAndAttachments(),
});