Databricks compute patterns

Choose SQL Warehouses, Lakeflow Jobs and notebooks, Databricks Apps, or an isolated sandbox without conflating their execution models.

Databricks offers several execution surfaces. Fabric Harness keeps them explicit so policy, identity, idempotency, and output handling match the workload.

Rendering diagram...

Workload	Fabric API	Durable result	Policy effect
SQL statement	`databricksSqlTool()` or `databricksSqlSandbox()`	Statement id/result	`execute`; gate mutations
Existing Job	`databricksJobs().runJob()`	Typed run receipt and state	`execute`; use delivery id as idempotency token
Notebook	`databricksJobs().submitNotebook()`	Typed run receipt and task output	`execute`; gate data mutations
Agent hosting	`fh build --target databricks-app`	Lakebase session/submission stream	HTTP runtime, not a shell
General code	Docker, Kubernetes, E2B, Daytona, Modal, or another sandbox	Provider-specific snapshot/ref	Enforce filesystem, process, and network boundaries

Run and monitor a Job

import { createDatabricksClient, databricksJobs } from '@fabric-harness/databricks';

const client = createDatabricksClient({
  host: process.env.DATABRICKS_HOST!,
  token: process.env.DATABRICKS_TOKEN!,
});
const jobs = databricksJobs(client);

const receipt = await jobs.runJob({
  jobId: Number(process.env.DATABRICKS_JOB_ID),
  idempotencyToken: submission.id,
  notebookParams: { customer: tenant.id },
});
const state = await jobs.wait(receipt.runId, {
  timeoutMs: 15 * 60_000,
  signal: abortController.signal,
});

if (state.resultState !== 'SUCCESS') {
  throw new Error(`Job ${receipt.runId} ended in ${state.resultState}`);
}

The caller-supplied idempotency token survives HTTP retries and upstream message redelivery. Calling cancel(runId) is safe to retry. wait() returns typed lifecycle/result state and throws DatabricksRunTimeoutError rather than returning an ambiguous running result after its deadline.

Submit a notebook

const receipt = await jobs.submitNotebook({
  notebookPath: '/Workspace/Shared/fabric/daily-report',
  existingClusterId: process.env.DATABRICKS_CLUSTER_ID!,
  idempotencyToken: submission.id,
  baseParameters: { report_date: '2026-07-09' },
});

Notebook submission is asynchronous Jobs compute. It does not turn the SQL sandbox into a shell and does not execute TypeScript inside Model Serving.

Persist logs and outputs in a UC Volume

import { UcVolumesAttachmentStore } from '@fabric-harness/databricks';

const store = new UcVolumesAttachmentStore({
  host: process.env.DATABRICKS_HOST!,
  token: process.env.DATABRICKS_TOKEN!,
  catalog: 'main',
  schema: 'agents',
  volume: 'fabric_attachments',
  rootPrefix: 'job-output',
});

const ref = await jobs.exportOutput(receipt.runId, store, `submission:${submission.id}`);

For multi-task Jobs, getOutputs() retrieves each task run output. exportOutput() stores one content-addressed JSON envelope with notebook result, driver logs, truncation state, errors, and raw metadata. Unity Catalog controls access to the Volume.

The runnable compute example uses a deterministic mock by default and switches to a real workspace when credentials are present.

Databricks compute patterns

Run and monitor a Job

Submit a notebook

Persist logs and outputs in a UC Volume

On this page