Databricks compute patterns

Databricks offers several execution surfaces. Fabric Harness keeps them explicit so policy, identity, idempotency, and output handling match the workload.

flowchart TD
  Q{What must execute?}
  Q -->|Governed SQL| SQL[SQL Warehouse]
  Q -->|Existing workflow| JOB[Lakeflow Job]
  Q -->|One-off notebook| NB[Notebook task]
  Q -->|Fabric HTTP runtime| APP[Databricks App]
  Q -->|General shell or untrusted code| SB[External isolated sandbox]

  SQL --> UC[Unity Catalog permissions and lineage]
  JOB --> VOL[Logs and output envelope in UC Volumes]
  NB --> VOL
  APP --> LB[Lakebase durable state]
  SB --> NET[Container, cluster, or provider egress boundary]

  classDef decision fill:#fff4d6,stroke:#d97706,color:#451a03
  classDef compute fill:#e8f0fe,stroke:#2563eb,color:#172554
  classDef governed fill:#dcfce7,stroke:#16a34a,color:#052e16
  class Q decision
  class SQL,JOB,NB,APP,SB compute
  class UC,VOL,LB,NET governed

Run and monitor a Job

import { databricksJobs, databricksSdk } from '@fabric-harness/databricks';

const sdk = databricksSdk({
  host: process.env.DATABRICKS_HOST!,
  principal: { kind: 'pat', token: process.env.DATABRICKS_TOKEN! },
});
const jobId = Number(process.env.DATABRICKS_JOB_ID);
const jobs = databricksJobs(sdk.jobs, {
  runPolicy: { allowedJobIds: [jobId] },
  notebookPolicy: { allowedNotebookPathPrefixes: ['/Workspace/Shared/fabric'] },
});

const receipt = await jobs.runJob({
  jobId,
  idempotencyToken: submission.id,
  notebookParams: { customer: tenant.id },
});
const state = await jobs.wait(receipt.runId, {
  timeoutMs: 15 * 60_000,
  signal: abortController.signal,
});

if (state.resultState !== 'SUCCESS') {
  throw new Error(`Job ${receipt.runId} ended in ${state.resultState}`);
}

The caller-supplied idempotency token survives HTTP retries and upstream message redelivery. Calling cancel(runId) is safe to retry. wait() returns typed lifecycle/result state and throws DatabricksRunTimeoutError rather than returning an ambiguous running result after its deadline.

Bound what a model can run

Every model-callable execute/write factory takes a required resource policy. There is no unbounded default: omitting it is a compile error, runtime checks run before the Databricks API, and opting out uses a deliberate, greppable allowAny...: true arm.

import { databricksNotebookTool, databricksRunJobTool } from '@fabric-harness/databricks';

const tools = [
  databricksRunJobTool(sdk.jobs, { allowedJobIds: [Number(process.env.DATABRICKS_JOB_ID)] }),
  databricksNotebookTool(sdk.jobs, {
    allowedNotebookPathPrefixes: ['/Workspace/Shared/fabric'],
  }),
];

const sql = databricksSqlTool(
  sdk.statements,
  { allowedStatements: ['CALL main.ops.refresh_daily()'] },
  { warehouseId },
);
const ai = databricksAiQueryTool(
  sdk.statements,
  { allowedEndpoints: ['support-classifier'] },
  { warehouseId },
);
const pipeline = databricksPipelineStartTool(sdk.pipelines, {
  allowedPipelineIds: [pipelineId],
});
const metric = databricksMlflowLogMetricTool(sdk.experiments, {
  allowedRunIds: [runId],
});

Pinned input schemas guide the model, but runtime enforcement is the security boundary. An out-of-policy SQL statement, endpoint, pipeline id, or MLflow run id never reaches the generated client.

The Jobs and notebook model tools use the Harness tool-call id as their default Databricks idempotency token. Harness guarantees that id is unique per logical call even when a model provider omits an id, while preserving it across the call's approval, lineage, execution, and result records. Typed databricksJobs() callers should continue to supply a stable delivery or submission id when they need retry coalescing across process boundaries.

Enforcement lives in databricksJobs(), before run-now or runs/submit is called: an out-of-policy target throws DatabricksRunNotAllowedError and never reaches the workspace. Any notebookPath containing a .. segment is rejected while a policy is configured. The tool's inputSchema is also pinned (const for one id, enum for several) — that is a model-facing hint, not the boundary; the runtime check is.

The boundary is job ids, not job names. The run client is deliberately narrow (runNow, submitRun, getRun, cancelRun, getRunOutput) and cannot resolve names, and resolving a name at call time would be a time-of-check/time-of-use hole. Resolve names to ids yourself at configuration time; a helper built on the authoring client may return number[] in a future release.

Note that computePolicy does not bound this path. It applies to Jobs authoring and one-off compute specs, not to triggering an existing job.

Submit a notebook

const receipt = await jobs.submitNotebook({
  notebookPath: '/Workspace/Shared/fabric/daily-report',
  existingClusterId: process.env.DATABRICKS_CLUSTER_ID!,
  idempotencyToken: submission.id,
  baseParameters: { report_date: '2026-07-09' },
});

Submission is bounded the same way as runJob: the jobs client above already carries a notebookPolicy, so an out-of-policy notebookPath throws DatabricksRunNotAllowedError before runs/submit is called.

Notebook submission is asynchronous Jobs compute. It does not turn the SQL sandbox into a shell and does not execute TypeScript inside Model Serving.

Persist logs and outputs in a UC Volume

import { UcVolumesAttachmentStore } from '@fabric-harness/databricks';

const store = new UcVolumesAttachmentStore({
  client: sdk.files,
  catalog: 'main',
  schema: 'agents',
  volume: 'fabric_attachments',
  rootPrefix: 'job-output',
});

const ref = await jobs.exportOutput(receipt.runId, store, `submission:${submission.id}`);

For multi-task Jobs, getOutputs() retrieves each task run output. exportOutput() stores one content-addressed JSON envelope with notebook result, driver logs, truncation state, errors, and raw metadata. Unity Catalog controls access to the Volume.

The runnable compute example uses a deterministic mock by default and switches to a real workspace when credentials are present.