Databricks

@fabric-harness/databricks is a first-party integration for building governed agents that consume Databricks services. One databricks() call wires Unity AI Gateway, governed SQL, Unity Catalog, AI Functions, Genie, Lakeflow, AI Search, Feature Serving, governance policy, and cost reporting under a single principal. Everything is layered on top of Unity Catalog; it never re-implements UC permissions.

For the product-level architecture and complete feature map, start with Fabric Harness on Databricks. This page is the deployment and package reference.

For Lakebase, set DATABRICKS_LAKEBASE_ENDPOINT (or ENDPOINT_NAME) to the full projects/.../branches/.../endpoints/... resource name. Run the App, OBO, permission-denial, and Lakebase restart certification suite in the target workspace before rollout. See Databricks' Node/manual credential flow.

For a complete scaffold-to-deploy walkthrough, start with the Databricks App tutorial.

The bundle

import { databricks } from '@fabric-harness/databricks';

const dbx = databricks({
  host: process.env.DATABRICKS_HOST!,
  // UC enforces this principal's grants on every call.
  principal: {
    kind: 'service-principal',
    host: process.env.DATABRICKS_HOST!,
    clientId: process.env.DATABRICKS_CLIENT_ID!,
    clientSecret: process.env.DATABRICKS_CLIENT_SECRET!,
  },
  model: 'system.ai.gpt-oss-20b',
  warehouseId: process.env.DATABRICKS_WAREHOUSE_ID!,
  sqlRead: true,                      // SELECT-only model tool
  aiSearch: { index: 'main.kb.docs_index', textColumn: 'chunk', idColumn: 'id' }, // RAG
  aiFunctions: { allowedEndpoints: ['support-classifier'] },
  genie: { conversations: { agentId: 'agent-123' } }, // Existing Genie Agent
  lakeflow: { runPolicy: { allowedPipelineIds: ['pipeline-123'] } },
  consumption: true,                 // System-Tables cost reporting
  featureServing: { endpoint: 'user-features' },
  governance: { stewardAudience: 'data-steward', onLineage: (r) => console.log('[lineage]', r) },
});

// dbx.modelProvider, dbx.tools, dbx.policy, dbx.retriever, dbx.consumption, dbx.store, dbx.identity

Provider-qualified model refs also resolve through the generic SDK path. For the default Gateway model, use FABRIC_MODEL=databricks/system.ai.gpt-oss-20b. Inside databricks() or defineDatabricksAgent(), use the native service name system.ai.gpt-oss-20b.

Unity AI Gateway and custom endpoints

databricksFoundationModelProvider({ host, token }) uses Unity AI Gateway for system.ai.* model services and retains /serving-endpoints for explicitly selected custom endpoints. Both paths are OpenAI-compatible and inherit streaming, tool calls, and reasoning content. The same rotating token threads through every REST and data call.

Identity & Unity Catalog governance

databricksIdentity() produces a rotating token from a PAT, an OAuth service principal (cached + refreshed), or on-behalf-of a specific end user. UC enforces that principal's table/row/column grants natively — the agent physically cannot read what it lacks SELECT on. On top of UC, Fabric adds:

• Lineage/audit — withGovernance() stamps every tool call (principal, service, catalog/schema) to an onLineage sink (secrets redacted).
• Approval routing — databricksGovernancePolicy() routes sensitive (write/execute) tools to a steward audience via CapabilityPolicy.approvalRules.
• Egress allowlist — outbound network pinned to the workspace host.

With channel actor propagation, an agent can act on behalf of the human who triggered it, so UC enforces that user's grants — per-user data boundaries with no per-user policy code.

Tools

Feature Serving inference uses Databricks' data-plane route /serving-endpoints/{name}/invocations. Harness corrects the modular JavaScript SDK 0.21 client's erroneous /api/serving-endpoints/... serialization at the shared authenticated transport boundary while retaining the generated request and response codecs. The endpoint principal still requires CAN_QUERY.

Job, notebook, and MLflow tools (databricksRunJobTool, databricksNotebookTool, databricksMlflowLogMetricTool) and the SQL-warehouse SandboxEnv (databricksSqlSandbox, @fabric-harness/databricks/sql-sandbox) remain available as building blocks. Model-callable job, notebook, SQL, AI Functions, Lakeflow mutation, and MLflow factories require explicit resource policies. Deliberate allowAny...: true arms remain available for operator-reviewed advanced code and are intentionally greppable. See Bound what a model can run.

State — Lakebase

Lakebase is Postgres-compatible, and Fabric's Postgres session, submission, and conversation-stream stores are structurally reusable there. lakebaseClient() exchanges workspace OAuth for a database credential, caches it until the early-refresh window, and single-flights refreshes. Configure the workspace host and full endpoint resource name; never pass the workspace token as a direct database password. databricksApp().serverOptions() supplies all three stores to the shared Node server.

import { databricksPersistence, databricksSdk } from '@fabric-harness/databricks';
import { startDevServer } from '@fabric-harness/node';

const principal = {
  kind: 'service-principal',
  host: process.env.DATABRICKS_HOST!,
  clientId: process.env.DATABRICKS_CLIENT_ID!,
  clientSecret: process.env.DATABRICKS_CLIENT_SECRET!,
} as const;
const sdk = databricksSdk({ host: process.env.DATABRICKS_HOST!, principal });

const persistence = databricksPersistence({
  lakebase: {
    endpoint: process.env.DATABRICKS_LAKEBASE_ENDPOINT!,
    host: process.env.DATABRICKS_LAKEBASE_HOST!,
    database: process.env.DATABRICKS_LAKEBASE_DATABASE!,
    user: process.env.DATABRICKS_LAKEBASE_USER!,
    credentialClient: sdk.postgres,
  },
});

await persistence.migrate();
const { store, submissionStore, conversationStreamStore } =
  await persistence.connectAll();

await startDevServer({
  sessionStore: store,
  submissionStore,
  conversationStreamStore,
});

For local Postgres tests, use the explicit password option instead of credentialClient and endpoint. Production Lakebase should always use credential exchange.

SDK requests and safe retries

Stable API requests use the generated Databricks clients. Fabric retries only operations it can classify as idempotent (for example Feature Serving reads and MLflow metric/parameter upserts), while Jobs runs carry a caller-supplied Databricks idempotency token. Ambiguous creates are never retried.

import { databricksJobs, databricksSdk } from '@fabric-harness/databricks';

const sdk = databricksSdk({
  host: process.env.DATABRICKS_HOST!,
  principal,
});

const jobs = databricksJobs(sdk.jobs, { runPolicy: { allowedJobIds: [42] } });
await jobs.runJob({
  jobId: 42,
  idempotencyToken: submission.id,
});

Fabric's private raw protocol transport is reserved for documented APIs absent from the generated SDK and retains explicit safe/never retry classification at those narrow call sites. Callers use typed bundle capabilities instead of constructing the transport. Genie ACLs use the official @databricks/sdk-accessmanagement client and are not part of this exception.

Deploy targets

Build a deployable artifact with fabric-harness build --target <name>:

Release orchestrators that already have an immutable databricks-app artifact can use the same Harness deployment implementation programmatically. Verify the digest before calling it:

import {
  deployDatabricksAppArtifact,
  destroyDatabricksAppArtifact,
  digestDatabricksArtifact,
  validateDatabricksAppArtifact,
} from '@fabric-harness/databricks';

const digest = await digestDatabricksArtifact(artifactDir);
await verifyExpectedReleaseDigest(digest);
await validateDatabricksAppArtifact({ artifactDir, target: 'prod', profile: 'production' });
await deployDatabricksAppArtifact({
  artifactDir,
  target: 'prod',
  profile: 'production',
  appName: 'analytics-agent',
  sourceCodePath: '/Workspace/Shared/analytics-agent/app',
  mode: 'SNAPSHOT',
});

// PR-close cleanup uses the same immutable artifact and destroys only its bundle target.
await destroyDatabricksAppArtifact({ artifactDir, target: 'preview-42', profile: 'production' });

The function synchronizes the Declarative Automation Bundle, creates the App snapshot, and waits for the Apps CLI operation to succeed. It does not rebuild the directory, which keeps the verified artifact identical to the deployed artifact. Commands use an argument array with shell execution disabled. Successful results report bundleSynchronized: true, appSnapshotActivated: true, and the final non-empty output line from each operation.

appName, sourceCodePath, and mode: 'SNAPSHOT' make activation explicit when an orchestrator already knows the Databricks App resource and the Bundle-synchronized workspace source. Omit them to use Databricks CLI project mode in the artifact directory. The source path must be the workspace location populated by the preceding Bundle deployment, not an unverified local directory.

Validation or deployment fails when databricks.yml is missing, CLI authentication is unavailable, the Bundle target is invalid, or either CLI command exits non-zero. A Bundle synchronization failure prevents App snapshot activation. The API does not delete a successful deployment automatically; roll back by deploying a previously approved artifact. The runnable examples/with-databricks-simple/release-artifact.ts path is digest-only by default and requires an explicit environment opt-in plus an expected digest before it mutates a workspace.

• databricks-app — intended to run the agent in-workspace as a Databricks App (the app's service principal is the acting UC identity). Emits app.yaml (bridges DATABRICKS_APP_PORT) + deploy docs. fh deploy synchronizes the Declarative Automation Bundle, creates a new App snapshot, and waits for the snapshot to reach SUCCEEDED before returning. App artifacts omit bundled definition source maps and minify each executable definition because Databricks rejects individual snapshot files larger than 10 MiB. The build checks every generated job, persistent-agent, and server .mjs file and fails locally with the definition path, actual size, and limit before a Bundle or App deployment can begin.

When BUNDLE_VAR_lakebase_endpoint and BUNDLE_VAR_lakebase_database_resource are set at build time, the bundle attaches Lakebase Autoscaling as a managed postgres App resource. The database variable is the database resource name, not the PostgreSQL database name:

export BUNDLE_VAR_lakebase_endpoint='projects/my-project/branches/production/endpoints/primary'
export BUNDLE_VAR_lakebase_database_resource='projects/my-project/branches/production/databases/app-db'
fh deploy --target databricks-app

Find both values with databricks postgres list-endpoints <branch> and databricks postgres list-databases <branch>. The database API response separately reports the PostgreSQL name under status.postgres_database; Databricks injects that value as PGDATABASE. It also injects the other PG* connection variables on every start, while app.yaml resolves the endpoint path with valueFrom. No database password or workspace token is stored in the artifact.

• databricks-serving — proxy-only: packages an MLflow ResponsesAgent proxy to a persistent agent deployed elsewhere, registers it to Unity Catalog, and creates a serving endpoint that Agent Bricks and the Playground consume as a model/tool. predict_stream preserves Responses text-delta and final output-item events with stable item IDs; predict returns a ResponsesAgentResponse. It does not execute the TypeScript agent inside Model Serving. Calls from Model Serving to a Databricks App use a least-privilege OAuth M2M service principal; a static Harness bearer token is only appropriate when the upstream is a non-Databricks service that accepts that token.

Install @fabric-harness/databricks in projects that use fh deploy --target databricks-serving. The CLI intentionally lazy-loads this optional integration, keeping ordinary CLI commands free of the native SDK's Node and install-size requirements. Databricks init templates and recipes declare the dependency for you. Generated projects pin CLI, SDK, Node runtime, and Databricks integration ranges independently to versions that exist in the registry; a CLI patch never invents matching patch versions for independently released packages. Executable, interactive, and managed-recipe scaffold tests compare every generated range with the workspace package manifests to guard against release drift. Check release and documentation status for the current published versions rather than copying a historical template range from a certification record.

In a source checkout, the Node build pipeline resolves the Databricks integration directly from its workspace source so clean-clone build tests do not depend on pre-existing dist output. Published @fabric-harness/node resolves the separately installed @fabric-harness/databricks package in the normal way. The release gate tests both paths and rejects a Databricks App server bundle that leaves an unresolved package import.

Node-derived targets use the same v2 server as local development, so finite jobs keep /jobs/:name and persistent agents keep durable /agents/:name/:id admission after deployment. Cloudflare uses its own Durable Object-backed persistent runtime; Node-derived targets use the shared submission server.

Cost reconciliation

databricksTenantCostLimit() enforces a perScope budget against real Databricks spend from System Tables (estimates still guard perCall/perSession). System Tables are delayed accounting, so this is a reconciliation guard rather than a real-time kill switch. See Cost attribution.

See also

• Examples: with-databricks (analytics copilot), with-databricks-rag (support agent), with-databricks-dataeng (pipelines), with-databricks-cost-attribution.
• Model providers · Channels · Cost attribution The public deployDatabricksAppArtifact() API also accepts appName, sourceCodePath, mode: "SNAPSHOT", and startBeforeDeploy when a governed release orchestrator must start and activate an explicit App from the immutable source synchronized by its Asset Bundle. Keep this call at the trusted deployment edge; human-facing tools should request a governed Platform action instead of activating production.