Databricks resource management

Fabric Harness lets an agent (or a user driving one) manage the Databricks resources it operates — create a Job, define a pipeline, build an AI Search index, size a serving endpoint, create a Genie Agent, grant access — through the same governed, typed seams the consumption tools already use.

Resource management is opt-in per surface and fails closed: a consumption-only databricks() bundle keeps its existing tools, and any model-exposed write surface refuses to initialize unless approval routing is configured. Stable services run through Databricks' official modular TypeScript SDK. Fabric owns the agent layer: approval binding, policy, lineage, durable retries, managed-resource fingerprints, and cleanup. Protocols absent from the SDK stay in a small, explicit raw protocol adapter. This does not replace Databricks Asset Bundles, Terraform, or checked-in pipeline source: Harness never generates that IaC. But the lifecycle of a checked-in bundle — validate, deploy, run, destroy — is itself a governed agent surface; see Checked-in Asset Bundles.

The docs use resource management as the developer-facing term. “Authoring” remains in a few flag and certification names where it distinguishes create/update/delete from consumption; it does not mean a second infrastructure-as-code system.

Quickstart: author your first Job

Enable one surface, bound its compute, and route writes to an approval audience:

import { init } from '@fabric-harness/sdk';
import { databricks } from '@fabric-harness/databricks';

const dbx = databricks({
  host: process.env.DATABRICKS_HOST!,
  principal: {
    kind: 'service-principal',
    host: process.env.DATABRICKS_HOST!,
    clientId: process.env.DATABRICKS_CLIENT_ID!,
    clientSecret: process.env.DATABRICKS_CLIENT_SECRET!,
  },
  jobAuthoring: true,
  computePolicy: {
    maxConcurrentRuns: 1,
    maxRunTimeoutSeconds: 1800,
    requiredTags: { 'cost-center': 'agents' },
    // schedules stay disabled unless you opt in with `schedules: true`
  },
  governance: {
    stewardAudience: 'data-platform',
    catalogs: ['main'],
    onLineage: (record) => audit(record),
  },
});

const fabric = await init({ modelProvider: dbx.modelProvider, tools: dbx.tools, policy: dbx.policy });

What happens when the model calls databricks_create_job:

1. The call routes to the data-platform approval audience. Execution does not start without a grant bound to this exact call (see approvals).
2. The job spec is validated against computePolicy before any API call — schedules off, timeout and concurrency capped, required tags present.
3. The created job is stamped with the fabric-harness:managed tag and a canonical spec fingerprint, so later updates and deletes can prove ownership.
4. The operation is recorded in lineage with the executing principal and every approver.

For local development only, allowUnapprovedAuthoring: true bypasses the fail-closed approval requirement. It is the single escape hatch and should never be set in a deployed app.

Runnable end-to-end versions of this flow live in the authoring examples.

How approval and governance work

An approval grant binds the logical tool-call id, a canonical digest of the input, the executing principal, the approver identities, and an expiry. A durable retry may replay the same operation, but a grant can never be reused for different input or a different identity. Grant TTL starts when the approval is decided, not when it was requested.

The governance block controls routing and audit:

Read tools remain approval-free when an admin surface is enabled. The typed clients (DatabricksJobsAuthoring, DatabricksGenieAdmin, …) can always be used without model approval policy — "the SDK may author" is deliberately separate from "a model may autonomously author". Native Databricks/Unity Catalog authorization is authoritative in either path.

Narrow by service only when that service boundary is explicit in the tool inventory. For example, use separate deploy_dev (service: "dev") and deploy_prod (service: "prod") tools, then set approvalServices: ["prod"]. The governance wrapper and approval policy use the same selection, so the dev tool runs without a grant while the production tool fails closed until approved.

Genie conversations, lifecycle management, and Agent Mode are configured together under genie. Supplying genie.manage is the explicit write opt-in; no separate authoring boolean is required.

Request-scoped user identity

Use a principal returned by forwarded-token validation/current-user inspection, never a label supplied by the request:

const inspection = await inspectDatabricksAppUserAuthorization({ headers, host });
const user = dbx.forPrincipal({ tokenProvider: forwardedTokenProvider, principal: inspection.principal });

The scoped bundle has isolated generated SDK/model/embedding/credential clients and does not mutate the parent. Persistence wiring and policy/lineage sinks remain app-scoped. Approval records and lineage distinguish the executing principal from all approvers.

Native SDK boundary

databricks() exposes the official principal-bound clients at bundle.sdk. Application code can also create them directly:

import { databricksSdk, databricksJobs } from '@fabric-harness/databricks';

const sdk = databricksSdk({ host, principal });
const jobs = databricksJobs(sdk.jobs);

Generated clients own stable Jobs, Statement Execution, Lakeflow, Vector Search, Model Serving, Files, Secrets, Genie, Access Management, Unity Catalog, MLflow Experiments, and Lakebase request/response models. Fabric's private raw protocol transport has an exhaustive method-and-path allowlist for endpoints that the modular SDK does not yet expose. Anything else is denied before credentials are resolved or network I/O starts. Source-boundary and runtime-denial tests enforce that quarantine. Every generated module is exact-pinned to SDK 0.21.0, and CI verifies the package pins, runtime constant, compatibility page, README, generated serialization, and packaged import together.

Genie Agent permissions use the generated @databricks/sdk-accessmanagement client; they are not a raw-protocol exception. The only stable-API fallbacks are the exact Genie query-result and AI Search floating-score decoder mismatches documented on the compatibility page.

Resource-management surfaces

Each surface is an independent opt-in. Enable only what the agent needs:

A bundle with every surface enabled looks like this — treat it as a reference for field names, not a starting point:

const dbx = databricks({
  host,
  principal,
  jobAuthoring: true,
  oneOffCompute: true,
  computePolicy: {
    allowedPolicyIds: ['policy-id'],
    allowedExistingClusterIds: ['explicit-shared-cluster-id'],
    maxWorkers: 2,
    maxConcurrentRuns: 1,
    maxRunTimeoutSeconds: 1800,
    allowedNodeTypes: ['Standard_D4ds_v5'],
    allowedRuntimeVersions: ['15.4.x-scala2.12'],
    serverlessPerformance: 'STANDARD',
    schedules: false,
    requiredTags: { 'cost-center': 'agents' },
  },
  lakeflowAuthoring: true,
  lakeflowEvents: true,
  assetBundles: { bundleDir: '.', target: 'dev' },
  genie: {
    manage: {
      resourceStore: managedResourceStore,
      delete: 'managed-only',
    },
  },
  aiSearchAdmin: {
    allowedOperations: ['createIndex', 'syncIndex', 'describeIndex'],
    allowedIndexes: ['main.kb.docs_idx'],
    allowedEndpoints: ['kb-search'],
    allowedEmbeddingModelEndpoints: ['databricks-bge-large-en'],
    allowedSourceTables: ['main.kb.docs'],
  },
  servingAdmin: true,
  ucAdmin: true,
  ucDestructiveAdmin: true,
  workspaceWrite: true,
  secretsWrite: true,
  secretProvider,
  warehouseId,
  governance: {
    stewardAudience: 'data-platform',
    approvalTtlSeconds: 900,
    catalogs: ['main'],
    onLineage: audit,
  },
});

Compute policy reference

computePolicy bounds every Jobs-authoring and one-off-run spec before any API call:

Allowlist fields must be non-empty when configured; numeric caps must be non-negative. Violations fail at bundle initialization or before the write executes — never after.

computePolicy does not bound execution of an existing job. databricksRunJobTool and databricksNotebookTool carry their own required runPolicy / notebookPolicy, enforced on the run path itself. See Bound what a model can run.

Managed resources and safe deletion

Destructive tools only touch resources Fabric Harness can prove it created. Because Databricks services expose different metadata capabilities, three marker schemes exist:

• Jobs — created jobs carry the tags fabric-harness:managed=true and fabric-harness:fingerprint=<canonical spec hash>. ifExists: 'reuse' adopts only a single job matching name, managed tag, and fingerprint; anything ambiguous is refused. Updates refuse to remove the ownership tags and re-stamp the fingerprint.
• Unity Catalog — created catalogs and schemas carry the property fabric-harness.managed="true"; volumes (which have no properties) carry a comment prefixed [fabric-harness:managed]. deleteManaged() reads the object first and refuses to delete anything not carrying its marker. No force or cascade options exist.
• Genie Agents — the Genie management API supports neither tags nor properties, so ownership lives in a durable DatabricksManagedResourceStore manifest recording fingerprint, creator, and tool-call provenance. Deletion requires the exact managed id and expected fingerprint; an already-trashed managed id is treated as an idempotent success. MemoryDatabricksManagedResourceStore is only for tests and local development; use LakebaseDatabricksManagedResourceStore or another durable implementation in production.
• Asset Bundles — bundle ownership lives in the same DatabricksManagedResourceStore manifest (resourceType: "databricks-bundle"), recording a sha256 fingerprint of the sorted bundle source tree (excluding .databricks/ CLI state, node_modules/, and dist/). Deploy records the fingerprint after a successful CLI run; destroy requires the record and refuses to tear down a drifted tree unless force is set. See Checked-in Asset Bundles.

A create whose API response is ambiguous is never retried automatically, so a transient failure cannot orphan an unmarked resource. Schedules default off. Serverless jobs take the serverless branch; classic VM job compute requires a non-empty allowedPolicyIds and a matching policyId on every new cluster. Existing all-purpose clusters are denied unless their exact id appears in allowedExistingClusterIds.

Checked-in Asset Bundles

A job/pipeline bundle authored in YAML — by hand, by databricks bundle init, or by exporting a Lakeflow Designer canvas as a .designer.ipynb referenced from a notebook_task — stays checked-in IaC. What Harness governs is its lifecycle. The assetBundles option points the bundle at a directory containing databricks.yml and adds four model tools plus a typed client at bundle.assetBundle:

const dbx = databricks({
  host,
  principal,
  assetBundles: { bundleDir: './bundle', target: 'dev' },
  governance: { stewardAudience: 'data-platform' },
});

The fingerprint is a deterministic sha256 over the sorted bundle source tree (relative path plus content), excluding .databricks/, node_modules/, and dist/. The lifecycle shells out to the Databricks CLI (executable defaults to databricks, so it must be on PATH) and authenticates through the CLI's normal environment or --profile resolution; target, profile, and env pass through unchanged. CLI failures surface with a bounded stderr tail and are never retried automatically.

Several bundles in one agent

Passing a name-keyed map instead of one entry configures several bundles — a separate lifecycle per name, exposed as bundle.assetBundles:

const dbx = databricks({
  host,
  principal,
  assetBundles: {
    jobs: { bundleDir: './bundles/jobs', target: 'dev' },
    pipelines: { bundleDir: './bundles/pipelines', target: 'dev' },
  },
  governance: { stewardAudience: 'data-platform' },
});

await dbx.assetBundles?.jobs?.validate();

The tool names stay the same four. Every one of their input schemas gains a required bundle selector enumerating the configured names (jobs, pipelines), so a call must always say which bundle it means — including when only one bundle is configured, which keeps the schema and its approval digests stable if a second bundle is added later. Names must match ^[A-Za-z0-9][A-Za-z0-9_-]*$, and an empty map is rejected. Governed lineage names only the bundle the call selected, never the whole configured set, because the descriptor is a /bundle input path rather than a static value.

An approval grant binds the tool-call id, the executing principal, and the canonical tool input. bundle is part of that input, so a grant approved for { bundle: 'jobs' } cannot be replayed against { bundle: 'pipelines' } — distinct names cannot cross-approve. A name's target, however, is configuration rather than input: retargeting jobs from dev to prod does not invalidate outstanding grants for jobs, so a durable approved call replayed after that change would hit the new target. When dev and prod need separate approvals, model them as separate named entries (jobs_dev, jobs_prod) instead of editing one name's target.

Managed-resource identity is scoped per name and target — <bundle name from databricks.yml>#<configured name>[#<target>], where the #<target> suffix is present only when that entry configures a target — so several lifecycles can share one durable DatabricksManagedResourceStore without overwriting each other's fingerprint records. The single-bundle shape keeps the bare manifest name, so existing records stay addressable.

One limitation: approval routing is per tool name, so every named bundle routes to the same governance.stewardAudience. Per-bundle audiences are not expressible — if jobs and pipelines need different approvers, build two agents.

Like every authoring surface, assetBundles fails closed: without governance.stewardAudience (or the local-only allowUnapprovedAuthoring: true), bundle initialization throws instead of exposing ungated deploy/run/destroy tools. Application code can also drive the same lifecycle directly through databricksAssetBundleLifecycle() / databricksAssetBundleTools() without a model. See examples/with-databricks-bundle-deploy for a runnable steward-gated flow and fh add databricks bundle for the scaffold.

Genie Agents

Databricks renamed Genie spaces to Genie Agents, while the stable management and Conversation API paths retain /genie/spaces. Fabric consistently calls the resource identifier agentId.

The stable consumption path uses the typed DatabricksGenieClient. It starts or continues a conversation, requires a terminal COMPLETED state, retrieves every query result from its attachment-specific endpoint, and returns typed text, query, result, suggested-question, and visualization metadata. Missing ids, terminal failure, polling exhaustion, cancellation, oversized responses, and malformed attachments fail explicitly. Application code can also list conversation history and message comments or delete an exact conversation id; only ask is model-facing by default.

Genie lifecycle management is stable for the certified create/query/update/trash scope:

const dbx = databricks({
  host,
  principal,
  genie: {
    manage: {
      resourceStore: new LakebaseDatabricksManagedResourceStore({ client: lakebase }),
      delete: 'managed-only',
    },
  },
  governance: { stewardAudience: 'data-platform', catalogs: ['main'] },
});

const salesAnalyst = await dbx.genieAdmin?.create({
  version: 2,
  title: 'Sales analyst',
  parentPath: '/Shared/agents',
  warehouseId: 'warehouse-id',
  dataSources: [{ table: 'main.sales.orders' }],
});

DatabricksGenieAdmin provides list/get/create/update/delete, export/import, and permissions APIs. The serializer emits Databricks serialized_space version 2, deterministic 32-character ids, and the required sorted collections. Updates compare the caller's expected canonical fingerprint and send the last observed Databricks ETag. The management API has no general field mask: description is the only field that supports explicit clearing via remove; other fields can only be set to new values. Ambiguous creates, updates, and deletes are not retried. Raw serialized_space is accepted only by typed-client import/export; no built-in ToolDef accepts it.

Model-facing create, inspect, update, and optional delete tools use a normalized schema containing tables, columns, sample questions, and one text instruction. SQL examples, functions, joins, filters, expressions, measures, and benchmarks become available only when genie.manage.sqlPolicy is configured. The policy runs server-side for every SQL fragment, must return the complete parsed catalog.schema.object set, and denies any resource outside dataSources. An allowlisted prepared statement is also valid; returning an empty or partial parse for dynamic SQL is not. Mutation requires steward approval and a durable ownership record.

const certifiedSql = new Map([
  ['SELECT COUNT(*) FROM main.sales.orders', ['main.sales.orders']],
]);

const dbx = databricks({
  // ...identity, ownership store, governance...
  genie: {
    manage: {
      resourceStore: managedResourceStore,
      sqlPolicy: ({ sql }) => {
        const resources = certifiedSql.get(sql);
        if (!resources) throw new Error('SQL is not in the certified statement set');
        return { resources };
      },
    },
  },
});

Fabric's ordinary Genie client uses the Conversation API. The separately configured Agent Mode client supports Databricks' Beta SSE endpoint:

const dbx = databricks({
  host,
  principal,
  genie: {
    agentMode: {
      agentId: '0123456789abcdef0123456789abcdef',
      acknowledgeBeta: true,
      timeoutMs: 5 * 60_000,
      idleTimeoutMs: 2 * 60_000,
      modelTool: { maxOutputBytes: 512 * 1024 },
      onEvent: (event) => auditAgentModeProgress(event),
    },
  },
});

for await (const event of dbx.genieAgentMode!.respond('Summarize revenue by region')) {
  if (event.type === 'response.completed') console.log(event.response.output);
}

It validates the preview's 32-hex agent id, never retries the response-creating POST, propagates cancellation, enforces idle and overall deadlines, parses arbitrarily chunked SSE, preserves unknown Beta events, enforces monotonic sequence numbers and event/stream limits, and requires response.completed or response.failed. modelTool adds the governed databricks_genie_agent_mode tool; progress goes only to onEvent, while the model receives one bounded terminal response. listConversationItems() exposes the cursor-based history endpoint. A failed terminal event is yielded and then raises DatabricksGenieAgentModeResponseError on the next iterator step.

The workspace must be enrolled by the Databricks account team and a workspace admin must enable Genie Agents Agent Mode API in Previews. Agent Mode allows one in-flight response per conversation and can keep an SSE connection open for up to 90 minutes. Fabric does not silently substitute it for ordinary Genie conversations. See the Genie Agents API and Beta Agent Mode API for the native contracts.

Bind a Genie Agent to a Databricks App

export default {
  target: 'databricks-app',
  databricks: {
    app: {
      genie: {
        agentId: '0123456789abcdef0123456789abcdef',
        // permission defaults to CAN_RUN
      },
    },
  },
};

The generated bundle attaches a genie_space resource and injects DATABRICKS_GENIE_AGENT_ID through valueFrom; it does not embed the id in model context. CAN_EDIT and CAN_MANAGE are rejected unless the deployment block also sets authoring: true.

Serving and AI Search constraints

aiSearchAdmin takes a resource policy, not a boolean. allowedOperations decides which of the six tools are registered at all — an agent that only needs to sync an index never sees databricks_delete_search_endpoint:

Names are pinned into the tool inputSchema (const for one, enum for several) and re-checked at call time; an out-of-policy name throws DatabricksResourceNotAllowedError before the Vector Search API is reached.

AI Search direct-access creation requires a full schema and vector column dimension. The model-facing serving tools reject environmentVars; applications that deliberately need raw values must use the typed client so those values never enter model context. See the native Serving Endpoints API, AI Search Indexes API, and Unity Catalog Grants API.

Secrets and lineage

The databricks_put_secret input is { scope, key, secretRef: { kind: 'secret', name } }. The configured SecretProvider resolves material during server-side execution and passes it directly to Databricks. The value never appears in model context, tool input, or lineage. The raw-value method exists only on the typed client and is an application-level trust decision.

Structured governance metadata uses extended JSON Pointer paths, including * for arrays. Bundle initialization validates every authoring descriptor against its input schema. At execution, a required zero-match fails closed, every catalog-qualified resource is allowlist-checked, and every resource is recorded in lineage.

Examples and failure behavior

• examples/with-databricks-jobs-authoring: policy-bound multi-task Job lifecycle.
• examples/with-databricks-bundle-deploy: governed checked-in Asset Bundle validate/deploy/run/destroy with fingerprint drift detection.
• examples/with-databricks-dataeng: Lakeflow operation, authoring, and event expectations.
• examples/with-databricks-rag-admin: AI Search endpoint/index lifecycle and query verification.
• examples/with-databricks-authoring-admin: serving, non-destructive UC, workspace, and secret-reference writes.
• examples/with-databricks-genie-authoring: normalized Genie Agent lifecycle with a durable Lakebase ownership manifest.

All examples document exact credentials and expected output in their README. Missing approval routing fails at bundle initialization. Missing warehouse/secret provider dependencies, governance descriptor errors, grant/principal mismatches, catalog denial, native Databricks authorization, timeouts, ambiguous creates, and cleanup leaks fail explicitly.

Stability

Jobs, Lakeflow, AI Search administration, non-preview custom-model serving, managed-only UC administration, workspace writes, secret-reference writes, and Genie Agent management (for its create/query/update/trash scope) are stable for their documented scope, backed by retained protected-workspace lifecycle evidence. Jobs evidence covers serverless STANDARD and policy-bound classic definitions; AI Search evidence covers direct-vector and Delta Sync lifecycles. Provisioned Throughput, AI Gateway administration, Agent Mode streaming, and Databricks App OBO authentication retain their separate Databricks preview constraints. The full release gate, destructive certification lifecycle, and retained evidence are documented in Authoring certification and release evidence.

Upgrade note

Approvals persisted before grant provenance existed cannot authorize a post-upgrade tool call. An idempotent session detects the missing grant and requests approval again under a new provenance id. This is intentionally fail-closed; operators may see one new approval after upgrading a durable session.

Pair compute limits with databricksTenantCostLimit() and System Tables reconciliation when authoring Jobs or serving resources. Compute lifecycle outside governed Jobs remains deferred until budget guardrails can be enforced server-side.