Databricks workload map

Fabric Harness uses three integration levels. First-party workloads have typed APIs and an explicit certification level in the package capability registry; Beta surfaces can remain contract-tested until their protected live gate passes. Managed recipes orchestrate the official Databricks CLI, Python SDK, SQL, or Jobs for workloads whose authoring experience is Python-native. The escape hatch lets an agent invoke any current Databricks REST, SQL, notebook, Job, or Declarative Automation Bundle resource without waiting for a dedicated TypeScript wrapper. One bundle operation is no longer escape-hatch territory: the validate/deploy/run/destroy lifecycle of a checked-in bundle is a first-party governed surface through the assetBundles option, with steward approval and source-fingerprint drift detection.

Start from the business workload

Analytics and BI copilots

Use ordinary Genie conversations for governed natural-language analytics and databricksSqlReadTool() when the model needs inspectable SQL. The analyticsCopilotGovernance() pack leaves Genie questions and single-statement SELECT reads interactive while routing arbitrary SQL and Genie lifecycle changes to a steward. The runnable with-analytics-copilot example proves the allow, approval, and local-denial behavior without credentials.

For an employee-facing Databricks App, build the bundle per request with a verified OBO principal. For a shared automation service, use OAuth M2M and a least-privilege service principal. In both cases, retain the generated SQL, tool decision, acting principal label, Unity Catalog outcome, and cost correlation with the submission.

Knowledge assistants and RAG

Keep indexing, embedding pipelines, and governed source tables native to Databricks. Fabric queries the AI Search index, builds bounded context, invokes AI Gateway or Model Serving, validates citations, and exports evaluation rows to MLflow. This makes the application layer testable without pretending to replace Databricks retrieval or evaluation.

Data engineering and operations agents

Use Jobs and Lakeflow for compute. Fabric should admit the request, bind approval to the exact definition or run input, submit idempotently, persist the receipt, collect terminal status and outputs, and make cleanup visible. Keep pipeline source, cluster policies, service-principal grants, and production promotion in the platform team's existing Databricks delivery process.

Stateful customer and employee Apps

Deploy the Node runtime as a Databricks App, persist sessions/submissions/conversation streams in Lakebase, and store governed attachments in UC Volumes. The release proof is behavioral: create a session, restart or redeploy the App, recover it, continue from the prior offset, and verify tenant deletion. A successful build alone is not durability evidence.

flowchart TB
  NEED[Databricks workload] --> NATIVE{Typed Harness API?}
  NATIVE -->|Yes| FIRST[First-party package surface]
  NATIVE -->|No| PYTHON{Python or SQL native?}
  PYTHON -->|Yes| RECIPE[Managed Job or notebook recipe]
  PYTHON -->|No| ESCAPE[REST, SQL, Job, or Bundle escape hatch]
  FIRST --> POLICY[Harness policy, identity, approvals, telemetry]
  RECIPE --> POLICY
  ESCAPE --> POLICY
  POLICY --> EVIDENCE[Receipts, MLflow traces, lineage, cost, cleanup]

  classDef decision fill:#fef3c7,stroke:#d97706,color:#422006
  classDef route fill:#dbeafe,stroke:#2563eb,color:#172554
  classDef evidence fill:#dcfce7,stroke:#16a34a,color:#052e16
  class NATIVE,PYTHON decision
  class FIRST,RECIPE,ESCAPE,POLICY route
  class EVIDENCE evidence

Agent and governance workloads

Use the Responses endpoint as the common boundary for Apps and subagents. Use databricksWithManagedMcp() for Databricks managed MCP endpoints, Unity Catalog MCP Services, Genie tools, or another App's MCP endpoint. The adapter rotates OBO/M2M credentials, enforces workspace origin, applies allowTools, and requires an explicit effect for surfaces where read versus mutation cannot be inferred. Remote discovery therefore cannot silently widen the local capability policy.

Unity Catalog Agent Services is a typed discoverability and permission surface. The package requires explicit Beta acknowledgement, defaults external Harness registrations to /responses, and supports create, get, list, update, permission, grant, revoke, and delete operations. Runtime invocation is not available through the Databricks service during the current Beta; call the Harness endpoint directly.

For long-lived conversations, use Lakebase-backed Harness sessions today. Databricks managed agent memory can be introduced as another session-memory adapter without changing agent code because Harness sessions already depend on the common store interface.

Retrieval and data workloads

Databricks AI Search is the current product name. Harness uses databricksAiSearch() and aiSearch for new code while retaining the former names as deprecated source-compatible aliases. The REST API continues to use its existing vector-search path. The online RAG path is documented in Databricks RAG. Create and update indexes using a Databricks Job or Bundle, then give the App service principal SELECT and endpoint access before invoking the retriever.

Lakeflow authoring belongs in declarative SQL or Python source. Keep pipeline definitions in the deployed project, run them through a Job or Bundle, and use Harness Lakeflow tools for start, stop, status and agent approvals. When the bundle itself is checked in, hand its validate/deploy/run/destroy lifecycle to the governed assetBundles surface instead of an ungated shell call. Treat expectations and event-log quality metrics as release evidence.

ML and evaluation workloads

Use MLflow for trace storage, evaluation datasets, judges, labeling sessions, reusable scorers and production monitoring. Harness supplies execution identity and correlation; MLflow remains the evaluation engine. Use RAG evaluation as the working pattern for a generated Python evaluation Job in evaluation and quality.

Training and fine-tuning should run on Databricks AI Runtime through a Job or notebook. Register the result in Unity Catalog, promote with model aliases, and deploy with Model Serving traffic controls. Do not build new workflows on the deprecated Foundation Model Training API.

Adding a new workload

Before exposing a new Databricks API, define its auth modes, UC permissions, cloud availability, preview status, cleanup behavior, retry/idempotency rules, mock contract, live gate, runnable example, and documentation route. The package capability registry is the source of truth for these claims:

import { listDatabricksCapabilities } from '@fabric-harness/databricks';

const productionSurfaces = listDatabricksCapabilities({ status: 'stable' });

This keeps experimental Databricks surfaces usable without presenting them as generally available or silently enabling them in production.