Databricks development with Fabric Harness

Fabric Harness is the application runtime around Databricks AI and data services. It does not replace Unity Catalog, Spark, Lakeflow, MLflow, Jobs, SQL Warehouses, or Model Serving. It gives developers one TypeScript workflow for composing those services into agents that can run locally, deploy to Databricks Apps, preserve identity, enforce policy, persist state, and produce operational evidence.

Where Fabric fits

flowchart LR
  subgraph Author[Developer workflow]
    CODE[TypeScript job or agent]
    MOCK[Local mock and tests]
    BUILD[Fabric build]
  end

  subgraph Runtime[Fabric application runtime]
    APP[Databricks App]
    LOOP[Session and model loop]
    POLICY[Policy and approvals]
    AUDIT[Lineage, cost, and telemetry]
  end

  subgraph Platform[Databricks services]
    GATEWAY[Unity AI Gateway]
    DATA[SQL and Unity Catalog]
    RAG[AI Search and Genie]
    COMPUTE[Jobs and Lakeflow]
    STATE[Lakebase and Volumes]
    MLFLOW[MLflow and system tables]
  end

  CODE --> MOCK
  MOCK --> BUILD
  BUILD --> APP
  APP --> LOOP
  LOOP --> POLICY
  POLICY --> GATEWAY
  POLICY --> DATA
  POLICY --> RAG
  POLICY --> COMPUTE
  LOOP --> STATE
  LOOP --> AUDIT
  AUDIT --> MLFLOW

  classDef author fill:#f4f4f5,stroke:#71717a,color:#18181b
  classDef fabric fill:#dbeafe,stroke:#2563eb,color:#172554
  classDef control fill:#fef3c7,stroke:#d97706,color:#422006
  classDef dbx fill:#dcfce7,stroke:#16a34a,color:#052e16
  class CODE,MOCK,BUILD author
  class APP,LOOP,AUDIT fabric
  class POLICY control
  class GATEWAY,DATA,RAG,COMPUTE,STATE,MLFLOW dbx

Databricks remains authoritative for data permissions, compute, model access, and platform operations. Fabric owns the agent lifecycle around those services: admission, sessions, tools, approvals, durable state, retries, typed results, audit correlation, and deployment artifacts.

Start locally, connect later

Create a Databricks-oriented project without requiring workspace credentials:

npx @fabric-harness/cli init \
  --template databricks \
  --dir analytics-agent

cd analytics-agent
npm install

The template creates a finite analytics job, role, skill, governed SQL policy, Databricks App configuration, environment sample, and certification manifest.

fh agents
fh describe databricks-analyst
fh run databricks-analyst \
  --question 'Describe main.sales.orders' \
  --mock

Mock mode exercises discovery, input validation, tool assembly, the model loop, and HTTP routing. It deliberately does not simulate Unity Catalog grants or claim that a workspace API succeeded.

When the local behavior is ready, configure a PAT for single-user development or OAuth M2M for a production-like service principal:

DATABRICKS_HOST=https://adb-1234567890123456.7.azuredatabricks.net
DATABRICKS_CLIENT_ID=00000000-0000-0000-0000-000000000000
DATABRICKS_CLIENT_SECRET=resolve-from-your-secret-manager
DATABRICKS_WAREHOUSE_ID=0123456789abcdef
DATABRICKS_MODEL=system.ai.gpt-oss-20b
DATABRICKS_INFERENCE_MODE=auto

Run the same definition without --mock to exercise the real workspace. See Local naming and authentication for PAT, OAuth M2M, App identity, OBO, tenant, and deployment-profile behavior.

Use AI Gateway without custom HTTP plumbing

Use a discovered system.ai.* service as the model:

import { defineDatabricksAgent } from '@fabric-harness/databricks';
import { schema } from '@fabric-harness/sdk';
import policy from '../policies/databricks.js';

export default defineDatabricksAgent({
  name: 'databricks-analyst',
  description: 'Answer governed questions using workspace data.',
  input: schema.object({ question: schema.string() }),
  output: schema.string(),
  model: 'system.ai.gpt-oss-20b',
  tools: ['sql-read', 'tables', 'table-info'],
  triggers: { manual: true, webhook: true },
  sandbox: 'empty',
  policy,
});

Fabric automatically routes system.ai.* through the workspace Unity AI Gateway path. Custom endpoint names route through Model Serving. The Databricks provider adds:

• OAuth token acquisition, caching, and early refresh;
• request tags for submission, attempt, tenant, and agent identifiers;
• retry-safe requests and redacted provider errors;
• model usage and cost correlation;
• a common provider contract for local tests and deployed runtimes.

Discover enabled services in the target workspace instead of assuming a model is available. The integration map documents automatic routing and explicit overrides.

Compose the governed Databricks stack

Use databricks() when an application needs several Databricks services under one identity:

import { databricks } from '@fabric-harness/databricks';
import { init } from '@fabric-harness/sdk';

const dbx = databricks({
  host: process.env.DATABRICKS_HOST!,
  principal: {
    kind: 'service-principal',
    host: process.env.DATABRICKS_HOST!,
    clientId: process.env.DATABRICKS_CLIENT_ID!,
    clientSecret: process.env.DATABRICKS_CLIENT_SECRET!,
  },
  model: 'system.ai.gpt-oss-20b',
  warehouseId: process.env.DATABRICKS_WAREHOUSE_ID,
  aiSearch: {
    index: 'main.knowledge.docs_index',
    textColumn: 'chunk',
    idColumn: 'id',
    strategy: 'hybrid',
  },
  genie: {
    conversations: { agentId: process.env.DATABRICKS_GENIE_SPACE_ID! },
  },
  lakeflow: true, // list/status only; add { runPolicy } for start/stop
  consumption: true,
  governance: {
    catalogs: ['main'],
    stewardAudience: 'data-steward',
  },
});

const runtime = await init({
  modelProvider: dbx.modelProvider,
  tools: dbx.tools,
  policy: dbx.policy,
  store: dbx.store,
});

One principal is threaded through model calls, generated Databricks SDK clients, SQL, and optional Lakebase credential exchange. Fabric policy can narrow access or require approval, while Unity Catalog and Databricks resource ACLs make the final authorization decision.

Add only the workloads you need

Managed recipes add project-local wiring, compatible dependencies, environment stubs, and verification commands:

fh add databricks core
fh add databricks sql
fh add databricks ai-search
fh add lakebase
fh add lakeflow
fh add jobs

Recipes are dependency-aware and refuse incompatible ranges rather than silently replacing them. Use fh add --dry-run to inspect changes and fh update to update managed recipe files.

Workload patterns

Databricks-native RAG quality

Fabric's deterministic RAG chain follows a preprocess, retrieve, augment, generate, and validate flow. AI Search performs retrieval, AI Gateway performs generation, and MLflow 3 performs managed evaluation.

flowchart LR
  QUESTION[Golden-set question] --> RETRIEVE[AI Search retrieval]
  RETRIEVE --> GENERATE[AI Gateway generation]
  GENERATE --> TRACE[MLflow trace]
  TRACE --> JUDGES[Managed judges]
  JUDGES --> REL[Answer relevance]
  JUDGES --> RREL[Retrieval relevance]
  JUDGES --> GROUND[Groundedness]
  JUDGES --> SUFF[Sufficiency]
  JUDGES --> CORRECT[Correctness]
  REL --> GATE{All meet threshold?}
  RREL --> GATE
  GROUND --> GATE
  SUFF --> GATE
  CORRECT --> GATE
  GATE -->|Yes| PASS[Release evidence]
  GATE -->|No| BLOCK[Block release]

  classDef source fill:#f4f4f5,stroke:#71717a,color:#18181b
  classDef fabric fill:#dbeafe,stroke:#2563eb,color:#172554
  classDef dbx fill:#dcfce7,stroke:#16a34a,color:#052e16
  classDef decision fill:#fef3c7,stroke:#d97706,color:#422006
  classDef deny fill:#fee2e2,stroke:#dc2626,color:#450a0a
  class QUESTION source
  class RETRIEVE,GENERATE,TRACE,JUDGES dbx
  class REL,RREL,GROUND,SUFF,CORRECT,PASS fabric
  class GATE decision
  class BLOCK deny

The certification fixture uses managed relevance, retrieval relevance, groundedness, sufficiency, and correctness judges with a configurable threshold. A project should replace the small certification fixture with its own domain questions, expected facts, retrieval expectations, insufficient-context cases, and adversarial inputs. See RAG on Databricks.

Durable Apps instead of stateless demos

databricks-app bundles the Node server and agent definitions for Databricks Apps. Optional Lakebase persistence stores sessions, submissions, and conversation streams. Unity Catalog Volumes store governed attachments.

fh build --target databricks-app
fh deploy --preview --target databricks-app --profile analytics-dev
fh deploy --target databricks-app --profile analytics-dev

The generated artifact contains app.yaml, databricks.yml, the self-contained server bundle, roles, skills, definitions, and build manifest. Runtime credentials come from App identity and resource bindings, not the developer's deployment profile.

For configuration-preserving recovery, redeploy the generated bundle with the same variables. A bare App start can omit values injected during deployment. The Databricks App tutorial documents the certified deployment and recovery procedure.

Governance and operations are runtime behavior

Fabric controls are evaluated around every agent operation rather than described only in a prompt:

• capability and catalog policies constrain available actions;
• sensitive SQL, pipeline, and mutation tools can require durable approval;
• Fabric principals and tenants propagate into submissions and request tags;
• Unity Catalog remains authoritative for tables, schemas, volumes, rows, and columns;
• Lakebase telemetry joins actors, submissions, governed objects, outcomes, and estimated cost;
• System Tables reconcile delayed actual usage with tenant and agent budgets;
• cascade deletion removes sessions, submissions, streams, and attachments;
• MLflow and OpenTelemetry expose traces and operational evidence.

This is most useful when an agent moves from a notebook experiment to a shared application with multiple users, governed data, long-running work, or production operating requirements.

When to use Fabric Harness

Use Fabric when the application needs several of these together:

• local TypeScript development and Databricks App deployment;
• model calls plus SQL, retrieval, Jobs, Lakeflow, Genie, or Feature Serving;
• persistent agents or durable workflow state;
• user, service-principal, tenant, and OBO identity propagation;
• approval, policy, audit, deletion, or cost enforcement;
• repeatable workspace certification and deployment evidence.

A direct Databricks SDK or notebook is usually simpler for a one-off query, a standalone Spark transformation, or a single model request with no agent lifecycle. Fabric is valuable when those calls need to become a governed application.

Production validation

Fabric ships local contracts and a protected workspace certification runner. Before approving a deployment, validate the actual cloud, region, workspace, identity, resources, data, and workload:

1. Verify OAuth M2M or App identity and every required resource permission.
2. Prove one allowed and one deliberately denied Unity Catalog operation.
3. Exercise AI Gateway, SQL, retrieval, Jobs, Lakeflow, and other enabled services.
4. Deploy the App, persist work, redeploy, and verify recovery and cascade deletion.
5. Run the project's MLflow evaluation dataset and enforce quality thresholds.
6. Verify lineage and System Tables cost reconciliation for real tenant tags.
7. Test concurrency, rate limits, long-running work, and expected failure modes.
8. Exercise OBO login, expiry, refresh, and user-specific grants when OBO is enabled.
9. Retain redacted certification, compatibility, recovery, and conformance evidence.

Certification records are environment-specific. Do not infer that a passing Azure workspace also certifies an untested AWS/GCP workspace, region, preview feature, or production corpus. Use the workspace compatibility matrix and authoring certification guide for the complete evidence path.

Next steps

• Create the reference project
• Understand local names and identities
• Choose Databricks integrations
• Build and evaluate RAG
• Deploy a Databricks App
• Apply enterprise controls