Databricks App Tutorial

This tutorial deploys a finite analytics job into Databricks Apps. The App service principal calls Model Serving and Unity Catalog APIs, while optional Lakebase stores sessions, durable submissions, and conversation streams.

Read the Databricks architecture first when evaluating identity, governance, or service boundaries. Use this page for the scaffold-to-deployment procedure.

Prerequisites

• Node.js 22+ and pnpm or npm.
• A Databricks workspace with Apps enabled.
• A Model Serving endpoint.
• A SQL warehouse for the generated analytics tools.
• Optional: a Lakebase Autoscaling endpoint and OAuth Postgres role.

1. Scaffold the project

npx @fabric-harness/cli init --template databricks --dir analytics-agent
cd analytics-agent
npm install

The template creates:

.fabricharness/
  jobs/databricks-analyst.ts
  roles/data-analyst.md
  skills/analyze-table/SKILL.md
  config.ts
.env.example
AGENTS.md
package.json

The generated job uses defineDatabricksAgent():

import { defineDatabricksAgent } from '@fabric-harness/databricks';
import { schema } from '@fabric-harness/sdk';

export default defineDatabricksAgent({
  name: 'databricks-analyst',
  description: 'Answer questions using SQL and Unity Catalog.',
  input: schema.object({ question: schema.string() }),
  output: schema.string(),
  triggers: { webhook: true, manual: true },
  model: 'system.ai.gpt-oss-20b',
  tools: ['sql', 'tables', 'table-info'],
  sandbox: 'local',
});

2. Run without credentials

The mock path validates discovery, schemas, tool assembly, and the model loop without contacting a workspace:

fh agents
fh describe databricks-analyst
fh run databricks-analyst \
  --question "What tables are available in main?" \
  --mock

Mock mode does not validate OAuth scopes, Unity Catalog grants, SQL execution, or deployment.

3. Configure a live workspace

Copy the template and fill the required values:

cp .env.example .env.local

DATABRICKS_HOST=https://<workspace-host>
DATABRICKS_CLIENT_ID=00000000-0000-0000-0000-000000000000
DATABRICKS_CLIENT_SECRET=replace-with-secret-reference
DATABRICKS_WAREHOUSE_ID=0123456789abcdef
DATABRICKS_MODEL=system.ai.gpt-oss-20b

Run a live source invocation:

fh run databricks-analyst --question "Describe main.sales.orders"

The service principal must have workspace access, permission to invoke the serving endpoint, warehouse usage, and the required Unity Catalog grants.

4. Build the App

fh build --target databricks-app

Expected summary:

Build complete
  Output    .fabricharness/build/databricks-app
  Manifest  .fabricharness/build/databricks-app/manifest.json
  Jobs      databricks-analyst
  Agents    none

Inspect the v2 manifest before deployment:

jq '{schemaVersion, jobs, agents, entrypoint}' \
  .fabricharness/build/databricks-app/manifest.json

{
  "schemaVersion": 2,
  "jobs": [{ "name": "databricks-analyst", "kind": "job" }],
  "agents": [],
  "entrypoint": "dist/server.mjs"
}

The artifact includes app.yaml, databricks.yml, the shared v2 Node server, and bundled .mjs definitions. It does not regenerate a separate legacy HTTP runtime.

5. Deploy with an Asset Bundle

cd .fabricharness/build/databricks-app
databricks auth login --host "$DATABRICKS_HOST"
databricks bundle validate
databricks bundle deploy

Set provider and resource configuration through Databricks App environment/resources rather than committing secrets. Apps supplies DATABRICKS_APP_PORT; the generated app.yaml starts dist/server.mjs on that port.

For a configuration-preserving production restart, pass the App name explicitly so the Databricks CLI uses the Apps API and restarts the existing active deployment:

databricks apps stop "$DATABRICKS_APP_NAME"
databricks apps start "$DATABRICKS_APP_NAME"

Running databricks apps start without a name inside the generated bundle directory enters project mode and resolves bundle variables again. Use the explicit-name form for a restart, or rerun fh deploy --target databricks-app when applying new source or configuration. The protected live workflow verifies the actual platform behavior: a bare stop/start creates a new deployment snapshot from the same configured source path. It then proves that the managed App resources, pending human approval, Lakebase sessions, submissions, conversation offsets, and UC Volume attachments remain usable after the restart. Static, non-secret workspace settings are emitted into app.yaml, so the new snapshot retains its catalog, schema, volume, model endpoint, and SQL warehouse configuration.

Once the App is running, invoke the finite job through its App URL:

curl -sS "$DATABRICKS_APP_URL/jobs/databricks-analyst" \
  -H "authorization: Bearer $DATABRICKS_TOKEN" \
  -H 'content-type: application/json' \
  -d '{"question":"What were yesterday’s top products?"}'

6. Add Lakebase durability

Fabric needs both Postgres connection information and the full Lakebase endpoint resource name:

DATABRICKS_LAKEBASE_HOST=ep-id.database.us-west-2.cloud.databricks.com
DATABRICKS_LAKEBASE_DATABASE=databricks_postgres
DATABRICKS_LAKEBASE_USER=00000000-0000-0000-0000-000000000000
DATABRICKS_LAKEBASE_ENDPOINT=projects/project-id/branches/branch-id/endpoints/endpoint-id
PGPORT=5432

The scaffold includes the pg dependency. At runtime Fabric:

1. Gets a workspace OAuth token from the App service principal.
2. Exchanges it at POST /api/2.0/postgres/credentials using the endpoint resource name.
3. Supplies the database credential through the pool password callback.
4. Refreshes before expiry and single-flights concurrent refreshes.
5. Injects Lakebase session, submission, and conversation-stream stores into the shared server.

Never place the workspace OAuth token directly in PGPASSWORD.

For fh deploy, provide the Lakebase resource names to the bundle separately from the direct PostgreSQL connection values shown above:

export BUNDLE_VAR_lakebase_endpoint='projects/project-id/branches/production/endpoints/primary'
export BUNDLE_VAR_lakebase_database_resource='projects/project-id/branches/production/databases/app-db'
fh deploy --target databricks-app

app-db is the Lakebase database resource ID. It can differ from the PostgreSQL database name such as databricks_postgres. Use the name field returned by databricks postgres list-databases projects/project-id/branches/production.

For a custom server entrypoint, the equivalent wiring is:

import { databricksApp } from '@fabric-harness/databricks';
import { startDevServer } from '@fabric-harness/node';

const app = databricksApp();
const server = await startDevServer({
  port: Number(process.env.DATABRICKS_APP_PORT ?? 8080),
  host: '0.0.0.0',
  ...(await app.serverOptions()),
});

const shutdown = async () => {
  await server.close();
  await app.close();
};
process.once('SIGTERM', () => void shutdown());
process.once('SIGINT', () => void shutdown());

7. Add a persistent agent

Create .fabricharness/agents/copilot.ts:

import { defineAgent } from '@fabric-harness/sdk';

export default defineAgent(({ id }) => ({
  name: 'copilot',
  model: 'databricks/system.ai.gpt-oss-20b',
  instructions: `You are the governed analytics copilot for account ${id}.`,
  triggers: { webhook: true },
}));

Rebuild. The manifest now reports the finite job under jobs and copilot under agents. Persistent input returns a durable receipt:

curl -i "$DATABRICKS_APP_URL/agents/copilot/acct-42" \
  -H "authorization: Bearer $DATABRICKS_TOKEN" \
  -H 'content-type: application/json' \
  -d '{"message":"Summarize this account’s sales."}'

Use @fabric-harness/client to wait or stream by offset. See Persistent agents.

8. Validate before production

Run the repository live suite with a controlled workspace:

FABRIC_DATABRICKS_TEST=1 \
FABRIC_DATABRICKS_LAKEBASE_TEST=1 \
pnpm --filter @fabric-harness/databricks test

The production gate should cover App deployment, service-principal access, user OBO where used, Unity Catalog denials, Model Serving, SQL, and Lakebase restart recovery. Local contract tests do not replace that workspace-specific validation.

See also

• Databricks integration reference
• Build and run artifacts
• Enterprise controls
• Production readiness