Live tests

Fabric Harness keeps live provider tests opt-in. Default CI uses mocks and structural tests only; live tests require credentials and are skipped unless their FABRIC_*_TEST=1 flag is set. A green workflow therefore proves only the steps that actually ran. The public Validation status records unconfigured suites explicitly; do not interpret a skipped credentialed step as provider evidence.

Sandbox providers

Daytona

FABRIC_DAYTONA_TEST=1 \
DAYTONA_API_KEY=... \
pnpm --filter @fabric-harness/connectors test

Optional:

DAYTONA_IMAGE=ubuntu:latest

The sandbox jobs create a provider resource, wrap it in SandboxEnv, and run assertSandboxCertification(). The report verifies shell, binary files, cwd/env, timeout, abort, streaming callbacks, portable refs, reconnect, and cleanup. Credentialed reports are uploaded as JSON CI artifacts without credentials or workspace content.

E2B

FABRIC_E2B_TEST=1 \
E2B_API_KEY=... \
pnpm --filter @fabric-harness/connectors test

Optional:

FABRIC_E2B_PACKAGE=@e2b/code-interpreter
E2B_TEMPLATE=...

Modal

The Modal suite creates a native TypeScript SDK sandbox and validates the complete portable contract:

FABRIC_MODAL_TEST=1 \
MODAL_TOKEN_ID=... \
MODAL_TOKEN_SECRET=... \
pnpm --filter @fabric-harness/connectors test

Azure

Azure OpenAI

FABRIC_AZURE_OPENAI_TEST=1 \
AZURE_OPENAI_ENDPOINT=... \
AZURE_OPENAI_API_KEY=... \
AZURE_OPENAI_DEPLOYMENT=... \
pnpm --filter @fabric-harness/azure test

Foundry Agent Service

FABRIC_AZURE_FOUNDRY_TEST=1 \
AZURE_FOUNDRY_PROJECT_ENDPOINT=... \
AZURE_FOUNDRY_AGENT_ID=... \
AZURE_TOKEN=... \
pnpm --filter @fabric-harness/azure test

Optional:

FABRIC_AZURE_FOUNDRY_POLL=0

ARM tools

FABRIC_AZURE_ARM_TEST=1 \
AZURE_SUBSCRIPTION_ID=... \
AZURE_TOKEN=... \
pnpm --filter @fabric-harness/azure test

Optional resources enable individual tests:

AZURE_CONTAINER_APPS_JOB_RESOURCE_GROUP=...
AZURE_CONTAINER_APPS_JOB_NAME=...
AZURE_AKS_RESOURCE_GROUP=...
AZURE_AKS_CLUSTER_NAME=...
AZURE_AKS_TEST_COMMAND='kubectl get ns'
AZURE_ACI_RESOURCE_GROUP=...
AZURE_ACI_CONTAINER_GROUP=...
AZURE_ACI_CONTAINER_NAME=...
AZURE_ACI_TEST_COMMAND='echo fabric-harness-aci-smoke'

Databricks

Use the package suite for low-level API diagnostics:

FABRIC_DATABRICKS_TEST=1 \
DATABRICKS_HOST=https://<workspace-host> \
DATABRICKS_TOKEN=... \
pnpm --filter @fabric-harness/databricks test

Optional resources:

DATABRICKS_WAREHOUSE_ID=...
DATABRICKS_CATALOG=main
DATABRICKS_SCHEMA=default
DATABRICKS_JOB_ID=...
FABRIC_DATABRICKS_WAIT_FOR_JOB=1
DATABRICKS_NOTEBOOK_PATH=/Repos/acme/smoke
DATABRICKS_CLUSTER_ID=... # optional; omit for serverless notebook execution
DATABRICKS_MLFLOW_RUN_ID=...
DATABRICKS_WORKSPACE_ROOT=/Repos/acme
DATABRICKS_AGENT_SERVICES_TEST=1
DATABRICKS_AGENT_SERVICE_CONNECTION=main.agents.fabric_agent_connection
DATABRICKS_AGENT_SERVICE_TEST_PRINCIPAL=fabric-agent-consumers # optional

Use the certification gate for release evidence. It covers the App, ResponsesAgent proxy, AI Gateway, SQL and UC allow/deny, AI Search and citation-validated RAG, MLflow managed RAG evaluation, Genie, Feature Serving, Lakeflow, Jobs, serverless notebook execution, Volumes, Lakebase, lineage, Unity Catalog Agent Services lifecycle, System Tables, and cost reconciliation:

pnpm databricks:cert:plan
FABRIC_DATABRICKS_PROVISION=1 pnpm databricks:cert:provision
pnpm databricks:certify

Protected CI runs non-user live gates daily and runs Databricks certification for releases or on demand. User-OBO and two-user evidence is operator-attended until an audited JIT broker exists: maintainers force-refresh two named CLI profiles with the repository runbook command, update the protected Environment through stdin, and dispatch immediately. Scheduled Databricks execution reports the credential window as not configured instead of treating an expired static token as a product regression. Certification records retain only the acquisition mode and remaining lifetime, not token values or user identifiers. See authoring certification for the public evidence contract.

The rolling workflow remains red until every required calendar day and cross-tier date is present. Whether it passes or fails, it retains databricks-first-class-rolling.json; a failed report has status not-established, missing dates, observed isolation/dynamic bindings, and redacted validation errors. This diagnostic artifact measures progress but never counts as certification. For a bounded customer evaluation, use the exact-artifact controlled-pilot gate instead of weakening the rolling standard.

The burst gate deliberately reaches the production App's 429 boundary. CI then waits one configured rate-limit window before MLflow validates and deploys the ResponsesAgent proxy. Keep that boundary when adapting the workflow: removing it makes a healthy proxy inherit the load probe's exhausted bucket and turns independent release evidence into an order-dependent test.

Existing live test families

Model providers:

FABRIC_OPENAI_TEST=1 OPENAI_API_KEY=... pnpm --filter @fabric-harness/sdk test
FABRIC_ANTHROPIC_TEST=1 ANTHROPIC_API_KEY=... pnpm --filter @fabric-harness/sdk test
FABRIC_AZURE_OPENAI_TEST=1 AZURE_OPENAI_API_KEY=... AZURE_OPENAI_ENDPOINT=... pnpm --filter @fabric-harness/sdk test

Docker sandbox:

FABRIC_DOCKER_TEST=1 pnpm --filter @fabric-harness/sdk test

Temporal:

# Local/dev-server workflow smoke. Requires a reachable Temporal frontend.
FABRIC_TEMPORAL_TEST=1 \
FABRIC_TEMPORAL_ADDRESS=localhost:7233 \
pnpm --filter @fabric-harness/temporal test -- integration.test.ts

# Real-model Temporal smoke. Requires model/provider credentials too.
FABRIC_TEMPORAL_REAL_MODEL_TEST=1 pnpm --filter @fabric-harness/temporal test -- live-model.test.ts

When using temporalio/auto-setup with Postgres in CI, set DB=postgres12, DB_PORT=5432, POSTGRES_SEEDS, POSTGRES_USER, and POSTGRES_PWD, then wait for a real Temporal gRPC connection before running tests. TCP port-open checks can pass before Temporal is ready.

Cloudflare:

FABRIC_CLOUDFLARE_TEST=1 \
FABRIC_CLOUDFLARE_WORKER_URL=... \
pnpm --filter @fabric-harness/cloudflare test

CI guidance

The protected live workflow uploads a live-status-* JSON artifact for optional MCP, direct model, remote sandbox, and channel suites. Each artifact lists configured checks, missing environment variable names, and the job outcome without serializing credential values. executed-passed is evidence for that run; unconfigured is not.

• Keep default CI credential-free.
• Run live tests in scheduled or manually triggered workflows.
• Use isolated resource groups/workspaces/projects.
• Prefer read-only tests unless a test is explicitly validating an execution target.
• Clean up provider sandboxes with adapter cleanup: true where available.
• Never echo tokens or provider SDK objects into prompts, logs, artifacts, or session history.
• Retain the executed-step summary or certification artifact with the release. Record a missing credential as unconfigured, not passed.