Voice Providers

Fabric Harness supports two voice modes: realtime, where one model owns audio input, audio output, and tools; and pipeline, where you choose separate STT, LLM, and TTS providers.

Choose a voice architecture

• Choose OpenAI Realtime when you want a single WebSocket connection with model-managed turn detection and tool calling.
• Choose a Deepgram + ElevenLabs pipeline when independent STT, LLM, and TTS selection matters.
• Choose a Cartesia pipeline when you want STT and TTS from one pipeline provider while retaining your preferred LLM.
• Serve either mode through WS /sessions/:id/voice to keep provider credentials on the server.

Mode comparison

Provider matrix

Provider models, languages, and prices change independently of Fabric Harness. Confirm the current provider offering, then use registerModelPrices to apply your public or negotiated rate card to cost telemetry.

Realtime mode (OpenAI)

import { OpenAIRealtimeVoiceProvider } from '@fabric-harness/sdk';

const provider = new OpenAIRealtimeVoiceProvider({
  apiKey: process.env.OPENAI_API_KEY!,
  model: 'gpt-realtime',
});

const voice = await provider.connect({
  instructions: 'You are a friendly intake agent.',
  voice: 'alloy',
  audioFormat: 'pcm16',                 // or 'g711_ulaw' for Twilio.
  tools: [submitFieldTool],
  turnDetection: 'server_vad',
});

The model owns the audio loop end-to-end. Tool calls relay through the existing tool_call / submitToolResult contract — same as pipeline mode.

Pipeline mode (BYO STT + LLM + TTS)

import {
  PipelineVoiceProvider,
  DeepgramSttProvider,
  ElevenLabsTtsProvider,
  AnthropicModelProvider,
} from '@fabric-harness/sdk';

const provider = new PipelineVoiceProvider({
  stt: new DeepgramSttProvider({
    apiKey: process.env.DEEPGRAM_API_KEY!,
    model: 'nova-3',
  }),
  tts: new ElevenLabsTtsProvider({
    apiKey: process.env.ELEVENLABS_API_KEY!,
    defaultVoice: '21m00Tcm4TlvDq8ikWAM',          // 'Rachel'
    model: 'eleven_turbo_v2_5',
  }),
  llm: new AnthropicModelProvider({ apiKey: process.env.ANTHROPIC_API_KEY! }),
  model: 'claude-haiku-4-5-20251001',
});

const voice = await provider.connect({
  instructions: 'You are a friendly intake agent. One question at a time.',
  audioFormat: 'pcm16',
  tools: [submitFieldTool],
});

mic.on('frame', (pcm) => voice.sendAudio(pcm));

for await (const event of voice.events()) {
  if (event.type === 'audio_delta') speaker.write(event.audio);
  if (event.type === 'tool_call') {
    const output = await runTool(event.name, event.input);
    await voice.submitToolResult({ id: event.id, output });
  }
  if (event.type === 'response_done') {
    // event.usage contains rolled-up tokens, sttSeconds, ttsCharacters, costUsd.
  }
}

The contract is identical to realtime mode — VoiceSession events, tool calls, cost telemetry, costBudget, cost_limit events. Swap providers without rewriting your loop.

Single-vendor variant (Cartesia)

import {
  PipelineVoiceProvider,
  CartesiaSttProvider,
  CartesiaTtsProvider,
} from '@fabric-harness/sdk';

const provider = new PipelineVoiceProvider({
  stt: new CartesiaSttProvider({ apiKey: process.env.CARTESIA_API_KEY! }),
  tts: new CartesiaTtsProvider({ apiKey: process.env.CARTESIA_API_KEY! }),
  llm: someLlmProvider,
  model: 'claude-haiku-4-5-20251001',
});

Barge-in

Pipeline mode wires barge-in through the STT VAD. When the user starts speaking while the agent is mid-utterance, the STT emits speech_started; the pipeline aborts the in-flight LLM call, cancels the TTS stream, and returns to listening. Your UI is responsible for stopping playback when audio_delta events stop arriving.

Realtime mode handles this server-side via turnDetection: 'server_vad'.

Cost telemetry

Both modes feed the same telemetry surface. response_done.usage includes:

{
  inputTokens: number;       // LLM input
  outputTokens: number;      // LLM output
  audioInputTokens?: number; // realtime mode only
  audioOutputTokens?: number;// realtime mode only
  sttSeconds?: number;       // pipeline mode (Deepgram/Cartesia)
  ttsCharacters?: number;    // pipeline mode (ElevenLabs/Cartesia)
  costUsd?: number;          // rolled up via static price table
}

Wire costBudget to enforce per-call / per-session / per-tenant ceilings — voice participates in the same v1.4 cost-budget machinery as text agents.

import { CostBudgetTracker } from '@fabric-harness/sdk';

const budget = new CostBudgetTracker({ perCallUsd: 0.50, perSessionUsd: 5 });
await provider.connect({ costBudget: budget });

Evaluate a voice stack

Test candidate providers with representative audio before choosing a production stack. Measure:

• time to first transcript and first synthesized audio;
• transcription quality for your languages, accents, vocabulary, and audio channel;
• interruption behavior under real network conditions;
• voice consistency and pronunciation for your domain;
• provider region, retention, residency, and audit controls;
• end-to-end cost using your own traffic distribution and rate card.

Because realtime and pipeline mode share the VoiceSession contract, you can run the same application loop against each candidate and compare session events and usage telemetry.

When to bring your own provider

Implement TtsProvider, SttProvider, or VoiceProvider directly and pass it into PipelineVoiceProvider. Reasons to build your own:

• On-prem TTS (Riva, Coqui) for compliance.
• Whisper-via-vLLM for cheap multilingual STT.
• Translate-on-the-wire layers (e.g. STT in Spanish → MT → English LLM → TTS in Spanish).

The interfaces (TtsProvider, SttProvider) are intentionally narrow — synthesize(text) → AsyncIterable<Uint8Array> and open() → SttSession are the only required methods.

See also

• Voice — the VoiceSession contract and OpenAI Realtime usage.
• Cost telemetry — pricing and rate-card overrides.
• Connector catalog — telephony bridges (Twilio, Vonage) for phone audio.