A sub-second voice agent loop, end to end.

// STT adapters implement: feed, flush, reset, id
// LLM adapters implement: stream, id
// TTS adapters implement: feed, stream, end, reset, id

// Each layer is one TS file behind a small interface,
// selected by an env var through a registry.

// STT_PROVIDER=whispercpp | deepgram | whisper
// LLM_PROVIDER=groq       | sarmalink | openai
// TTS_PROVIDER=opentts    | cartesia | elevenlabs

// Adding an OpenAI-compatible LLM adapter is a handful of lines
// because all three OpenAI-compatible adapters share
// apps/server/src/adapters/llm/sse.ts

// The orchestrator advertises the registered tool definitions to the
// model on every LLM call. The shared SSE reader assembles fragmented
// tool_calls deltas into complete calls.

// When the model requests a tool:
//   1. orchestrator runs the matching handler from ToolRegistry
//   2. appends an "assistant" turn recording the request
//   3. appends a "tool" turn carrying the result
//   4. re-streams so the model finishes with grounded data

// Bounded by maxToolRounds. Handler errors are returned to the model
// as a tool result rather than crashing the session.

// Browser to server, 20ms PCM16 frames, base64 encoded.
{ "type": "audio", "payload": "<base64 PCM16>" }

// Server to browser, sentence-by-sentence TTS chunks.
{ "type": "tts.chunk", "payload": "<base64 PCM16>" }
{ "type": "barge-in" }
{ "type": "turn.end" }

The orchestrator is transport-agnostic. Terminating over mediasoup or LiveKit is a swap at the edge without touching the pipeline. The starter keeps the transport simple on purpose; the engineering value is in the duplex machinery, not the negotiation layer.

Both the in-flight LLM and TTS streams, through their respective AbortControllers. The abort signal propagates through the fetch body reader and the for await loops, so there are no orphaned streams talking over the user. STT and TTS adapters are reset, a barge-in control message is emitted, and the machine drops to LISTEN.

Yes for the state machine, barge-in, and tool calls. To get real transcripts and audio you set GROQ_API_KEY or point WHISPERCPP_URL and OPENTTS_URL at running servers. The test suite drives the full pipeline through fake adapters and a fake socket.

A handful of lines. Implement the LLM interface (stream, id) and reuse apps/server/src/adapters/llm/sse.ts for streaming and tool_calls assembly. Register it and set LLM_PROVIDER to the new id.

In apps/server/src/pipeline/tools.ts. The starter ships with get_time and add_numbers as worked examples. Your tool advertises a JSON Schema; the orchestrator advertises every registered tool to the model on every call.

Because the orchestrator is the design surface, not the wire format. A plain {type, payload} message is the simplest thing to swap for a binary frame, a Datachannel, or a media-server track without changing the pipeline.

Yes. Set LLM_PROVIDER=sarmalink and the adapter calls into the SarmaLink-AI failover stack, giving you 36-engine routing under the same voice loop. The shared SSE reader handles the wire format.

The default Whisper.cpp adapter surfaces window-level partials, which is enough for the barge-in path. If you need finer granularity, wire the Deepgram streaming SDK and set STT_PROVIDER=deepgram. The interface is the same.