Architecture

The bridge is a small, stateless-per-call relay. It holds two WebSockets per call - the StandIn media bridge on one side, an OpenAI Realtime session on the other - and mostly copies bytes between them.

System overview

flowchart LR
    Teams["Microsoft Teams<br/>voice/video call"]
    StandIn["StandIn media bridge<br/>(hosted)<br/>joins the call, speaks<br/>the wire protocol"]
    Bridge["this bridge (Node/TS)<br/>verifies HMAC, terminates<br/>the wire protocol,<br/>one WS per call"]
    OAI["OpenAI Realtime API<br/>speech-to-speech<br/>one session per call"]

    Teams <--> StandIn
    StandIn -- "HMAC WebSocket, 16 kHz" --> Bridge
    Bridge -- "Realtime WebSocket, 24 kHz" --> OAI
    Bridge -. "relay audio, map barge-in,<br/>inject context, run tools<br/>and governors" .-> OAI

The StandIn media bridge handles everything about Teams itself and exposes each call as a single WebSocket carrying audio.frame (PCM 16 kHz), video.frame (JPEG) and control messages. This bridge has no idea what is on the other end of the Teams call - it only speaks the wire protocol.

The 16k/24k audio boundary

The Teams wire is base64 PCM 16 kHz, 16-bit, mono. The Realtime API accepts and emits PCM at 24 kHz only. All conversion lives in the provider adapter (openai.ts / resample.ts), so the relay core (session.ts) stays copy-only at the wire rate:

• Upsampling (16k to 24k, caller audio): linear interpolation on the clean 2:3 ratio.
• Downsampling (24k to 16k, agent audio): a 31-tap windowed-sinc low-pass at the 8 kHz target Nyquist first (so 8-12 kHz content cannot alias into the audible band), then decimation.

Dependency-free, a fraction of one audio frame of CPU per frame, telephony-adequate for voice - a real (if mild) resample, not bit-transparent.

Call lifecycle

flowchart TD
    A["upgrade + HMAC verify<br/>401 on bad / replayed / stale signature<br/>409 if callId already live"]
    B["session.start<br/>callId cross-checked vs the URL<br/>10s pre-start timeout if it never arrives"]
    C["connect OpenAI Realtime<br/>session.update: instructions + caller context,<br/>voice, VAD, PCM 24k, tools<br/>caller audio buffered ~5s while connecting"]
    D["relay<br/>audio.frame &lt;-&gt; input_audio_buffer.append / output_audio.delta<br/>speech_started -&gt; assistant.cancel + ghost-drop<br/>participants / dtmf -&gt; system context items<br/>look / show_image / express / end_call / custom tools"]
    E["teardown<br/>worker close, agent close, session.end,<br/>governor, 60-min API ceiling, or drain<br/>closes BOTH sockets once, clears timers,<br/>records call duration, de-registers the call"]

    A --> B --> C --> D --> E

Barge-in: the Realtime API cancels its own in-flight response when the caller starts speaking (interrupt_response); the bridge mirrors the cut to the Teams side with assistant.cancel and drops in-flight audio deltas from the cancelled response id. A second, unbounded-safe filter drops deltas from any response that is no longer the current one, so stale audio can never resurrect even on extremely bargey calls.

Source module map

Module	Responsibility
src/server.ts	HTTP server + WS upgrade, HMAC validation, connection guards (caps, replay, pre-start, dup-callId 409), session registry, tool registry wiring, opt-in signal drain
src/session.ts	One call: the StandIn WS ⇄ Realtime WS relay, ghost-drop, governors, goodbye, tool dispatch (built-in + custom), speaker attribution, vision buffering
src/openai.ts	Realtime socket, session-config builders, tool schemas, goodbye TTS, the 16k/24k conversion boundary, MCP entry normalization
src/resample.ts	Dependency-free PCM16 16k/24k resampler with anti-aliasing low-pass
src/protocol.ts	Wire message types (JSON, camelCase, discriminated on type) + PCM duration helper
src/hmac.ts	HMAC-SHA256("{timestampMs}.{callId}") sign/verify (constant-time), header names, freshness
src/ssrf.ts	Public-URL guard for the agent-supplied show_image fetch (one re-validated redirect hop)
src/vision.ts	Path-2 describe-then-answer vision hook (OpenAI-compatible endpoint)
src/config.ts	Env config, fail-loud numeric parsing, OPENAI_HOST allowlist, MCP/vision-URL validation
src/cli.ts	CLI entry point + friendly startup errors
src/log.ts	Minimal leveled logger

Trust and security model

Layer	Protection
Upgrade auth	HMAC-SHA256("{timestampMs}.{callId}"), constant-time compare, fails closed when the secret is unset
Replay	Single-use (callId, ts, sig) guard within a 60 s freshness window
Duplicate call	A second live connection for the same callId is rejected (409) - no second billed Realtime session
DoS	Max connections (64), per-IP cap (default = total cap), 2 MB inbound frame cap, 1 MB outbound backpressure cap (audio frames only - control frames and one-shot images always pass), 10 s pre-start timeout, 90 s dead-peer window
Key hygiene	OPENAI_API_KEY is server-side only, never sent to the Teams side; OPENAI_HOST is pinned to *.openai.com so the key cannot be exfiltrated to another host
SSRF	The agent-supplied show_image URL is resolved to public hosts only, connect-time DNS pinned against rebind, at most one re-validated redirect, bounded time and size
Tool bounds	Agent-supplied strings relayed to the worker are length-capped; unknown tools get an error result, never a crash
Crash safety	Every async entry point is guarded so a single malformed frame or throwing handler cannot take the process down
Shutdown	The opt-in signal drain (CLI) ends live calls (session.end + close) instead of hard-dropping them