Features

A tour of what the plugin can do on a Teams call. Everything here is implemented in this repo; the hosted StandIn media bridge handles the Teams media so these features “just work” once you’re connected.

Dialogue

Realtime speech-to-speech (--handler realtime) - OpenAI/Azure Realtime; low latency, full-duplex feel.
Streaming STT → agent → TTS (--handler streaming) - half-duplex, works with any STT/TTS provider (needs ffmpeg). Note: the visual call tools (show_file, show_web_page, walkthrough, progress panel, set_call_language) are realtime-mode features; streaming mode covers dialogue, vision auto-attach, and minutes.
Gateway-resident mode (phase 2b) - enable the msteams_bridge platform in the gateway and hermes gateway run hosts the voice bridge itself: one process, streaming utterances become real gateway agent turns (sessions, authorization, approvals), and cron jobs can deliver=msteams_bridge - the bot calls you and speaks the result. hermes msteams-bridge serve remains the standalone fallback.
Barge-in - the caller can interrupt the bot mid-reply; playback is flushed (assistant.cancel) and the model response is cancelled immediately.
Verbal interrupts (EN/AR/FR/DE) - deterministic “stop” / “توقف” / “arrête” / “stopp” / “⟨name⟩, stop” detection that cuts playback even without VAD.
Recording gate - unless require_recording_status is off, no media-derived data is processed until Teams recording is active. Greetings fire on answer, not while ringing.
Echo guard - suppresses the bot hearing its own output (the self-answer fix), using an RMS + playout-clock heuristic.
Greet on answer - the caller is greeted by first name once they answer.

Group / meeting etiquette

Speak only when addressed - in a call with 2+ humans, the agent stays silent unless someone addresses it by a wake phrase (wake_phrases, default assistant, hermes), then a short follow-up window lets the exchange continue without repeating the name. 1:1 calls always respond.
Race-free on realtime - server-VAD auto-response stays OFF until a 1:1 is confirmed (via the participants count), so no audio can leak into a meeting before the gate decides. Addressed turns are triggered manually.
Response-active reset - a rejected/failed model response clears the response-active latch so the next turn can always speak (no permanent muting).
Per-speaker attribution - unmixed-audio speakerName attributes each turn in the minutes.

Vision

look_at_screen - the agent looks at the caller’s shared screen or camera to answer a question; scope: "live" (current frame) or scope: "history" (recent keyframes, a 16-frame ring, to answer about something shown earlier).
Continuous / ambient vision (realtime) - the latest changed frame per source is pushed to the model about every 6 s (no forced response), so the model stays visually aware between explicit looks.
Per-call vision budget - max_vision_per_minute (default 30) caps spend across look_at_screen + ambient push; over budget, ambient pushes back off.

Avatar rendering cues

Expression - a cheap lexical classifier infers neutral / happy / sad / surprised from the reply text and sends an expression cue; a thinking face shows while a tool runs.
Visemes - a viseme timeline (speech.marks) drives lip-sync. Timing is a provider-agnostic registry: any timing-capable TTS provider can supply real per-character timing (ElevenLabs /with-timestamps is the built-in one, used only when it is the operator’s selected tts.provider); every other provider gets estimator visemes.
show_to_caller - generate an image and render it on the bot’s own video tile (display.image), fullscreen with an optional caption and a paced, interruption-aware slideshow.

Realtime tools

The realtime model is given these function tools (dispatched by the handler):

Tool	What it does
`hermes_agent_consult`	Delegate a question/action to the Hermes agent inline; returns a short spoken result.
`hermes_agent_task`	Run a long background job; acknowledge now, deliver the result to the Teams chat (call-back as fallback), durable across restarts.
`look_at_screen`	Look at the shared screen/camera (live or history) and answer.
`show_to_caller`	Generate an image and show it on the bot’s tile.
`show_file`	Display a real workspace file (image / PDF page / Office page) on the tile - workspace-contained, fullscreen.
`show_web_page`	Browse a public URL with the host’s browser and show the actual page on the tile.
`walkthrough`	Step-by-step guided tour: show a file per step, speak the explanation, stop on interruption.
`call_me_back`	Place an outbound Teams call back to deliver a pending result.
`post_meeting_minutes`	Summarize the meeting and post minutes to the Teams chat.

Meetings & productivity

End-of-call recap (meeting_recap) - post minutes (key points, decisions, action items) to the Teams chat when the call ends.
On-demand minutes - post_meeting_minutes or “summarize the meeting” posts minutes mid-call.
.docx file card - the minutes are attached to the Teams chat as a Word document (delivered over the same Bot Framework attachment contract the Hermes Teams adapter uses - needs TEAMS_CLIENT_ID/SECRET/TENANT_ID, which the chat plane already sets); text-only fallback when unavailable. A Word-openable copy is always saved under the Hermes workspace (workspace/teams_minutes/). Inline attachments cap at 4 MB; share_point_site_id is reserved for a future large-file SharePoint path.
Chat-to-call (call_user agent tool) - ask in any Hermes chat surface (“call me and explain X”); the bot places the Teams call and speaks the message on answer. Deny-by-default: the callee must be on an explicit allowlist.
No-answer fallback - a placed call that never rings through gets its message posted to the originating Teams chat instead of being lost (applies to chat-to-call, call-backs, and background-task deliveries).
“Watch it work” - hand the agent a background task mid-call and the tile shows a live progress panel until the result is ready.
Chat/call parity - the call answers with the SAME identity (SOUL.md) and knows the SAME installed skills as Hermes chat, delegating skill work to the agent.

Chat lane (StandIn Managed Bot)

Voice messages (transcribe_voice_messages, off by default) - a Teams voice note sent to the bot is fetched, transcribed with your configured stt.provider, and the words go into the same agent turn as the message text, so “listen to this and tell me what you think” is a question the agent can answer rather than a filename it can only read back. Off by default because each clip is a paid STT call and a voice note can run for minutes. A clip that cannot be fetched or transcribed still reaches the agent as a placeholder telling it the clip could not be played, so a dropped voice note is never mistaken for an empty message. Fetches are pinned to the StandIn gateway origin, refuse redirects, cap at 16 MiB per clip, and stop after two clips per message.
Attachments - every attachment is named in the turn (kind, filename, and the signed URL), whether or not it could be relayed.
Card submits - an Action.Submit payload from a card the agent itself sent is folded into the turn, so a button press is a meaningful message.

Telephony & languages

DTMF / IVR - keypad presses are surfaced to the model so it can run “press 1 to…” flows.
Languages (languages: [en, fr, de, ar]) - the model replies in the caller’s language when it is configured (empty list = auto-detect and mirror); bilingual remains as a deprecated alias for [ar, en]. STT/TTS language selection follows the host’s stt:/tts: provider config.

Sessions

session_scope - memory continuity: per-call (fresh each call), per-thread (keyed by Teams thread), or per-aad (keyed by caller identity).

Reliability & safety

Cutoff goodbye - on a StandIn limit, the agent speaks a goodbye (assistant.say) before StandIn ends the call gracefully.
Provider-drop teardown - if the realtime provider is unreachable or drops mid-call, the Teams call is torn down cleanly instead of leaving dead air.
Max-duration reaper - max_call_duration_s bounds a call’s wall-clock time so a wedged call can’t run forever and leak a socket.
Caller allowlist - deny-by-default by AAD id (allowlist / allow_all; optional weaker display-name matching via allowlist_allow_names).
DoS guards - global + per-IP connection caps, a max frame size, the pre-start timeout, and the single-use HMAC replay guard.
Outbound SSRF guard - “call me back” refuses a non-loopback target unless allow_remote_worker is set (the shared secret would otherwise be sent to that host).