Vision and Tools

The ElevenLabs agent drives the Teams side through client tools. Define these on the agent (Agent > Tools > Client tools); the bridge maps each one onto a Teams capability.

look - see the caller’s camera or screen

Define a client tool named look with optional parameters source (camera or screenshare) and question. When the agent calls it, the bridge takes the latest buffered frame and answers one of two ways.

Path 2 - describe (preferred, if VISION_API_URL/VISION_MODEL are set)

The frame goes to your OpenAI-compatible vision endpoint (or your custom VisionDescriber) and the text description comes back as the tool result. The raw frame never leaves the bridge - only the description does - and it works regardless of recording state.

Caution

Know the data flow: the description becomes ElevenLabs conversation content, which ElevenLabs persists by default. So a description of the caller’s screen or camera can be stored by a third party even when Teams recording is off. This is a deliberate choice so vision stays usable without recording. If your deployment needs “no vision until recording is on,” enable ElevenLabs zero-retention on the agent, or leave VISION_API_URL unset so only the recording-gated path 1 is available.

Path 1 - upload (fallback)

If no vision endpoint is configured, the frame is uploaded to the live ElevenLabs conversation and injected as a multimodal_message (the agent’s LLM must be multimodal). Because this persists the raw frame with a third party, it is refused unless Teams recording is active.

Ambient vision - see without being asked (opt-in)

look requires the agent to decide to look. Ambient vision is the standing half of the same capability: with AMBIENT_VISION_ENABLED=true, every change to the caller’s screen-share or camera is described once and handed to the agent as a labelled, non-interrupting contextual update:

[ambient vision] Sara's shared screen: A pricing table with three tiers; the middle column is highlighted.

Screen-share is described before camera, so under a tight budget the last slot goes to the thing the caller is actually pointing at. The label degrades to the source kind (a shared screen, a camera) when the worker does not name the participant - it is never dropped, because a description with no owner is unusable in a meeting.

It never makes the agent speak

That is the design constraint the delivery route follows from. multimodal_message (path 1) and user_message are both interrupting user turns, so an ambient frame delivered either way would make the agent start talking every time the caller scrolled. A contextual_update does not, which is why ambient vision sends a description rather than the frame - and why it requires a describer (VISION_API_URL + VISION_MODEL, or a VisionDescriber passed to startServer). Enabled without one, the bridge logs a warning once per call and stays inactive.

What it costs, and what it does not

• A frozen screen costs nothing. Each source has its own change latch, so unchanged bytes are skipped.
• A failure costs nothing either. The latch is set only after a delivery succeeds; a failing vision endpoint refunds the budget slot and the frame stays retryable, instead of being marked “already sent” and lost.
• MAX_VISION_PER_MINUTE (default 30) is a sliding 60-second cap for the whole call, shared with look so an ambient burst cannot starve the tool the agent explicitly invoked.
• AMBIENT_VISION_REQUIRE_RECORDING (default true) holds pushes back until Teams recording is active.

Scene change is the trigger; a 6-second poll is only a backstop for a change that landed while a description was in flight.

show_image - put an image on the bot’s tile

Parameters: either inline {dataBase64, mime} or {url} (jpeg/png). The bridge sends a display.image to the Teams side.

Caution

A url is agent-controlled, i.e. indirectly caller-controlled. The bridge SSRF-guards it: public hosts only, no redirects (closing a DNS-rebind bypass), and bounded fetch time and size.

express - avatar emotion

Parameter: {emotion}. The bridge forwards an expression cue so the bot’s avatar reflects the agent’s sentiment.

end_call - hang up

The agent decides the call is done. The bridge acknowledges the tool, sends session.end to StandIn, and tears down both sockets.

Group-call awareness (no tool needed)

The bridge feeds the agent non-interrupting context automatically: participant counts, DTMF key presses, and - in group calls - a rate-limited note when the active speaker changes.

In a call with two or more humans the participant line also carries an etiquette clause naming the agent’s wake phrases, and the bridge enforces it at the audio egress: the reply is withheld until someone addresses the agent. See Group calls.