Vision and Tools

The agent drives the Teams side through function tools. Unlike agent platforms where you define tools in a dashboard, the bridge registers these automatically in every session’s configuration - there is nothing to set up on the OpenAI side. The model calls a tool, the bridge executes it and returns the result as function_call_output.

To add your own tools (or remote MCP servers), see Extending the Agent’s Tools.

look - see the caller’s camera or screen

Parameters: optional source (camera or screenshare) and question. When the agent calls it, the bridge takes the latest buffered video frame and answers one of two ways.

Path 2 - describe (preferred, if VISION_API_URL/VISION_MODEL are set)

The frame goes to your OpenAI-compatible vision endpoint (or your custom VisionDescriber) and the text description comes back as the tool result. The raw frame never leaves the bridge - only the description does - and it works regardless of recording state.

Caution

Know the data flow: the description becomes Realtime conversation content, retained per your OpenAI data controls. This is a deliberate choice so vision stays usable without recording. If your deployment needs “no vision until recording is on,” leave VISION_API_URL unset so only the recording-gated path 1 is available.

Path 1 - native image input (fallback)

If no vision endpoint is configured, the frame is attached to the live Realtime conversation as native image input (gpt-realtime is multimodal) together with the question, and the model answers from the image directly. Because this shares the raw frame with the conversation, it is refused unless Teams recording is active.

show_image - put an image on the bot’s tile

Parameters: either inline {dataBase64, mime} or {url} (jpeg/png), plus optional caption and durationMs. The bridge sends a display.image to the Teams side.

Caution

A url is agent-controlled, i.e. indirectly caller-controlled. The bridge SSRF-guards it: public hosts only, connect-time DNS pinned against rebind, at most one redirect hop (the redirect target is re-validated through the same guard - image CDNs commonly 302 to the asset), and bounded fetch time and size (5 MB).

express - avatar emotion

Parameter: {emotion}. The bridge forwards an expression cue so the bot’s avatar reflects the agent’s sentiment.

end_call - hang up

The agent decides the call is done (the caller said goodbye, or asked to hang up). The bridge acknowledges the tool, sends session.end to StandIn, and tears down both sockets.

Group-call awareness (no tool needed)

The bridge feeds the agent non-interrupting context automatically, as system items that do not trigger a response: participant counts (“N humans on the call, stay quiet unless directly addressed”), DTMF key presses, and - in group calls - a rate-limited note when the active speaker changes. This helps the agent stay quiet in meetings until addressed.