Realtime Speech to Speech

One WebSocket carries the whole conversation: you stream the caller's microphone up, the engine streams the agent's voice back, and small JSON messages control language, voice, grounding and turn-taking. This is the same engine behind the Voice Agent demo and the platform's Live talk studio.

Connection

wss://platform.oogam.ai/v1/realtime — three ways to authenticate:

Audio format

Message protocol

Everything that is not binary audio is a one-line JSON object.

You send

You receive

Recommended call flow

1. Connect (auth above).
2. set_lang → set_voice → wait for voice_ack.
3. set_context with your system prompt (optional but what makes it your agent).
4. say a greeting so the agent opens the call.
5. Stream mic frames continuously; play reply frames as they arrive.
6. On user interruption, send stop_utterance and keep streaming.

Voices & languages for realtime

The realtime engine uses short voice ids — f1–f4 / m1–m4 for presets, and a clone's live_voice_id (a v_… value). To discover what a key may use:

• The REST GET /voices catalog lists each preset and every clone; a clone that can carry a live call carries a live_voice_id — that is the value you pass to set_voice. Clones marked realtime_capable: false are batch-only.
• For the realtime-native catalog (short codes and per-tenant clones) the platform also serves GET /api/naad/sts/voices and GET /api/naad/sts/languages, which the Live talk studio uses to build its pickers.
• Language uses the short code (hi, gu, ta…) — the "Live talk" column in the language table. Send it in set_lang before set_voice.

Function calling (preview)

Every session receives, as its first frame, a capability advertisement:

{"type":"session.updated","capabilities":{"function_calling":true,"parallel_tool_calls":false,"max_tools":20,"max_calls_per_turn":10}}

Gate tool registration on capabilities.function_calling — while it is false the model never requests a tool run, so register tools only when it reads true (no client update needed; the flag is the signal). Read the values you are sent rather than the ones printed above: the advertisement is resolved per session from what the gateway and the model can both actually serve, so it can differ from this example and can change between releases.

parallel_tool_calls is false today, deliberately: the model runs tools strictly one at a time. The correlation rules below already allow answering several calls in any order, so a client written to them needs no change if this becomes true — but do not design around concurrency you are currently told you do not have. It will not change silently.

What works today:

• {"type":"set_tools","tools":[…]} — declare tools in OpenAI function-schema format, verbatim. Each declaration replaces the previous set (never appends); tools: [] clears it; sent mid-turn it applies from the next turn. Optional fields: tool_choice (auto · none · a named function), filler, timeout_ms (1,000–15,000, default 5,000), max_calls_per_turn (1–10, default 3).
• tools_ack — one per declaration. It echoes the effective config, so a clamped value is visible rather than silent (ask for timeout_ms: 60000 and the ack tells you it became 15,000):
  jsonCopy

  {"type":"tools_ack","count":2,"names":["lookup_order","end_call"],"applied_from_turn":"turn_7","tool_choice":"auto","filler":"auto","timeout_ms":5000,"max_calls_per_turn":3}

  On a validation failure you get an error event instead and the previous tool set survives intact — a bad declaration never leaves the session with no tools.
• Limits: 20 tools per session, 32 KB serialized declaration, tool results 16 KB.

The call → result loop

Once function_calling is true, the model emits a tool_call whenever it wants one of your declared functions run. arguments is always a parsed JSON object, never a JSON string — if the engine cannot produce a valid object the call is rejected instead of reaching you half-formed.

{"type":"tool_call","call_id":"call_9f2c1ab4","name":"lookup_order","arguments":{"order_id":"A-1"},"turn_id":"turn_7"}

Run the function and reply with the result:

{"type":"tool_result","call_id":"call_9f2c1ab4","content":"Order A-1 shipped on 12 Aug.","is_error":false}

• call_id is opaque — echo it, never parse it. It is minted by Oogam, unique within a session, at most 64 characters, and is the only correlation key. Echo it back verbatim; do not parse, re-order or re-mint it, and never match it against a pattern. Today it happens to look like call_ plus 8 lowercase hex characters, which is useful when reading logs, but that shape is an observation and not a promise: a client that pattern-matches it will break the day the format widens, and gains nothing in the meantime. Correlate by equality only.
• Correlation is order-independent — guaranteed. With parallel_tool_calls, a single turn may open several calls at once. Answer them in any order, including completely reversed; each tool_result is matched on its call_id alone. Arrival order carries no meaning and is never used to pair a result with a call.
• content must be a string (serialize JSON yourself) of at most 3,000 Unicode code points, and 16 KB outright. Oversized results are rejected, never truncated — a clipped retrieval silently yields a confidently wrong answer on a live call. The 3,000 figure is the model’s own limit on the decoded string, so it counts Devanagari and emoji the same as ASCII.
• tool_notice is an advisory frame the model may send about a call in flight — most usefully to report that it clipped something. It carries a call_id and a reason, is relayed to you byte-for-byte, and requires no reply. Log it: it is how you learn a result did not survive intact.
• Set is_error: true to tell the model the function failed; it will apologise and move on rather than inventing a result.
• Answer within timeout_ms (default 5,000). After that the call is abandoned and a late result is rejected with call_already_timed_out. Sending stop_utterance (barge-in) or reset cancels every outstanding call in the abandoned turn — late results for those get the same code, and are never injected into the turn that replaced them.
• max_calls_per_turn is a budget per conversational turn, not per session. It refreshes at every turn boundary — the agent finishing its reply, a barge-in, or a reset.

Rejection taxonomy

Every rejection is an explicit error event — nothing is ever silently dropped. Shape: {"type":"error","code":"…","message":"…"}.

unknown_tool, max_calls_exceeded, invalid_tool_call and internal_error are not your client's fault — you receive them for visibility, but there is nothing on your side to fix. The rest are yours to correct.

Node.js example

import WebSocket from "ws";

const ws = new WebSocket("wss://platform.oogam.ai/v1/realtime", {
  headers: { Authorization: `Bearer ${process.env.OOGAM_API_KEY}` },
});

ws.on("open", () => {
  // 1. Pin language and voice, then wait for voice_ack.
  ws.send(JSON.stringify({ type: "set_lang", lang: "hi" }));
  ws.send(JSON.stringify({ type: "set_voice", voice: "f1" }));
});

ws.on("message", (data, isBinary) => {
  if (isBinary) {
    playPcm16(data);            // 24 kHz mono PCM16 reply audio
    return;
  }
  const msg = JSON.parse(data.toString());
  if (msg.type === "voice_ack") {
    // 2. Optional grounding, then speak an opening line.
    ws.send(JSON.stringify({ type: "set_context", system: SYSTEM_PROMPT }));
    ws.send(JSON.stringify({ type: "say", text: "नमस्ते! मैं आपकी क्या मदद करूँ?" }));
  }
  if (msg.type === "state") console.log("engine:", msg.state);
  if (msg.type === "reply") console.log("agent said:", msg.text);
});

// 3. Stream the caller's mic continuously: 16 kHz mono PCM16 binary frames.
micStream.on("frame", (pcm16Buffer) => ws.send(pcm16Buffer));

Browser connection

// Browsers cannot set an Authorization header on a WebSocket.
// Servers: keep your key out of the page — mint a short-lived session
// token server-side and pass that instead.
const ws = new WebSocket(
  "wss://platform.oogam.ai/v1/realtime?api_key=sk-setu-...");        // server-side tools only
const ws2 = new WebSocket(
  "wss://platform.oogam.ai/v1/realtime?session_token=" + token);      // browser apps (recommended)

Limits, billing & disconnects

• Billed per minute from your project wallet while connected.
• Concurrency: 5 sessions per API key, 2 Live talk calls per user; max session length 120 minutes (closes with code 1000). Contact us to raise limits.
• Per-language capacity: concurrent calls are also admitted per language (Hindi has the most headroom). At capacity, set_lang answers error: language_at_capacity — back off and retry shortly rather than hammering.
• Voice ids are validated: builtins are exactly f1–f4 / m1–m4; anything else that is not a clone's v_… id is rejected with invalid_voice instead of silently substituting a default voice.
• Close code 4402 means the wallet ran out mid-call — top up and reconnect.
• Close code 1011 ("Realtime service unavailable — try again") is transient and retryable: reconnect with 1s → 2s → 4s backoff; it recovers within a minute in practice. Keep the telephony leg open and play a short hold prompt while the engine leg reconnects.
• On any drop, reconnect and replay steps 2–4 — session state does not survive a reconnect; the engine keeps no conversation memory across connections.