Live Voice Agent Latency & Extended Thinking Workbench

Benchmark streaming speech-to-speech architectures. Simulate reasoning token budgets, asynchronous tool calling, acoustic fillers, and user barge-in interruptions in real time.

Agent Architecture

CONFIG

Architecture Presets

Model Engine Mode Extended Thinking computes chain-of-thought tokens before or during tool invocation.

Extended Thinking Budget 1,200 tokens (380ms)

Higher budget verifies spatial safety / constraints before tool dispatch. Acoustic Filler Strategy Keeps the conversational floor during reasoning latencies > 350ms.

External Tool Latency (ms) 320 ms

Simulates database API, gripper controller, or vector retrieval round-trip.

Audio Chunk Size (Frames) 40 ms (24kHz)

Duplex streaming chunk duration transmitted over WebRTC / WebSocket.

Barge-in VAD Sensitivity 92%

Voice Activity Detection cut-off threshold for user interruptions. User Utterance Prompt

Web Audio Feedback

TTFA (Time to 1st Audio) 320 ms Immediate speech streaming

Extended Thinking Time 380 ms 1,200 tokens evaluated

Tool Round-Trip 320 ms Success (200 OK)

Interruption Latency — Floor clear

Duplex Audio & Thinking Stream (Time 0 - 3,500ms)

User Speech

Extended Thinking

Tool Call Dispatch

Agent Speech Stream

Streaming Audio Transcript READY

User: Move the robotic gripper to coordinate X:45, Y:12 and verify payload clearance. t = 0.00s

Gemini Live: [Filler] Scanning trajectory envelope now. Clearances verified at 14.2cm. Gripper positioned to X:45, Y:12. t = 1.08s

Extended Thinking & Tool Execution Log TRACE

💭 Reasoning Trace (1200 tokens): 1. Parse coordinates: X=45, Y=12. 2. Query collision mesh for table perimeter. 3. Clearance envelope check: 14.2cm safe margin. 4. Prepare dispatch signature for `robotics_kinematics_move`.

⚡ Tool Invocation: robotics_kinematics_move({x:45, y:12, safety_check:true}) ↳ Returned: {status: "ACK", arm_state: "LOCKED", latency: 320ms}

Ready. Click "Simulate Conversational Turn" to inspect live streaming telemetry.

Gemini 3.8 Live vs. Extended Thinking

Standard speech-to-speech models optimize purely for lowest Time-to-First-Audio (TTFA ~200-300ms), making them ideal for casual conversational banter. However, when complex robotics, multi-step customer workflows, or safety verifications are required, direct generation can hallucinate or dispatch premature physical actions.

Gemini 3.8 Live Extended Thinking introduces integrated chain-of-thought tokens inside the audio duplex session. The model can stream acoustic filler cues ("Checking clearances...") while synchronously executing hundreds of reasoning tokens to guarantee safety constraints before executing irreversible API calls.

Feature Dimension	Gemini 3.8 Live	Gemini 3.8 Live Ext. Thinking
Primary Optimization	Lowest turn latency (TTFA < 300ms)	Reasoning depth + physical accuracy
Tool Decision Timing	Immediate greedy prediction	Pre-call reasoning validation
Acoustic Management	Direct audio output stream	Smart fillers / prosodic hold during reasoning
Best Application	Creative voice, rapid chat, language practice	Robotics, medical triage, database mutations

Handling Barge-in & Conversational Floor

In human speech, conversational overlap is natural. A robust real-time agent must continuously monitor incoming audio frames even while synthesizing outgoing speech. The key engineering parameters modeled in this workbench include:

Voice Activity Detection (VAD) Sensitivity: Determines the decibel and acoustic feature threshold required to declare a user interruption.
Cancellation Latency: The duration between detecting user energy, canceling in-flight neural speech synthesis chunks, and purging local audio playback buffers (typically 60-120ms).
Tool Abort vs. Idempotency: If a user interrupts while extended thinking is resolving a tool call, the pipeline must decide whether to cancel the external mutation or let it complete asynchronously.

Experiment by triggering "User Barge-In" while the simulated agent is responding to test cancellation speed and transcript truncations.