Level: Intermediate · Env: Local machine or Google Cloud Shell with Node.js 18+ and TypeScript
A real-time, bidirectional conversational voice commerce shopping agent (Talk to Shop) powered by the Google Gemini Live API (gemini-3.1-flash-live-preview), Fastify, WebSockets, and client-side Web Audio API.
Instead of a sluggish cascading voice bot (Record → Speech-to-Text → LLM → Text-to-Speech) that dumps walls of text over synthesized audio, you will build an agent with Dual Delivery (simultaneous low-latency voice personality + synchronized visual product cards and cart state) and an Authoritative Backend that strictly governs catalog pricing, shipping estimates, and financial transactions.
flowchart LR
subgraph Client["Client Browser (Dual Delivery)"]
direction TB
MIC["Microphone (16kHz PCM Capture)"]
UI["Visual Shelf and Cart Drawer"]
SPK["Speakers (24kHz Scheduled Playback)"]
end
subgraph Backend["Fastify Proxy Server (:3000)"]
direction TB
WS["WebSocket Duplex Handler"]
AUTH["Authoritative Commerce Engine"]
WS --- AUTH
end
subgraph Cloud["Google Cloud"]
LIVE["Gemini Live API (gemini-3.1-flash-live-preview)"]
end
MIC -->|audio_chunk 16kHz PCM| WS
WS <-->|Duplex Audio and Tools| LIVE
WS -->|audio_chunk 24kHz PCM| SPK
WS -->|dual_delivery JSON| UI
By the end of this codelab, you will be able to:
Voice is transient and linear. While voice is unparalleled for expressing intent, nuance, and conversational guidance, it is terrible at conveying long lists of technical specifications, comparisons, or financial breakdowns.
Consider this comparison:
Attribute | Voice Channel | Visual Channel (Screen) |
Strengths | Empathy, high bandwidth for intent, hands-free convenience, natural guidance | High-density information, side-by-side spec comparison, transaction confirmation |
Weaknesses | Overwhelms working memory with lists; difficult to verify totals | Requires manual clicks; rigid search and filter menus |
Talk to Shop Solution | Concise Spoken Advice (2–3 sentences highlighting top trade-offs) | Dual Delivery Shelf & Cart (real-time product cards, spec table, price breakdown) |
flowchart TD
P1["1. Duplex Streaming WebSocket"] --> E1["About 1.5 seconds from end of speech to a spoken answer, including a tool call"]
P2["2. Dual Delivery UX"] --> E2["Parallel voice audio stream + WebSocket dual_delivery frames updating screen state"]
P3["3. Authoritative State Machine"] --> E3["LLM emits toolCall intents; backend validates pricing, stock, and orders"]
dual_delivery frame to the browser. The user hears the recommendation while viewing the exact photo, battery life, and price on their screen.node -v)Create a clean directory and initialize your project:
mkdir talk-to-shop-gemini-live
cd talk-to-shop-gemini-live
npm init -y
Replace your package.json with this configuration for ES module resolution and project scripts:
{
"name": "talk-to-shop-gemini-live",
"version": "1.0.0",
"description": "Talk to Shop: Voice-First E-Commerce Agent with Gemini Live API — Authoritative Node.js Backend & Dual-Delivery UI",
"type": "module",
"scripts": {
"dev": "tsx watch src/server.ts",
"build": "tsc",
"start": "node dist/server.js",
"tunnel": "cloudflared tunnel --url http://localhost:3000 || npx localtunnel --port 3000"
},
"dependencies": {
"@fastify/static": "^8.0.4",
"@fastify/websocket": "^11.0.2",
"@google/genai": "^3.0.0",
"dotenv": "^16.4.7",
"fastify": "^5.2.1",
"ws": "^8.18.0"
},
"devDependencies": {
"@types/node": "^22.10.7",
"@types/ws": "^8.5.14",
"tsx": "^4.19.2",
"typescript": "^5.7.3"
}
}
Install the dependencies:
npm install
Create a tsconfig.json:
{
"compilerOptions": {
"target": "ES2022",
"module": "NodeNext",
"moduleResolution": "NodeNext",
"esModuleInterop": true,
"strict": true,
"skipLibCheck": true,
"outDir": "./dist",
"rootDir": "./src"
},
"include": ["src/**/*"]
}
Create .env in the root of your project:
PORT=3000
GEMINI_API_KEY=YOUR_GEMINI_API_KEY
GEMINI_LIVE_MODEL=gemini-3.1-flash-live-preview
GEMINI_LIVE_VOICE=Aoede
This codelab focuses on the backend and the audio engine. The page layout and styling are plain HTML and CSS, so copy them from the solution repository, together with the complete app.js that we will dissect in later steps:
git clone --depth 1 https://github.com/adityajoshi12/ai-workshops.git /tmp/ai-workshops
mkdir -p public src
cp /tmp/ai-workshops/talk-to-shop-gemini-live/public/index.html public/
cp /tmp/ai-workshops/talk-to-shop-gemini-live/public/styles.css public/
cp /tmp/ai-workshops/talk-to-shop-gemini-live/public/app.js public/
Create src/commerce.ts. This module defines the types, the search helpers, the saved addresses, the cart, shipping rules, and order placement. We will build it in three parts. First, the types and helpers:
/**
* Authoritative Commerce Service & Data Layer
*
* Core Architectural Principle:
* The model NEVER directly accesses databases or decides prices, inventory, or order status.
* All operations execute through these validated, server-side functions.
*/
import { CATALOGUE } from './catalogue.js';
export interface Product {
id: string;
name: string;
category: string;
priceInr: number;
/** Omitted for products without a battery (e.g. security cameras). */
batteryHours?: number;
/** Audio products only. */
ancType?: string;
weightGrams: number;
/** Short, category-specific spec lines shown as pills (e.g. screen, storage, camera). */
highlights?: string[];
rating: number;
reviewsCount: number;
stock: number;
imageUrl: string;
description: string;
tags: string[];
}
export interface CartItem {
product: Product;
quantity: number;
itemSubtotalInr: number;
}
export interface Cart {
items: CartItem[];
itemCount: number;
subtotalInr: number;
gstInr: number;
shippingInr: number;
totalInr: number;
}
export interface Address {
id: string;
label: string;
recipient: string;
street: string;
city: string;
state: string;
pincode: string;
isDefault: boolean;
}
export interface Order {
orderId: string;
items: CartItem[];
totalInr: number;
shippingAddress: Address;
paymentMethod: string;
status: 'confirmed' | 'processing' | 'shipped';
estimatedDeliveryDate: string;
createdAt: string;
}
// ----------------------------------------------------------------------------
// Search helpers
// ----------------------------------------------------------------------------
const STOPWORDS = new Set(['for', 'the', 'and', 'with', 'under', 'best', 'some', 'any']);
/** Lowercase keywords with trailing plural 's' dropped, so "smartwatches" matches "smartwatch". */
function tokenize(query?: string): string[] {
return (query || '')
.toLowerCase()
.split(/[^a-z0-9]+/)
.filter((t) => t.length > 2 && !STOPWORDS.has(t))
.map((t) => t.replace(/s$/, ''));
}
/** Everything a keyword can match against, built once per product. */
const haystackCache = new Map<string, string>();
function haystack(p: Product): string {
let h = haystackCache.get(p.id);
if (!h) {
h = [p.name, p.category.replace(/_/g, ' '), p.description, ...p.tags, ...(p.highlights ?? [])].join(' ').toLowerCase();
haystackCache.set(p.id, h);
}
return h;
}
// ----------------------------------------------------------------------------
// Authoritative Customer Saved Addresses
// ----------------------------------------------------------------------------
export const SAVED_ADDRESSES: Address[] = [
{
id: 'addr_blr_home',
label: 'Home',
recipient: 'Aditya Joshi',
street: 'Flat 402, Green Glen Heights, Bellandur',
city: 'Bengaluru',
state: 'Karnataka',
pincode: '560103',
isDefault: true,
},
{
id: 'addr_mum_studio',
label: 'Orbit Studio',
recipient: 'Aditya Joshi',
street: 'Unit 12, Andheri Kurla Road, Marol',
city: 'Mumbai',
state: 'Maharashtra',
pincode: '400059',
isDefault: false,
},
];
A few design choices are worth pausing on:
batteryHours and ancType are optional. A power bank has no battery runtime and a laptop has no noise cancellation, so the model must not be forced to invent them.highlights holds short, category-specific spec lines (screen, storage, camera) that the UI shows as pills.tokenize() turns "gaming laptops" into the keywords gaming and laptop (plural dropped), and every keyword must appear somewhere in the product. A whole-string substring match would only find products containing that exact phrase.Create src/catalogue.ts. It is pure data with no business logic, so adding a category never touches the engine. The placeholder art is generated, so the demo has no image hosting dependency. Three products reuse real photos and four new ones span smartphones, smartwatches, speakers, and power banks:
/**
* Seeded Authoritative Product Catalogue
*
* Pure data: no business logic lives here. To add a category, just add products;
* the `category` enum exposed to Gemini (tools.ts) is derived from this file.
*/
import type { Product } from './commerce.js';
// Generated placeholder art (no network, never 404s). Swap `imageUrl` for real photos any time.
const CATEGORY_ART: Record<string, [emoji: string, from: string, to: string]> = {
smartphones: ['\u{1F4F1}', '#4f46e5', '#7c3aed'],
smartwatches: ['\u{231A}', '#be185d', '#e11d48'],
bluetooth_speakers: ['\u{1F50A}', '#b45309', '#ea580c'],
power_banks: ['\u{1F50B}', '#15803d', '#65a30d'],
};
function artFor(category: string): string {
const [emoji, from, to] = CATEGORY_ART[category];
const svg =
`<svg xmlns='http://www.w3.org/2000/svg' width='500' height='375'>` +
`<defs><linearGradient id='g' x1='0' y1='0' x2='1' y2='1'>` +
`<stop offset='0' stop-color='${from}'/><stop offset='1' stop-color='${to}'/></linearGradient></defs>` +
`<rect width='500' height='375' fill='url(#g)'/>` +
`<text x='250' y='215' font-size='140' text-anchor='middle'>${emoji}</text></svg>`;
return `data:image/svg+xml,${encodeURIComponent(svg)}`;
}
type NewProduct = Omit<Product, 'imageUrl'>;
/** Builds a catalogue entry and attaches the category art. */
function item(p: NewProduct): [string, Product] {
return [p.id, { ...p, imageUrl: artFor(p.category) }];
}
// Original audio range (real photos).
const AUDIO_CATALOGUE: Record<string, Product> = {
prod_tb_x: {
id: 'prod_tb_x',
name: 'TravelBeats X',
category: 'wireless_headphones',
priceInr: 2799,
batteryHours: 50,
ancType: 'Hybrid ANC',
weightGrams: 255,
rating: 4.8,
reviewsCount: 1420,
stock: 8,
imageUrl: 'https://images.unsplash.com/photo-1505740420928-5e560c06d30e?w=500&q=80',
description: 'Foldable over-ear headphones engineered for frequent travellers with 50-hour playback and deep active noise cancellation.',
tags: ['travel', 'wireless', 'headphones', 'long-battery', 'anc', 'budget', 'airplane'],
},
prod_sw_pro: {
id: 'prod_sw_pro',
name: 'SoundWave Pro',
category: 'wireless_headphones',
priceInr: 2499,
batteryHours: 40,
ancType: 'Adaptive ANC',
weightGrams: 240,
rating: 4.6,
reviewsCount: 980,
stock: 14,
imageUrl: 'https://images.unsplash.com/photo-1583394838336-acd977736f90?w=500&q=80',
description: 'Lightweight ergonomic travel headphones featuring adaptive noise cancellation and Bluetooth 5.3 multipoint pairing.',
tags: ['travel', 'wireless', 'headphones', 'multipoint', 'anc', 'budget'],
},
prod_ab_active: {
id: 'prod_ab_active',
name: 'AeroBuds Active',
category: 'tws_earbuds',
priceInr: 2299,
batteryHours: 36,
ancType: 'Environmental Noise Cancellation (ENC)',
weightGrams: 45,
rating: 4.5,
reviewsCount: 820,
stock: 18,
imageUrl: 'https://images.unsplash.com/photo-1590658268037-6bf12165a8df?w=500&q=80',
description: 'Pocket-sized True Wireless Earbuds with IPX5 sweat resistance, low-latency gaming mode, and 36-hour total battery with case.',
tags: ['wireless', 'earbuds', 'workout', 'commute', 'budget'],
},
};
// Wider electronics range.
const ELECTRONICS: NewProduct[] = [
{
id: 'prod_ph_nova_x5', name: 'Nova X5 5G', category: 'smartphones', priceInr: 17999,
batteryHours: 22, weightGrams: 188, rating: 4.4, reviewsCount: 2310, stock: 12,
description: 'Mid-range 5G phone with a bright 120Hz AMOLED display, 50MP OIS camera and 33W fast charging.',
tags: ['smartphone', 'phone', '5g', 'camera', 'midrange', 'amoled'],
highlights: ['6.67" AMOLED 120Hz', '8GB + 128GB', '50MP OIS camera'],
},
{
id: 'prod_wt_pulsefit_3', name: 'PulseFit Band 3', category: 'smartwatches', priceInr: 3499,
batteryHours: 336, weightGrams: 28, rating: 4.3, reviewsCount: 3890, stock: 40,
description: 'Slim fitness band with an AMOLED screen, 24x7 heart rate and SpO2 tracking, and a 14-day battery.',
tags: ['smartwatch', 'watch', 'band', 'fitness', 'budget', 'long-battery', 'workout'],
highlights: ['1.7" AMOLED', 'Heart rate + SpO2', '5ATM water resistant'],
},
{
id: 'prod_sp_boombox_mini', name: 'BoomBox Mini', category: 'bluetooth_speakers', priceInr: 2999,
batteryHours: 12, weightGrams: 540, rating: 4.5, reviewsCount: 2750, stock: 22,
description: 'Rugged pocket-friendly speaker with punchy stereo sound that survives rain, dust and pool splashes.',
tags: ['speaker', 'bluetooth', 'portable', 'waterproof', 'outdoor', 'budget'],
highlights: ['IP67 waterproof', '20W stereo', 'Pair two for party mode'],
},
{
id: 'prod_pb_voltmax_20k', name: 'VoltMax 20000', category: 'power_banks', priceInr: 2199,
weightGrams: 430, rating: 4.4, reviewsCount: 4300, stock: 35,
description: 'High-capacity power bank with 65W USB-C Power Delivery, enough to charge a laptop or a phone several times.',
tags: ['powerbank', 'power bank', 'charger', 'travel', 'fast-charging', 'usb-c', 'budget'],
highlights: ['20,000 mAh', '65W USB-C PD', 'Charges laptops'],
},
];
export const CATALOGUE: Record<string, Product> = {
...AUDIO_CATALOGUE,
...Object.fromEntries(ELECTRONICS.map(item)),
};
Back in src/commerce.ts, append the store class with search, comparison, inventory, and the cart. Server-side validation (for example, stock limits) lives here and never in the prompt:
// ----------------------------------------------------------------------------
// Session Cart State Manager
// ----------------------------------------------------------------------------
class CommerceStore {
private cartItems: Map<string, number> = new Map();
private orders: Order[] = [];
// Reset store for fresh demo runs
public reset(): void {
this.cartItems.clear();
this.orders = [];
}
// 1. Search Products
public searchProducts(params: {
query?: string;
category?: string;
maxPriceInr?: number;
minBatteryHours?: number;
ancRequired?: boolean;
}): { count: number; products: Product[] } {
const tokens = tokenize(params.query);
const results = Object.values(CATALOGUE).filter((p) => {
if (params.category && p.category !== params.category) return false;
if (params.maxPriceInr && p.priceInr > params.maxPriceInr) return false;
if (params.minBatteryHours && (p.batteryHours ?? 0) < params.minBatteryHours) return false;
if (params.ancRequired && !(p.ancType ?? '').toLowerCase().includes('anc')) return false;
return tokens.every((t) => haystack(p).includes(t));
});
return {
count: results.length,
products: results,
};
}
// 2. Get Product Details
public getProductDetails(productId: string): Product | null {
return CATALOGUE[productId] || null;
}
// 3. Compare Products
public compareProducts(productIds: string[]): { compared: Product[]; diff: Record<string, unknown> } {
const products = productIds.map((id) => CATALOGUE[id]).filter(Boolean) as Product[];
return {
compared: products,
diff: {
priceSpreadInr: products.map((p) => ({ name: p.name, price: p.priceInr })),
batteryComparisonHours: products.map((p) => ({ name: p.name, batteryHours: p.batteryHours })),
noiseCancellation: products.map((p) => ({ name: p.name, anc: p.ancType })),
keySpecs: products.map((p) => ({ name: p.name, highlights: p.highlights })),
recommendation: products.length > 0 ? products[0].name : 'None',
},
};
}
// 4. Check Inventory
public checkInventory(productId: string, pincode?: string): { inStock: boolean; availableQty: number; warehouseCity: string; estimatedDays: number } {
const p = CATALOGUE[productId];
if (!p) {
return { inStock: false, availableQty: 0, warehouseCity: 'Unknown', estimatedDays: 0 };
}
const isLocal = pincode?.startsWith('560') || pincode?.startsWith('400');
return {
inStock: p.stock > 0,
availableQty: p.stock,
warehouseCity: 'Bengaluru Hub #4',
estimatedDays: isLocal ? 1 : 2,
};
}
// 5. Add to Cart (Authoritative server validation)
public addToCart(productId: string, quantity: number = 1): { success: boolean; cart: Cart; message: string } {
const product = CATALOGUE[productId];
if (!product) {
return { success: false, cart: this.getCart(), message: `Invalid product ID: ${productId}` };
}
const cur = this.cartItems.get(productId) || 0;
const requested = cur + Math.max(1, quantity);
if (requested > product.stock) {
return {
success: false,
cart: this.getCart(),
message: `Only ${product.stock} units of ${product.name} are available in stock.`,
};
}
this.cartItems.set(productId, requested);
return {
success: true,
cart: this.getCart(),
message: `Added ${quantity}x ${product.name} to cart at ₹${product.priceInr.toLocaleString('en-IN')}.`,
};
}
// 6. Get Cart
public getCart(): Cart {
const items: CartItem[] = [];
let subtotal = 0;
for (const [id, qty] of this.cartItems.entries()) {
const p = CATALOGUE[id];
if (p) {
const itemSub = p.priceInr * qty;
subtotal += itemSub;
items.push({
product: p,
quantity: qty,
itemSubtotalInr: itemSub,
});
}
}
const gst = Math.round(subtotal * 0.18);
const shipping = subtotal > 1500 || subtotal === 0 ? 0 : 99;
const total = subtotal + gst + shipping;
return {
items,
itemCount: items.reduce((acc, i) => acc + i.quantity, 0),
subtotalInr: subtotal,
gstInr: gst,
shippingInr: shipping,
totalInr: total,
};
}
Finally, append delivery, saved addresses, and order placement. Notice that placeOrder refuses to run unless userConfirmed is true, and it deducts stock only after a confirmed order:
// 7. Check Delivery
public checkDelivery(pincode: string): { pincode: string; isServiceable: boolean; shippingCostInr: number; estimatedDeliveryText: string } {
const cleanPin = (pincode || '560103').replace(/\D/g, '');
const isServiceable = cleanPin.length === 6;
return {
pincode: cleanPin,
isServiceable,
shippingCostInr: 0, // Free delivery for demo items
estimatedDeliveryText: cleanPin.startsWith('560') ? 'Tomorrow by 2:00 PM (Express Hub)' : 'Within 2 business days',
};
}
// 8. Get Saved Addresses
public getSavedAddresses(): Address[] {
return SAVED_ADDRESSES;
}
// 9. Place Order (Transaction Safety & Explicit Confirmation)
public placeOrder(params: {
addressId?: string;
paymentMethod?: string;
userConfirmed: boolean;
}): { success: boolean; order?: Order; message: string } {
if (!params.userConfirmed) {
return {
success: false,
message: 'Order was not placed: User explicit confirmation is strictly required prior to transaction execution.',
};
}
const cart = this.getCart();
if (cart.items.length === 0) {
return {
success: false,
message: 'Cannot place order: Shopping cart is currently empty.',
};
}
const addr = SAVED_ADDRESSES.find((a) => a.id === params.addressId) || SAVED_ADDRESSES[0];
const orderId = `ORD-2026-${Math.floor(100000 + Math.random() * 900000)}`;
// Deduct inventory
for (const item of cart.items) {
const p = CATALOGUE[item.product.id];
if (p) p.stock -= item.quantity;
}
const newOrder: Order = {
orderId,
items: [...cart.items],
totalInr: cart.totalInr,
shippingAddress: addr,
paymentMethod: params.paymentMethod || 'Google Pay (UPI Verified)',
status: 'confirmed',
estimatedDeliveryDate: 'Tomorrow, by 2:00 PM',
createdAt: new Date().toISOString(),
};
this.orders.push(newOrder);
this.cartItems.clear(); // Clear cart after successful transaction
return {
success: true,
order: newOrder,
message: `Order #${orderId} confirmed! Total ₹${newOrder.totalInr.toLocaleString('en-IN')} paid via ${newOrder.paymentMethod}. Arriving ${newOrder.estimatedDeliveryDate} at ${addr.street}, ${addr.city}.`,
};
}
}
export const commerce = new CommerceStore();
Create src/tools.ts. When opening a Gemini Live session, you pass function declarations inside the setup handshake. The model determines when to trigger tools and passes arguments conforming to JSON schemas.
/**
* Official Gemini Live API Tool Declarations
*
* Each function declaration strictly adheres to the @google/genai schema specification.
* Gemini uses these declarations to generate structured functionCall events during the live audio stream.
*/
import { CATALOGUE } from './catalogue.js';
// Derived from the catalogue so new categories are automatically visible to Gemini.
const CATEGORIES = [...new Set(Object.values(CATALOGUE).map((p) => p.category))];
export const COMMERCE_FUNCTION_DECLARATIONS = [
{
name: 'search_products',
description:
'Searches the electronics catalogue (audio, smartphones, laptops, smartwatches, speakers, tablets, cameras, gaming, power banks, smart home) ' +
'by keywords, category, budget, battery life and noise cancellation. Prefer `category` when the user names a product type.',
parameters: {
type: 'OBJECT',
properties: {
query: {
type: 'STRING',
description:
'Short keywords, ALL of which must match. ALWAYS include the product type the user asked for ' +
'(e.g. "headphones", "phone", "watch", "speaker") together with any qualifier: "travel headphones", "gaming laptop", "5g phone". ' +
'Never send a qualifier alone (e.g. just "travel"), it matches unrelated products.',
},
category: {
type: 'STRING',
description: 'Restrict results to one product category.',
enum: CATEGORIES,
},
maxPriceInr: {
type: 'NUMBER',
description: 'Maximum budget in Indian Rupees (INR), e.g. 3000.',
},
minBatteryHours: {
type: 'NUMBER',
description: 'Minimum required battery life in hours, e.g. 40. Excludes products with no battery.',
},
ancRequired: {
type: 'BOOLEAN',
description: 'Set to true if active noise cancellation is explicitly required by the user.',
},
},
},
},
{
name: 'get_product_details',
description: 'Retrieves comprehensive specifications, warranty, ratings, and customer reviews for a specific product ID.',
parameters: {
type: 'OBJECT',
properties: {
productId: {
type: 'STRING',
description: 'The unique product ID, e.g. "prod_tb_x", "prod_sw_pro".',
},
},
required: ['productId'],
},
},
{
name: 'compare_products',
description: 'Performs a side-by-side comparison between 2 or 3 products focusing on price, battery, noise cancellation, and weight.',
parameters: {
type: 'OBJECT',
properties: {
productIds: {
type: 'ARRAY',
description: 'Array of 2 to 3 product IDs to compare, e.g. ["prod_tb_x", "prod_sw_pro"].',
items: { type: 'STRING' },
},
},
required: ['productIds'],
},
},
{
name: 'check_inventory',
description: 'Checks real-time warehouse stock availability and delivery lead times for a product.',
parameters: {
type: 'OBJECT',
properties: {
productId: {
type: 'STRING',
description: 'The product ID to verify in stock.',
},
pincode: {
type: 'STRING',
description: 'Delivery destination postal code (6 digits), e.g. "560103".',
},
},
required: ['productId'],
},
},
{
name: 'add_to_cart',
description: 'Adds a specified quantity of a product to the customer\'s active shopping cart.',
parameters: {
type: 'OBJECT',
properties: {
productId: {
type: 'STRING',
description: 'The product ID to add to cart.',
},
quantity: {
type: 'INTEGER',
description: 'Number of units to add (default 1).',
},
},
required: ['productId'],
},
},
{
name: 'get_cart',
description: 'Retrieves the customer\'s current shopping cart, including all items, quantities, subtotal, 18% GST, and grand total.',
parameters: {
type: 'OBJECT',
properties: {},
},
},
{
name: 'check_delivery',
description: 'Calculates the estimated delivery date, express cutoff times, and shipping fee for a postal pincode.',
parameters: {
type: 'OBJECT',
properties: {
pincode: {
type: 'STRING',
description: 'The 6-digit Indian postal code to evaluate (e.g. "560103").',
},
},
required: ['pincode'],
},
},
{
name: 'get_saved_addresses',
description: 'Retrieves the customer\'s verified saved delivery addresses on file (Home, Office).',
parameters: {
type: 'OBJECT',
properties: {},
},
},
{
name: 'place_order',
description: 'Finalizes the purchase and creates a confirmed order record. CRITICAL: The assistant MUST obtain explicit verbal confirmation from the user (reading out item, address, and total) before invoking this function.',
parameters: {
type: 'OBJECT',
properties: {
addressId: {
type: 'STRING',
description: 'The selected delivery address ID, e.g. "addr_blr_home".',
},
paymentMethod: {
type: 'STRING',
description: 'Payment method, e.g. "Google Pay (UPI)" or "Saved Credit Card".',
},
userConfirmed: {
type: 'BOOLEAN',
description: 'Must be true only if the user explicitly said "yes", "confirm", or "place order" after hearing the total.',
},
},
required: ['userConfirmed'],
},
},
];
Two details here came from real testing, not theory:
category enum is derived from the catalogue (CATEGORIES), so it can never go out of date.query description tells the model to always include the product type. Once the catalogue held more than headphones, Gemini often sent just query: "travel" for "travel headphones under 3,000 rupees", which also matched a travel power bank. Telling it to send "travel headphones" fixed this: all 18 follow-up test calls included the product type. Tool descriptions are part of your prompt, so treat them that way.Create src/server.ts. This server serves the static front-end assets and maintains two WebSocket streams per client session:
/ws/live)wss://generativelanguage.googleapis.com/...)First the configuration, the latency tuning constants, the REST endpoints the UI uses, and the tool dispatcher. The dispatcher is the only place a model-requested action touches business logic:
import path from 'path';
import { fileURLToPath } from 'url';
import Fastify from 'fastify';
import fastifyStatic from '@fastify/static';
import fastifyWebsocket from '@fastify/websocket';
import dotenv from 'dotenv';
import WebSocket from 'ws';
import { commerce } from './commerce.js';
import { COMMERCE_FUNCTION_DECLARATIONS } from './tools.js';
dotenv.config();
const rootDir = path.resolve(path.dirname(fileURLToPath(import.meta.url)), '..');
const PORT = parseInt(process.env.PORT || '3000', 10);
const GEMINI_API_KEY = process.env.GEMINI_API_KEY || '';
const GEMINI_LIVE_MODEL = process.env.GEMINI_LIVE_MODEL || 'gemini-3.1-flash-live-preview';
const GEMINI_LIVE_VOICE = process.env.GEMINI_LIVE_VOICE || 'Aoede';
// Latency tuning for the Live API session.
// - Thinking: 3.x models take `thinkingLevel`, 2.5 models take `thinkingBudget` (0 = off).
// - VAD: HIGH end-of-speech sensitivity + a short silence window commits the user's turn sooner.
const THINKING_CONFIG = /gemini-3\./.test(GEMINI_LIVE_MODEL)
? { thinkingLevel: 'MINIMAL' }
: { thinkingBudget: 0 };
const REALTIME_INPUT_CONFIG = {
automaticActivityDetection: {
startOfSpeechSensitivity: 'START_SENSITIVITY_HIGH',
endOfSpeechSensitivity: 'END_SENSITIVITY_HIGH',
prefixPaddingMs: 20,
silenceDurationMs: 300,
},
};
const SYSTEM_INSTRUCTION =
'You are ShopTalk AI, a friendly, concise voice-first shopping assistant. ' +
'Never invent product prices, stock, or delivery times. Always call the provided commerce functions. ' +
'Keep verbal answers to 2-3 crisp sentences. Never read out raw URLs. ' +
'Pronounce prices naturally as full words, for example: "two thousand seven hundred ninety-nine rupees". ' +
'Before invoking place_order, you MUST read out the items, delivery address, and total in INR and get explicit verbal confirmation.';
const app = Fastify({ logger: false });
await app.register(fastifyWebsocket);
await app.register(fastifyStatic, { root: path.join(rootDir, 'public'), prefix: '/' });
// ----------------------------------------------------------------------------
// REST endpoints used by the browser UI (initial shelf, cart, quick triggers)
// ----------------------------------------------------------------------------
app.get('/api/products', async () => commerce.searchProducts({}));
app.get('/api/cart', async () => commerce.getCart());
app.post('/api/cart/reset', async () => {
commerce.reset();
return { success: true, cart: commerce.getCart() };
});
app.post('/api/tools/execute', async (request) => {
const body = request.body as { name: string; args: Record<string, any> };
const result = executeTool(body.name, body.args || {});
return { tool: body.name, result, cart: commerce.getCart() };
});
// ----------------------------------------------------------------------------
// Tool dispatcher: the ONLY place model-requested actions touch business logic
// ----------------------------------------------------------------------------
function executeTool(name: string, args: Record<string, any>): any {
switch (name) {
case 'search_products':
return commerce.searchProducts({
query: args.query,
category: args.category,
maxPriceInr: args.maxPriceInr,
minBatteryHours: args.minBatteryHours,
ancRequired: args.ancRequired,
});
case 'get_product_details':
return commerce.getProductDetails(args.productId);
case 'compare_products':
return commerce.compareProducts(args.productIds || []);
case 'check_inventory':
return commerce.checkInventory(args.productId, args.pincode);
case 'add_to_cart':
return commerce.addToCart(args.productId, args.quantity || 1);
case 'get_cart':
return commerce.getCart();
case 'check_delivery':
return commerce.checkDelivery(args.pincode || '560103');
case 'get_saved_addresses':
return commerce.getSavedAddresses();
case 'place_order':
return commerce.placeOrder({
addressId: args.addressId,
paymentMethod: args.paymentMethod,
userConfirmed: Boolean(args.userConfirmed),
});
default:
return { error: `Unknown commerce function: ${name}` };
}
}
Now the proxy itself:
// ----------------------------------------------------------------------------
// WebSocket proxy: Browser <==> this server <==> Gemini Live API
// ----------------------------------------------------------------------------
app.get('/ws/live', { websocket: true }, (clientWs) => {
console.log('[Proxy] Client connected');
if (!GEMINI_API_KEY) {
clientWs.send(JSON.stringify({ type: 'status', text: 'GEMINI_API_KEY is not set in .env' }));
return;
}
const geminiWsUrl = `wss://generativelanguage.googleapis.com/ws/google.ai.generativelanguage.v1beta.GenerativeService.BidiGenerateContent?key=${GEMINI_API_KEY}`;
// permessage-deflate only burns CPU (and adds latency) compressing already-dense base64 audio frames.
const geminiWs = new WebSocket(geminiWsUrl, { perMessageDeflate: false });
geminiWs.on('open', () => {
clientWs.send(JSON.stringify({ type: 'status', text: 'Connected to Gemini Live API', connected: true }));
geminiWs.send(
JSON.stringify({
setup: {
model: `models/${GEMINI_LIVE_MODEL}`,
generationConfig: {
responseModalities: ['AUDIO'],
thinkingConfig: THINKING_CONFIG,
speechConfig: { voiceConfig: { prebuiltVoiceConfig: { voiceName: GEMINI_LIVE_VOICE } } },
},
realtimeInputConfig: REALTIME_INPUT_CONFIG,
systemInstruction: { parts: [{ text: SYSTEM_INSTRUCTION }] },
tools: [{ functionDeclarations: COMMERCE_FUNCTION_DECLARATIONS }],
},
})
);
});
// Gemini -> browser (audio, barge-in signal) and Gemini -> backend (tool calls)
geminiWs.on('message', (data: WebSocket.RawData) => {
const response = JSON.parse(data.toString());
if (response.serverContent?.interrupted) {
clientWs.send(JSON.stringify({ type: 'interrupted' }));
}
for (const part of response.serverContent?.modelTurn?.parts ?? []) {
if (part.inlineData?.data) {
clientWs.send(JSON.stringify({ type: 'audio_chunk', data: part.inlineData.data }));
}
if (part.text) {
clientWs.send(JSON.stringify({ type: 'transcript', role: 'assistant', text: part.text }));
}
}
if (response.toolCall?.functionCalls) {
const functionResponses = response.toolCall.functionCalls.map((call: any) => {
console.log(`[Proxy] Executing tool: ${call.name}(${JSON.stringify(call.args)})`);
const result = executeTool(call.name, call.args || {});
// Dual Delivery: tell the screen what the voice agent just did
clientWs.send(
JSON.stringify({ type: 'dual_delivery', tool: call.name, args: call.args, result, cart: commerce.getCart() })
);
return { id: call.id, name: call.name, response: { output: result } };
});
geminiWs.send(JSON.stringify({ toolResponse: { functionResponses } }));
}
});
// Browser -> Gemini
clientWs.on('message', (raw) => {
if (geminiWs.readyState !== WebSocket.OPEN) return;
const msg = JSON.parse(raw.toString());
if (msg.type === 'audio_chunk') {
// `realtimeInput.audio` replaces the deprecated `mediaChunks`.
geminiWs.send(
JSON.stringify({ realtimeInput: { audio: { mimeType: 'audio/pcm;rate=16000', data: msg.data } } })
);
} else if (msg.type === 'audio_stream_end') {
// Mic was turned off. Without this the server VAD never sees trailing silence,
// never commits the turn, and the model simply never answers.
geminiWs.send(JSON.stringify({ realtimeInput: { audioStreamEnd: true } }));
} else if (msg.type === 'text_query') {
geminiWs.send(JSON.stringify({ realtimeInput: { text: msg.text } }));
}
});
clientWs.on('close', () => {
console.log('[Proxy] Client disconnected');
geminiWs.close();
});
geminiWs.on('close', (code, reason) => {
console.log(`[Proxy] Gemini closed (code=${code} reason=${reason.toString()})`);
clientWs.send(JSON.stringify({ type: 'status', text: 'Gemini session closed', connected: false }));
});
geminiWs.on('error', (err) => {
console.error('[Proxy] Gemini WebSocket error:', err);
clientWs.send(JSON.stringify({ type: 'status', text: `Gemini API error: ${err.message}` }));
});
});
await app.listen({ port: PORT, host: '0.0.0.0' });
console.log(`ShopTalk AI listening on http://localhost:${PORT} (model: ${GEMINI_LIVE_MODEL})`);
The server above already contains the latency work. Here is what each piece does and why it is there. Speech-to-first-audio numbers were measured with synthetic speech streamed in real time to the Live API, two to six runs per row, on one office network. Treat them as relative, not as a benchmark.
Configuration | End of speech to first spoken answer |
| 3.5 – 5.1 s |
Same model with thinking off and tuned voice detection | 3.2 – 5.2 s (barely moved) |
| 1.0 – 1.4 s |
3.1 with a tool call, measured through this server | about 1.3 – 2.5 s |
3.1, mic switched off with | about 1.0 s |
3.1, mic switched off without | no answer within 12 s (2 of 2 runs) |
The biggest win by far was the model. The 2.5 native-audio preview stayed at 3 to 5 seconds no matter how we tuned the session, while gemini-3.1-flash-live-preview answered in about a second. Set GEMINI_LIVE_MODEL to compare them on your own network.
thinkingConfig goes inside generationConfig. The 3.x models take thinkingLevel and the 2.5 models take thinkingBudget (0 turns thinking off), which is why THINKING_CONFIG branches on the model name.
realtimeInputConfig.automaticActivityDetection decides when the server thinks you stopped talking. endOfSpeechSensitivity: HIGH and silenceDurationMs: 300 commit your turn sooner. In our runs, 200 ms was fastest, 500 ms was close, and the default was about 0.3 s slower. A shorter silence window raises the risk of cutting off someone who pauses mid-sentence, so 300 ms is a compromise.
Voice activity detection needs to hear silence to know you finished. If the user taps the mic off, no more audio is sent, so the server never sees that silence and never answers. Send realtimeInput.audioStreamEnd: true when the mic stops. The browser sends an audio_stream_end message and the proxy forwards it (see server.ts).
realtimeInput.audio replaces the deprecated mediaChunks, and realtimeInput.text is the way to send text mid-session.perMessageDeflate: false on the Gemini socket avoids spending CPU compressing base64 audio that barely compresses.Gemini streams its reply as many tiny chunks (about 44 ms each). Playing each one only after the previous one fires onended leaves a small gap after every chunk. The next step schedules them back to back on the audio clock instead.
The public/app.js you copied contains the complete client. The excerpts below are taken from it unchanged, apart from decorative emoji in log messages, which are left out. The Web Audio API operates in two independent directions:
sequenceDiagram
participant Mic as Microphone
participant Cap as AudioContext (device rate)
participant WS as WebSocket Client
participant Ply as AudioContext (24kHz buffers)
participant Spk as Speaker
Mic->>Cap: Stream Float32 Audio
Cap->>WS: Resample to 16kHz Int16 PCM (Base64)
WS->>Ply: Receive Base64 Int16 PCM Chunks
Ply->>Spk: Schedule chunks back-to-back on the audio clock
Browsers pick the microphone's sample rate for you, commonly 44.1kHz or 48kHz, and iOS Safari rejects a forced sampleRate constraint. So the client captures at the device rate and resamples to the 16kHz that Gemini expects:
function resampleAudioTo16k(inputData, fromSampleRate) {
if (!fromSampleRate || fromSampleRate === 16000) {
const int16 = new Int16Array(inputData.length);
for (let i = 0; i < inputData.length; i++) {
const s = Math.max(-1, Math.min(1, inputData[i]));
int16[i] = s < 0 ? s * 0x8000 : s * 0x7fff;
}
return int16;
}
const ratio = fromSampleRate / 16000;
const outputLength = Math.round(inputData.length / ratio);
const int16 = new Int16Array(outputLength);
for (let i = 0; i < outputLength; i++) {
const srcIndex = Math.min(Math.round(i * ratio), inputData.length - 1);
const s = Math.max(-1, Math.min(1, inputData[srcIndex]));
int16[i] = s < 0 ? s * 0x8000 : s * 0x7fff;
}
return int16;
}
Each audio processing callback resamples, Base64-encodes, and sends one audio_chunk. At 48kHz a 2048-sample buffer is about 43 ms of audio, a good chunk size for the Live API:
scriptProcessor.onaudioprocess = (e) => {
if (!isRecording) return;
const inputData = e.inputBuffer.getChannelData(0);
// Dynamically resample device sample rate (e.g. 44.1kHz or 48kHz on Mac/iPad) down to 16kHz Int16 PCM
const int16 = resampleAudioTo16k(inputData, audioCtx.sampleRate);
// Base64 encode
let binary = '';
const bytes = new Uint8Array(int16.buffer);
for (let i = 0; i < bytes.byteLength; i++) {
binary += String.fromCharCode(bytes[i]);
}
const b64 = btoa(binary);
if (ws && ws.readyState === WebSocket.OPEN) {
ws.send(JSON.stringify({ type: 'audio_chunk', data: b64 }));
}
};
When the mic stops, tell the server the stream ended, otherwise it will wait forever for silence (see the previous step):
// Tell Gemini the audio stream ended so it commits the turn now (no trailing silence will ever arrive).
if (wasRecording && ws && ws.readyState === WebSocket.OPEN) {
ws.send(JSON.stringify({ type: 'audio_stream_end' }));
}
Rather than waiting for each chunk to finish, schedule every chunk on the AudioContext clock right after the previous one. Two pieces of state track the schedule:
// Audio playback (24kHz PCM from Gemini): every chunk is scheduled on the AudioContext clock
// right after the previous one, instead of waiting for `onended` (which leaves a gap per chunk).
let nextPlayTime = 0;
const activeSources = new Set();
function playPcm24kChunk(base64Data) {
ensureAudioContext();
const binary = atob(base64Data);
const bytes = new Uint8Array(binary.length);
for (let i = 0; i < binary.length; i++) {
bytes[i] = binary.charCodeAt(i);
}
const int16 = new Int16Array(bytes.buffer);
const float32 = new Float32Array(int16.length);
for (let i = 0; i < int16.length; i++) {
float32[i] = int16[i] / 32768.0;
}
const buffer = audioCtx.createBuffer(1, float32.length, 24000);
buffer.copyToChannel(float32, 0);
const source = audioCtx.createBufferSource();
source.buffer = buffer;
source.connect(audioCtx.destination);
const analyser = getAnalyser();
if (analyser) {
source.connect(analyser);
}
// Small lead on the first chunk absorbs scheduling jitter; later chunks chain back-to-back.
const startAt = Math.max(nextPlayTime, audioCtx.currentTime + 0.02);
source.start(startAt);
nextPlayTime = startAt + buffer.duration;
activeSources.add(source);
currentSourceType = 'ai';
source.onended = () => {
activeSources.delete(source);
if (activeSources.size === 0 && !isRecording) currentSourceType = 'idle';
};
}
nextPlayTime always points at the moment the last scheduled chunk ends, so the next chunk starts exactly there. The 20 ms lead on the first chunk absorbs scheduling jitter.
When a human interrupts an AI voice assistant, any delay in cutting off the agent feels unnatural and frustrating. Gemini sends interrupted as soon as it hears you speak, and the proxy forwards it as { type: 'interrupted' }. Because every chunk is already scheduled, the client simply stops all of them and resets the clock:
function flushAudioQueue() {
for (const source of activeSources) {
source.onended = null;
try {
source.stop();
} catch (e) {}
}
activeSources.clear();
nextPlayTime = 0;
if (!isRecording) currentSourceType = 'idle';
}
Every message from the proxy goes through one handler. audio_chunk frames go to the speaker, interrupted stops playback, and dual_delivery frames update the screen. The branches for simulation and simulatedSpeech belong to the offline simulation mode in the repository version and are never triggered by the server you just wrote:
ws.onmessage = (event) => {
try {
const msg = JSON.parse(event.data);
if (msg.type === 'status') {
if (msg.simulation) {
isSimulated = true;
statusText.textContent = 'Simulation Mode';
logTelemetry('system', msg.text);
} else {
statusText.textContent = msg.text;
logTelemetry('system', msg.text);
}
} else if (msg.type === 'interrupted') {
logTelemetry('interrupted', 'Barge-in detected: Gemini interrupted by user audio! Flushing playback buffer.');
flushAudioQueue();
} else if (msg.type === 'audio_chunk') {
playPcm24kChunk(msg.data);
} else if (msg.type === 'transcript') {
logTelemetry('assistant', `<strong>Gemini:</strong> ${msg.text}`);
} else if (msg.type === 'dual_delivery') {
handleDualDeliveryEvent(msg);
}
} catch (e) {
console.error('Error parsing WS message:', e);
}
};
function handleDualDeliveryEvent(msg) {
logTelemetry('tool', `Tool Executed: <strong>${msg.tool}</strong>(${JSON.stringify(msg.args || {})})`);
// 1. Update Products Shelf
if (msg.tool === 'search_products' || msg.tool === 'compare_products') {
const products = msg.result.products || msg.result.compared || [];
renderProducts(products);
}
// 2. Update Cart State
if (msg.cart) {
renderCart(msg.cart);
}
// 3. Handle Order Confirmation
if (msg.tool === 'place_order' && msg.result?.success) {
modalOrderId.textContent = msg.result.order.orderId;
modalOrderMsg.textContent = msg.result.message;
orderModal.classList.add('open');
renderCart(msg.cart);
}
// 4. Handle Simulated Speech
if (msg.simulatedSpeech) {
logTelemetry('assistant', `<strong>ShopTalk AI:</strong> ${msg.simulatedSpeech}`);
speakTextSynthetic(msg.simulatedSpeech);
}
}
The shelf renders battery and noise-cancellation pills only when the product has them, and always adds the category-specific highlights. Long-lasting devices show days instead of hours:
function formatBattery(hours) {
return hours >= 72 ? `${Math.round(hours / 24)} days battery` : `${hours}h Battery`;
}
${p.batteryHours ? `<span class="spec-pill">\u{1F50B} ${formatBattery(p.batteryHours)}</span>` : ''}
${p.ancType ? `<span class="spec-pill">\u{1F3A7} ${p.ancType}</span>` : ''}
${(p.highlights || []).map((h) => `<span class="spec-pill">${h}</span>`).join('')}
Launch your server:
npm run dev
Open http://localhost:3000 in Google Chrome and tap the microphone button. Test the full shopping journey. The numbers in the right-hand column were produced by running these exact steps against the server from this codelab:
Stage | User Utterance | What Happens |
1. Discovery | "I need travel headphones under 3,000 rupees with long battery." | Gemini typically calls |
2. Comparison | "Compare TravelBeats X with SoundWave Pro." |
|
3. Another category | "Now show me smartwatches." then "Any power banks?" | Gemini typically uses the |
4. Cart | "Add TravelBeats X and the VoltMax power bank to my cart." | Subtotal 4,998 rupees, 18% GST 900 rupees, free shipping, total 5,898 rupees. |
5. Delivery | "Can you deliver to 560103?" |
|
6. Two-phase checkout | "Place the order." then, after the agent reads back the items, address and total, "Yes, confirm." |
|
audioStreamEnd the agent still answers.GEMINI_LIVE_MODEL to gemini-2.5-flash-native-audio-preview-12-2025, restart, and feel the latency difference yourself.Google Cloud Run fully supports persistent HTTP/WebSocket streaming connections out of the box with zero additional configuration.
DockerfileFROM node:20-slim AS builder
WORKDIR /app
COPY package*.json tsconfig.json ./
RUN npm ci
COPY src/ ./src
RUN npm run build
FROM node:20-slim
WORKDIR /app
COPY package*.json ./
RUN npm ci --omit=dev
COPY --from=builder /app/dist ./dist
COPY public/ ./public
ENV PORT=8080
EXPOSE 8080
CMD ["node", "dist/server.js"]
# 1. Build and push container image using Cloud Build
gcloud builds submit --tag gcr.io/$GOOGLE_CLOUD_PROJECT/talk-to-shop-live
# 2. Deploy to Cloud Run with WebSocket support
gcloud run deploy talk-to-shop-live \
--image gcr.io/$GOOGLE_CLOUD_PROJECT/talk-to-shop-live \
--platform managed \
--region us-central1 \
--allow-unauthenticated \
--set-env-vars GEMINI_API_KEY=$GEMINI_API_KEY
You have built a bidirectional voice commerce shopping agent with Google Gemini Live API!