Voice AI Agent Platform
Real-time, multilingual voice AI for inbound, outbound, and web calls with configurable agents and multi-tenant isolation.
3
Channels
Multi-tenant
Architecture
Vector DB
Context
Architecture
Overview
As a member of the founding team (Sr. AI Engineer) at a Voice AI startup, I implemented a production-grade, real-time multilingual voice AI system supporting inbound, outbound, and web calls. Users configure custom voice agents via prompts and tool setup — the platform handles SIP/Twilio telephony, WebRTC via LiveKit, streaming STT, LLM reasoning with tool calls, and TTS synthesis.
Problem
Building voice agents at scale requires more than connecting an LLM to a phone line. Key challenges included:
- Multi-channel support — same agent logic across SIP, outbound campaigns, and web calls
- Context awareness — agents need customer-specific knowledge from websites and PDFs
- Call quality — voicemail detection, ghost calls, and inactivity handling degrade UX fast
- Multi-tenancy — per-tenant resource limits, concurrency controls, and auth isolation
Architecture
The system follows a real-time voice pipeline: caller audio → Deepgram STT → LLM with tool calls → FastAPI backend → ElevenLabs TTS → caller. LiveKit handles RTC transport and data channels for auxiliary signals like voicemail predictions. See the interactive diagram above.
Challenges
- Voicemail & beep detection: Implemented detection models that relay predictions to the primary voice agent via RTC data channels, reducing false handling on automated systems
- Ghost calls & inactivity: LLM-driven inactivity detection with automatic call termination; warm transfer tooling routes conversations based on predefined conditions
- Context ingestion: Firecrawl-based ingestion of website and PDF content into a vector database, processed asynchronously via AWS SQS and Lambda
- Multi-tenant safety: Per-tenant resource limits and isolation across the voice AI stack, with controls applied dynamically based on customer pricing tiers
Scale / Performance
- Tenant-level concurrency controls prevent resource exhaustion across customers
- Async vector DB ingestion decouples content processing from live call latency
- RBAC, M2M auth, and Logto with RLS enforced across FastAPI APIs
Interesting Engineering Decisions
LiveKit for RTC transport
LiveKit provides sub-100ms audio transport and native data channels — critical for relaying voicemail/beep predictions to the agent without adding latency to the voice path.
Async ingestion via SQS + Lambda
Website and PDF content ingestion runs asynchronously. Calls start immediately; context enrichment happens in the background and becomes available for subsequent interactions.
LLM-driven call lifecycle
Rather than hard-coded timeouts, inactivity detection uses LLM reasoning to distinguish natural pauses from abandoned calls — reducing false terminations while eliminating ghost calls.
Per-tenant resource governance
Concurrency limits and resource caps are tier-based, enforced at the API and voice stack level. Prevents noisy-neighbor problems in a multi-tenant SaaS.
Lessons Learned
- Voice UX failures (voicemail loops, ghost calls) matter as much as LLM quality
- Data channels on the RTC layer are underrated for auxiliary signals
- Multi-tenant isolation must be designed in from day one, not bolted on
- Async context ingestion keeps call latency independent of knowledge base size
Outcome
Delivered a production multilingual voice AI platform with configurable agents, reliable call handling, context-aware responses, and secure multi-tenant operation (Aug – Dec 2025).