The scenario
VoiceDesk is a low-latency voice agent for a small business: it streams speech-to-text, retrieves answers from FAQs and price lists, can book appointments through a tool, and streams speech back while it is still generating. The focus is latency budgets — what each stage costs and how to hide it.
System architecture
01 Caller Browser / phone
02 Speech-to-text Streaming
03 Agent + RAG Answer or book
04 Text-to-speech Streaming reply
High-level flow of VoiceDesk, from input to output.
What you will learn
- Streaming STT → LLM → TTS pipelines
- Latency budgeting and barge-in handling
- RAG under real-time constraints
- Booking and handoff tools for voice
What you will build
- Browser voice client over WebRTC
- Streaming pipeline with interruption support
- Business knowledge RAG
- Appointment booking tool
Build plan
- Echo bot over WebRTC
- Add LLM and streaming TTS
- Add RAG over business docs
- Booking tool and barge-in
- Latency tuning and deploy