The scenario
PolicyDesk is an internal HR assistant that answers from the company’s own policy documents and cites the clause it used. Retrieval combines dense vectors with keyword search and a cross-encoder reranker, so both “how many casual leaves do I get” and “clause 4.2 travel” work. Prompt-injection protection and scope checks keep it on-topic, LLM routing with fallback keeps it available, and an LLM-as-judge pipeline tracks answer quality. It ships as a serverless container on Cloud Run behind Google sign-in.
System architecture
01 Policy files PDF · DOCX · PPTX
02 Ingestion Parse + chunk + embed
03 Qdrant Dense + BM25 hybrid
04 Reranker Cross-encoder top-k
05 LLM router Primary + fallback
06 Cloud Run app Cited answers behind SSO
High-level flow of PolicyDesk, from input to output.
What you will learn
- Hybrid dense + sparse retrieval and when each one wins
- Reranking to improve grounded answers
- Guardrails against prompt injection and off-topic use
- Model routing and automatic fallback
- Measuring faithfulness and groundedness with automated judges
What you will build
- Ingestion for PDF, DOCX and PPTX policy files
- Conversational assistant with citations and per-session memory
- Guardrail and semantic-cache layer
- Cloud Run deployment with Google OAuth and Secret Manager
Build plan
- Multi-format ingestion pipeline
- Hybrid retrieval with reranking
- Chat agent with citations and memory
- Guardrails, caching and evaluation
- Deploy to Cloud Run with sign-in