Public-Records AI Assistant
Lead engineer · retrieval, ingestion, evaluation & infra
Architected and evolved a citation-grounded RAG assistant for a public-sector authority across four generations, from hybrid search with LLM query rewriting to Azure AI Search agentic retrieval over two federated indexes, driven by offline evaluation of production traffic.
- production turns evaluated
- 416
- substantive answers carrying citations
- 93.7%
- architecture generations since 2023
- 4
- Evaluation-driven re-architecture. Built an offline evaluator over 416 production turns: regex failure taxonomy, citation parsing and a gpt-4.1 relevance judge. It exposed false rejections, follow-up and typo failures, and drove the move to agentic retrieval.
- Federated agentic retrieval. Chose push-based ingestion into two schema-specific indexes federated by one knowledge base over a single indexer pipeline, so each source owns its schema and deletion strategy while queries still span one corpus.
- Idempotent, fail-loud ingestion. Deterministic base64 parent IDs and indexed chunk IDs make re-ingestion delete-then-upsert with no orphans. Validation runs before deletes, OData filters are escaped, and vector-length checks fail loudly on a wrong embedding model.
Architecture
- 01
Ingest
Blob-triggered Python Function cracks disclosure PDFs with Document Intelligence, strips page noise, stitches cross-page tables and extracts reference numbers and letter dates.
- 02
Crawl
Containerised crawl4ai/Playwright deep crawler harvests the public website and linked documents, with incremental sync and guarded deletion reconciliation.
- 03
Index
Structure-aware token chunks embedded with text-embedding-3-large into two purpose-built HNSW indexes, upserted idempotently by deterministic parent and chunk IDs.
- 04
Retrieve
An Azure AI Search knowledge base plans subqueries over the last six turns, federates both indexes, then reranks and merges results with references.
- 05
Generate
A FastAPI backend grounds the model on the retrieved block and streams cited bullet answers over WebSockets, linking exact PDF pages or web URLs.
- 06
Observe
Turns logged to Cosmos DB by verified Entra role, then scored offline by regex failure classifiers and an LLM judge.
Evolution
2023 · Prototype
Document Q&A demos on early Azure OpenAI
2024 · Production RAG
Hybrid search, LLM query rewriting, queue ingestion, web crawler
2025 · Agentic
Tool-calling agent, WebSocket streaming, private networking
2026 · Agentic retrieval
KB query planner over federated indexes, LLM-judge evals