Public-Records AI Assistant
Architected and evolved a citation-grounded RAG assistant for a public-sector authority across four generations, from hybrid search with LLM query rewriting to Azure AI Search agentic retrieval over two federated indexes, driven by offline evaluation of production traffic.
- Role
- Lead engineer · retrieval, ingestion, evaluation & infra
- Year
- 2024–2026
- Type
- RAG & search
- Stack
- Azure OpenAI · Azure AI Search (agentic retrieval) · Python · FastAPI · Azure Functions · Document Intelligence · React · Cosmos DB · AutoGen · Azure Container Apps · Entra ID · managed identity
- production turns evaluated
- 416
- substantive answers carrying citations
- 93.7%
- architecture generations since 2023
- 4
The problem
Many Freedom-of-Information requests ask for information already disclosed in earlier responses or on the authority's website, and staff were finding it by hand. The assistant had to stay strictly grounded, cite every claim to a source page, never expose personal data, and fall back to the formal request route when it could not answer.
Architecture
- 01
Ingest
Blob-triggered Python Function cracks disclosure PDFs with Document Intelligence, strips page noise, stitches cross-page tables and extracts reference numbers and letter dates.
- 02
Crawl
Containerised crawl4ai/Playwright deep crawler harvests the public website and linked documents, with incremental sync and guarded deletion reconciliation.
- 03
Index
Structure-aware token chunks embedded with text-embedding-3-large into two purpose-built HNSW indexes, upserted idempotently by deterministic parent and chunk IDs.
- 04
Retrieve
An Azure AI Search knowledge base plans subqueries over the last six turns, federates both indexes, then reranks and merges results with references.
- 05
Generate
A FastAPI backend grounds the model on the retrieved block and streams cited bullet answers over WebSockets, linking exact PDF pages or web URLs.
- 06
Observe
Turns logged to Cosmos DB by verified Entra role, then scored offline by regex failure classifiers and an LLM judge.
Engineering highlights
01
Evaluation-driven re-architecture
Built an offline evaluator over 416 production turns: regex failure taxonomy, citation parsing and a gpt-4.1 relevance judge. It exposed false rejections, follow-up and typo failures, and drove the move to agentic retrieval.
02
Federated agentic retrieval
Chose push-based ingestion into two schema-specific indexes federated by one knowledge base over a single indexer pipeline, so each source owns its schema and deletion strategy while queries still span one corpus.
03
Idempotent, fail-loud ingestion
Deterministic base64 parent IDs and indexed chunk IDs make re-ingestion delete-then-upsert with no orphans. Validation runs before deletes, OData filters are escaped, and vector-length checks fail loudly on a wrong embedding model.
04
Private, keyless infrastructure
Consolidated three App Services into one, with no public network access: hub-and-spoke networking, private endpoints and VNet integration behind an application gateway, managed identity with documented RBAC roles instead of keys.
05
Structure-aware chunking
Replaced character windows with a heading-scoped 800-token chunker: small sections packed, breadcrumbs written into chunk text, tables split on row boundaries with headers repeated, navigation-only chunks detected and dropped.
06
Hardened token validation
Moved from unverified token decoding to full issuer, audience and expiry checks with a cached, rotation-aware JWKS client run off the event loop. Admin role comes only from verified claims; WebSocket origins are allow-listed.