Due-Diligence Document Search
Engineered an incremental SharePoint-to-Azure AI Search pipeline on Microsoft Graph delta, with resumable crawls, typed extraction from compliance PDFs and agentic hybrid retrieval behind a fail-closed audit trail.
- Role
- Lead engineer · ingestion, retrieval & infra
- Year
- 2026
- Type
- RAG & search
- Stack
- Azure AI Search (agentic retrieval) · Azure OpenAI · Python · Azure Functions · Microsoft Graph · Cosmos DB · React · TypeScript · Azure Table Storage · pytest · Entra ID · managed identity
- lines of pytest with in-memory fakes
- 3,200
The problem
A real-estate consultancy's compliance team answered audit requests by walking deeply nested SharePoint folders by hand. Native search matched file-name keywords only, missed address variants, ignored the folder convention and could not search inside structured decision forms, while every query needed an auditable record.
Architecture
- 01
Sync
Timer-triggered Python Function consumes the Microsoft Graph delta feed, with a resumable breadth-first initial crawl and checkpointed, time-boxed runs.
- 02
Parse
Folder paths decoded into division, office, transaction type, property, postcode and entity role; off-convention paths flagged.
- 03
Extract
Decision-form PDFs repaired and parsed into section-aligned chunks and typed fields, with template-version detection across two layouts.
- 04
Index
Token-budgeted embedding batches into one Azure AI Search index linking file and chunk rows by parent ID.
- 05
Retrieve
An LLM turns queries into search intents; an agentic knowledge base runs hybrid BM25 + vector search with semantic reranking.
- 06
Serve
HTTP Function writes a Cosmos DB audit record before returning ranked, de-duplicated SharePoint links to a React UI.
Engineering highlights
01
Durable delta cursor
The delta cursor never advances past non-durable writes. Backend failures abort the run and retry the page; per-item data faults go to a failure table. Time-boxed checkpoints let the backfill resume across timer ticks.
02
Folder-move repair
Graph does not re-emit descendants on folder renames or moves, so the pipeline marks moved folders and re-lists children page by page with in-folder checkpoints, keeping large flat folders progressing without livelock.
03
Epoch-based delete sweep
Each full enumeration stamps live rows with a monotonic sync epoch. After a clean pass, older rows are removed as deletions the feed never reported; the sweep is skipped if the pass had any errors.
04
Fail-closed compliance audit
No query is served unless its audit record, with user identity and result IDs, is durably written to Cosmos DB. A single-owner lease in Table Storage blocks concurrent ingest runs, returning HTTP 409.
05
Hash-based change detection
Compared quickXorHash instead of cTag, which this delta feed omits, so renames and moves refresh only path metadata and reuse existing embeddings. Deterministic chunk IDs and stored counts trim orphan chunks without a search query.
06
Private, keyless deployment
Flex Consumption Functions with managed identities, private endpoints and VNet integration, Entra-authenticated Static Web App, and index, synonym map, knowledge source and knowledge base deployed as REST JSON definitions.