Skip to content
← All workProduction

Due-Diligence Document Search

Engineered an incremental SharePoint-to-Azure AI Search pipeline on Microsoft Graph delta, with resumable crawls, typed extraction from compliance PDFs and agentic hybrid retrieval behind a fail-closed audit trail.

Role
Lead engineer · ingestion, retrieval & infra
Year
2026
Type
RAG & search
Stack
Azure AI Search (agentic retrieval) · Azure OpenAI · Python · Azure Functions · Microsoft Graph · Cosmos DB · React · TypeScript · Azure Table Storage · pytest · Entra ID · managed identity
lines of pytest with in-memory fakes
3,200

The problem

A real-estate consultancy's compliance team answered audit requests by walking deeply nested SharePoint folders by hand. Native search matched file-name keywords only, missed address variants, ignored the folder convention and could not search inside structured decision forms, while every query needed an auditable record.

Architecture

  1. 01

    Sync

    Timer-triggered Python Function consumes the Microsoft Graph delta feed, with a resumable breadth-first initial crawl and checkpointed, time-boxed runs.

  2. 02

    Parse

    Folder paths decoded into division, office, transaction type, property, postcode and entity role; off-convention paths flagged.

  3. 03

    Extract

    Decision-form PDFs repaired and parsed into section-aligned chunks and typed fields, with template-version detection across two layouts.

  4. 04

    Index

    Token-budgeted embedding batches into one Azure AI Search index linking file and chunk rows by parent ID.

  5. 05

    Retrieve

    An LLM turns queries into search intents; an agentic knowledge base runs hybrid BM25 + vector search with semantic reranking.

  6. 06

    Serve

    HTTP Function writes a Cosmos DB audit record before returning ranked, de-duplicated SharePoint links to a React UI.

Engineering highlights

  • 01

    Durable delta cursor

    The delta cursor never advances past non-durable writes. Backend failures abort the run and retry the page; per-item data faults go to a failure table. Time-boxed checkpoints let the backfill resume across timer ticks.

  • 02

    Folder-move repair

    Graph does not re-emit descendants on folder renames or moves, so the pipeline marks moved folders and re-lists children page by page with in-folder checkpoints, keeping large flat folders progressing without livelock.

  • 03

    Epoch-based delete sweep

    Each full enumeration stamps live rows with a monotonic sync epoch. After a clean pass, older rows are removed as deletions the feed never reported; the sweep is skipped if the pass had any errors.

  • 04

    Fail-closed compliance audit

    No query is served unless its audit record, with user identity and result IDs, is durably written to Cosmos DB. A single-owner lease in Table Storage blocks concurrent ingest runs, returning HTTP 409.

  • 05

    Hash-based change detection

    Compared quickXorHash instead of cTag, which this delta feed omits, so renames and moves refresh only path metadata and reuse existing embeddings. Deterministic chunk IDs and stored counts trim orphan chunks without a search query.

  • 06

    Private, keyless deployment

    Flex Consumption Functions with managed identities, private endpoints and VNet integration, Entra-authenticated Static Web App, and index, synonym map, knowledge source and knowledge base deployed as REST JSON definitions.

Next projectAircraft Records Q&A and Validation