Multi-Agent Advisory Workspace
Engineered a multi-agent AutoGen workspace serving five specialised agents behind one WebSocket chat, combining tool calling, hybrid-search grounding, deterministic code-side validation and per-user persisted team state for audit and deal-advisory work.
- Role
- Lead backend engineer · agents, prompts & state
- Year
- 2025
- Type
- Multi-agent
- Stack
- AutoGen AgentChat · Azure OpenAI · Azure AI Search · Python · FastAPI · WebSockets · React · TypeScript · Document Intelligence · pandas · python-docx
- specialised agents
- 5
- audit evidence categories
- ~40
- architecture iterations
- 3
The problem
Audit and deal-advisory teams at an advisory firm repeat rule-bound, document-heavy tasks: classifying client evidence, finding the right internal specialist, writing minutes, drafting due-diligence SOWs and normalising trial balances exported from different accounting packages. The rules lived in checklists, templates and people's experience, and every output had to land in an exact house format.
Architecture
- 01
Ingest
Attachments converted to Markdown text before reaching the model, via Document Intelligence layout extraction and later MarkItDown.
- 02
Session
A FastAPI WebSocket resolves user, session and agent, then restores the matching AutoGen team state with load_state().
- 03
Orchestrate
Each turn builds a RoundRobinGroupChat around a config-driven AssistantAgent with a 25-message buffered context at temperature 0.1.
- 04
Act
Agents call tools for hybrid vector + semantic search, Word template generation or a pandas-validated trial-balance CSV writer.
- 05
Terminate & persist
TextMessageTermination ends the turn at the agent's first text reply; save_state() checkpoints per user, session and agent.
- 06
Deliver
Markdown answers return over the WebSocket, generated documents as download links, and a lifespan task purges sessions after seven days.
Engineering highlights
01
Config-driven agent topology
Five agents defined in one config registry (system prompt, tools, reflect_on_tool_use). Each turn spins up a single-agent team with name-based termination, so tool calls resolve and control returns to the user deterministically.
02
Validation enforced in code
The trial-balance agent maps columns strictly by header or position into hidden structured JSON; the tool, not the model, enforces the debit/credit balance check in pandas and emits a zero-padded, four-column CSV.
03
Isolated, persisted agent state
Every agent gets a fresh 25-message buffered context per team, so no history leaks across agents. Team state is serialised per user, session and agent after each turn and expired after seven days.
04
Grounded routing with citations
The service-line routing agent answers only from hybrid keyword + vector retrieval with semantic reranking over firm documents, embedded with text-embedding-3-small, and cites the source file for every specialist it recommends.
05
Prompt-to-tool contracts
The minutes agent renders reviewable Markdown, then on export emits a fixed JSON schema that a python-docx tool maps onto template placeholders and table rows, with the prompt requiring both forms to match exactly.
06
Evidence classification
Classifies uploaded client files against about 40 audit evidence categories, returning a verdict plus ranked alternatives with likelihood percentages, and supports a filter-driven mode that tests a single subcategory.