Building a Multi-Turn, Multi-Agent, Data Insight Chat System Part 1: The Blank Page
Prior Experience
In late 2024 with AI hype in full swing the company I work for, like many others was wondering if bringing AI into our product line could bring benefits to our users. At this point I had a few personal AI projects under my belt mostly involving using LM Studio to host a variety of different models to invoke primitive tool calls for home for home automation.
The stack looked something like this
Python to orchestrate everything and pass data between the various models involved
Faster Whisper to take in audio from a microphone and transcribe it into text
A model hosted by LM Studio which provides an OpenAI like API takes in the transcribe text, with a prompt instructing it to select an appropriate tool call to control lights, heating, answer questions etc or to refuse if no appropriate tool could be found.
coqui-TTS to vocalize model output
The biggest challenge was fitting all the models into the meager 8GB of VRAM I was working with at the time, which required a very small context window and some dynamic loading and unloading of models at each stage.
It was a bit slow and hardly an enterprise grade solution but it did give me some experience in the capabilities of these models, OpenAI like APIs, Tool Calls, how to stitch it all together and crucially how to build the system in a model agnostic way, allowing for improved models to be plugged in as and when they became available.
The Ask
So what was the business ask on the AI front? From the start we laid out two principles which would act as a north star for all our AI related development:
Any use of AI must be genuinely useful and solve a real problem. No jumping on the bandwagon, shoving in AI where it didn’t belong (I’m looking at you Microsoft with copilot in notepad of all places!). No AI for AI’s sake.
AI is not a magic bullet. I liken this to the Pareto principle (the 80:20 rule), you can’t expect AI to get everything right, notoriously it hallucinates, makes mistakes and certainly doesn’t “think” in any traditional sense of the word. Any solutions/features must accept and even lean into this fact. Let AI do what it can (the 80%) and let humans finish the job (the 20%).
With these principles in place we began looking at where in our products (in particular our Portfolio, Programme and Project Management software) AI might be able to make a difference and identified 3 areas:
AI assisted data imports: Could we get AI to help turn unstructured data and turn it into structured data suitable for importing into our database. (I may write a post about exactly how this works in future, it’s a great example of a task AI can make possible by doing the 80% of the heavy lifting)
Data Insights: Many of the fields in our system are related. How good would AI be at filling in the gaps if one of those fields was missing or out of date, e.g. could it automatically decide what a project status should be and provide a clear accurate analysis as to why?
The Project Personified: Given knowledge of all the data in the project, could users have a meaningful “conversation” with their project and ask questions, get insights and turn natural language based asks into concrete action?
Example prompts/questions:
Are there any gaps in our financial planning?
Based on the latest updates how is this project trending, are there any risks in the update narratives that should be recorded formally?
- And could you record those for me?
Add a risk a new risk covering a new global pandemic as it relates to this project.
All three of these features ended up being implemented and are live in the product today. This series of posts will focus on the “The Project Personified” (which is not what it ended up being called!) how it was built, and the many changes it went through before being let loose in the wild.
In the next post I’ll discuss the first major challenge the idea ran into: a limited context window, and the innovation and workarounds which came about as a result.
