Project Brennan

Introduction #

Project Brennan is an agentic/LLM system to read works of fiction, and generate a database of characters, relationships, dialogue, locations, plot events, tropes and more. It’s specifically designed to scale well for books which contribute to a single shared world.

History #

I started work on Project Brennan back in 2024. Since the COVID-19 lockdown of 2020, my partner had started a new career publishing her own books online. She was steadily been building a name for herself, and since the genre she wrote in was short romance novels, she published at least one book a month.

Many of these books were set in the same shared world, and continuity became an appreciable burden. Moreso for small-town romance, where there are reoccurring characters. With romance, each book deals with a couple getting together of course, but you also want to entice readers to read the next and previous books in the series, so you intertwine the characters via familial connections. In the early days this was done haphazardly as she was learning, and later became a little more formalised.

Since I had started to play with local LLMs on a 128GB M1 Mac Studio, I wondered whether we could get the models to read the book, and compile a list of characters for us. Given the context limits of 4-8k tokens of the era, my partner’s short stories had chapters just about fit the windows, and allowed for some experimentation.

From 2024 until 2026, I could never get the system to become stable. When I found an issue scanning one book, and applied a prompt change, it would destabilise another book. I never got beyond about 5 or 10 books scanned with any useful data extracted. I tried each new opeb model as they came out; LLama2 70B, Llama3 70B, Mixtral 8x22B, GPT OSS 120B and others. There was progress, but the models weren’t smart enough to correctly pick up on all the nuance in books I was scanning.

2026 Breakthrough #

In early 2026 after I’d spent a few months working with Gemini 2.5 Flash and a rework of the backend to use Pydantic AI, things were looking up. The models were finally getting smart enough and quick enough to scan books within a reasonable time frame, and extract useful data. Crucially, they were also fault tolerant, able to interpret the spirit of the prompts, not just the literal text. This gave them a flexibility to bridge the gap between raw book text and prompts designed to extract structured data.

Following this, I started scaling the richness of the extracted data. This involved generating far more specialised agents which were a translation of the original micro-interrogations concept from 2024. One agent is responsible for detecting characters, another for attempting to infer a last name for each character based on familial relationships within the chapter. Another agent is responsible for assigning a gender label to each character, and so on.

It was a careful balance, as you were augmenting the original verbatim chapter text with additional derived data, then feeding that back into an agent which was designed to build upon this layered knowledge. If you were not careful, you could easily end up with a poison cascade where small errors in the derived data could propagate and corrupt the later results.

Present day #

The current alpha test software has now scanned about 84 books, including works in the public domain as well as my partner’s short romance books. The main frontend is a simple RAG system that allows you to query the extracted data in whatever way you like. A query decomposition agent is used to break down complex queries into simpler sub-queries, which are then executed individually and combined to produce the final result. It’s particularly good at queries which span multiple books, relationships between different entities, and dialogue.

I’ve extended the original genre capabilities beyond romance to include cosy mystery, historical fiction and comedy fiction and sci-fi. Book scanning has scaled up to be a fairly complex process, and it can take hours for all the stages to complete, but the resulting corpus is highly detailed, and allows for complex queries to be answered with high accuracy.