Reasoning audit system · Built solo in 2 months
Biassemble finds biased reasoning.
With evidence — or it stays silent.
Every finding is bound to a verbatim quote from the source. No quote, no claim. Built by one engineer as a live transition from 10+ years of frontend into LLM engineering.
- [Story]
- [Retrieve]
- [Assess]
- [Evidence]
Retrieval runs during the user's 30–120 s of think-time.
Every finding carries a verbatim quote — or none is emitted.
How it works
Story arrives
Any personal story, any domain.
Hybrid retrieval
One generative LLM call ∪ vector search over a 38-bias taxonomy. Each candidate carries a source tag: [llm], [vector], or both.
Evidence-bound assessment
The LLM must quote the exact words that justify every finding. Schema-enforced.
…or silence
No bias found is a first-class result. The system never invents findings to look useful.
False positives are worse than false negatives — that rule shaped every architectural decision.
The numbers
0.667 → 0.875
eval Recall@5, vector-only → hybrid
three retrieval architectures in two months
0.562 / 0.85
live Recall@5 vs target
the open gap — published, tracked, being closed
67 s → async
NLI latency, solved architecturally
retrieval now runs during the user's think time
~500 ms
vector path incl. pgvector
the fast signal that never left
4 → 0
knowledge-base experiments → recall gained
the assumption that had to die
38
cognitive biases in the taxonomy
each with indicators, examples, and false-positive notes
Every transition between architectures was forced by a measurement, not a preference.
The path
Four architectures. Two months.
Vector search
fast, topic-blind
Expand the knowledge base
4 experiments, zero movement
Zero-shot NLI
production: 67 s → timeout ✗
LLM ∪ vector hybrid — today
next: fine-tune on labeled production stories
Each stage earned its exit with a number.
Human ⇄ AI
Who improved whom.
Biassemble was built in constant dialogue with AI. The rule: treat every AI suggestion as a hypothesis to test — and expect it to do the same for mine. Receipts below.
AI improved my architecture
The async think-time idea
AI challenged my assumption that retrieval must finish before the user continues. Users spend 30–120 s answering follow-up questions anyway — retrieval now runs inside that window. 67 seconds of latency, hidden in human think time.
The review quorum
Every important artifact — findings, plans, even this site's copy — is reviewed by at least two different AI models before it ships. The quorum caught real design flaws my own reading missed.
I improved the AI's output
The IDs-only prompt cut
AI kept suggesting richer context — full definitions of all 38 biases in every prompt. I cut it to a bare list of IDs. Latency fell from ~50 s to ~8 s and results improved. More context isn't more information.
Catching AI's confident fiction
During model selection, one AI recommended a classifier by name and described its outputs in detail — outputs the model cannot produce (its labels are therapy categories, not reasoning biases). Another AI flatly denied a model variant existed; a 30-second check proved it did. Verification beat confidence both times.
The engine audits stories. The metrics audited me.
Grounded, faithful, or hallucinating? — a working glossary
What is RAG faithfulness?
Whether generated text is actually supported by the retrieved sources, not merely plausible. A faithful answer can quote its evidence; an unfaithful one only sounds right.
How do you measure groundedness?
Claim-level verification: split the output into checkable claims, then verify each against the source — supported, unsupported, or contradicted — with a verbatim quote or an explicit absence. Aggregate rates, not vibes.
How is this different from hallucination detection?
Hallucination checks ask "is this true?"; groundedness asks "do YOUR sources say this?". The second is answerable without knowing the world — and it's what an audit can actually guarantee.
Why does 'no finding' matter?
A verifier that always finds something is a noise generator. Biassemble treats silence as a first-class result and tracks its false-positive rate as a hard gate.
This is the direction Biassemble is growing: claim-level groundedness audits of AI-generated content.
The engineer
Frontend developer for 10 years. LLM engineer for two months. Same discipline.
I'm Dmitrii Lëmin. For a decade I shipped production frontend and full-stack systems in TypeScript, Node.js, and React.
Two months ago I started Biassemble to answer one question: can a solo engineer build a production-grade LLM system with real evaluation discipline — golden datasets, CI-gated quality metrics, versioned prompts, measured false-positive rates? The system above is the answer so far.
What transferred from frontend wasn't the stack. It was the habit of shipping, measuring, and letting the numbers overrule my taste.
See it run. Read the code. Or argue with me about evals.
Built with Lovable. Reviewed by two AIs and one stubborn human.