Peter Tinashe MundowaPeter Mundowa
← All workAI engineeringLive demo2026

Internal knowledge assistant: answers with sources, or an honest refusal

Try it live ↗Code on GitHub ↗

Runs on free hosting, so the first load can take up to a minute.

Captured from the live app
On this page
  1. In short
  2. Answer, cite or refuse
  3. Measured, not assumed
  4. Retrieval measured on its own
  5. Guards outside the model
  6. Built to run

In short

  • WhatAnswers only from the documents, and cites them.
  • HowGuards, hybrid search, a relevance gate and grounding checks.
  • Result89.6% of 48 labelled questions passed, 100% citation precision, zero safety failures, on llama-3.1-8b-instant.

My own project. The documents describe Contoso Financial, a fictional company.

Answer, cite or refuse

Answer, cite or refuseA question passes an input guard that refuses injection attempts and requests for customer data, then hybrid retrieval that combines meaning-based and keyword search, then a relevance gate that refuses without calling the model when nothing relevant is found. The model generates an answer with numbered citations in strict JSON, and a grounding check traces every citation and every figure to the cited text. The result is either an answer with its sources or a refusal in one fixed sentence.Questionfrom a member ofstaffInput guardinjection anddata requestsrefusedHybridretrievalmeaning andkeywords, fusedRelevancegatenothingrelevant, nomodel callGenerationnumberedcitations,strict JSONGroundingcheckevery citationand figuretracedAnswer withsourcespassages shownbeside itRefusalone fixedsentenceAnswer, cite or refuseA question passes an input guard that refuses injection attempts and requests for customer data, then hybrid retrieval that combines meaning-based and keyword search, then a relevance gate that refuses without calling the model when nothing relevant is found. The model generates an answer with numbered citations in strict JSON, and a grounding check traces every citation and every figure to the cited text. The result is either an answer with its sources or a refusal in one fixed sentence.Questionfrom a member of staffInput guardinjection and datarequests refusedHybrid retrievalmeaning and keywords,fusedRelevance gatenothing relevant, nomodel callGenerationnumbered citations,strict JSONGrounding checkevery citation andfigure tracedAnswer with sourcespassages shown besideitRefusalone fixed sentence

Passing every check gives an answer with sources; any failed check ends in the same refusal sentence.

Every reply either carries its sources or says plainly that the documents cannot answer.

  • Unsafe questions are refused before any search.
  • No relevant passage means no model call.
  • A claim that cannot be traced is refused.
Technical detail

Every refusal is the same sentence: “I don’t have sufficient information in the provided documents.” The reason (no relevant context, an ungrounded figure, an injection attempt and so on) goes in a separate field for operators, so a caller probing the system learns nothing about which rule fired. The confidence label is computed from the evidence (citation validity, figures traced to the cited text, lexical support and retrieval strength), not from the model’s tone.

Measured, not assumed

  • 48 labelled questionsFactual, multi-document, unanswerable, injection and confidential.
  • 89.6% passedAll 48, on llama-3.1-8b-instant.
  • Zero safety failuresNo forbidden content reached an answer.

A fixed question set, scored the same way on every run, with the misses published.

  • 20 factual, 9 multi-document, 8 unanswerable, 6 injection, 5 confidential.
Technical detail

With no model at all, the offline extractive baseline still passes injection 6/6 and confidential 5/5, because those controls are code rather than model behaviour. On llama-3.1-8b-instant the p95 latency is 1,438 ms. The weakest category is multi-document questions: 5 of 9 pass, where the answer has to combine two or three documents and the model sometimes drops a clause.

Retrieval measured on its own

  • Dense only: 0.948Meaning-based search alone.
  • BM25 only: 0.960Keyword search alone.
  • Hybrid, fused: 0.977Both, with their rankings fused.

An answer is only as good as the passages behind it, so retrieval is scored without the model.

  • Recall@6 over the 29 answerable questions, with no model calls.
Technical detail

Selection is scored on the fused score because scoring it on dense similarity alone threw the keyword evidence away at the last step: a policy that was the top keyword match never reached the final six passages. With selection on dense similarity, hybrid recall@6 was 0.925; on the fused score it is 0.977. The dense and BM25 rows return their raw top six passages, while both hybrid rows also remove near-duplicates.

Guards outside the model

  • Input guardInjection and customer-data requests refused before any search.
  • Trust tiersNotice-board text is untrusted, so planted instructions are stripped.
  • Grounding checkEvery citation and every figure must trace to the cited text.

The controls that matter are code, so a persuasive prompt cannot talk its way past them.

Technical detail

The input guard combines pattern rules with a classifier when the hosted model is on. Each document gets a trust tier from what it says about itself, not from its filename; instruction lines inside untrusted passages are replaced before the prompt is built, and an answer whose citations all point at untrusted content is refused, so a notice-board claim is never stated as policy.

Built to run

  • 94 tests
  • CI with a safety gate
  • Docker
  • No API key neededRuns on an offline extractive model.

Anyone can clone it and run it without a key.

Try it live ↗Runs on free hosting, so the first load can take up to a minute.

Want to talk about this work? Email me or download my CV.