Building the RAG Framework Behind Kubrius Core

I've spent a lot of time recently thinking about what a useful enterprise AI system actually looks like.

It isn't just dumping your documents into a vector database, asking a giant cloud model to "be helpful," and hoping nobody asks where the answer came from. That version of RAG is easy to demo. The hard version is building something a company can actually trust.

For Kubrius Core, the goal was a local RAG framework that searches across messy enterprise sources, answers only from evidence, exposes where the answer came from, and knows when to say, simply:

I do not know.

After plenty of testing, breaking things, and resisting the urge to over-tune the system into a benchmark-passing goblin, our current promoted baseline scores 95.32% in-scope strict clean efficiency on a 500-question enterprise benchmark, all running locally on local models and local infrastructure. The full breakdown:

500 total questions
470 in-scope sourced questions
Overall strict clean efficiency: 92.2%
In-scope strict clean efficiency: 95.32%
Strict unknowns: 39 / 500
In-scope strict unknowns: 22 / 470
Errors: 0
Extractive fallbacks: 0
Hardcoded rescues: 0
Benchmark-specific answer hacks: 0

That local-infrastructure part matters.

The achievement isn't just the score. It's that the framework improved while staying clean: no hardcoded document IDs, no expected-answer lookups, no "if benchmark question contains X, return Y," no secret trapdoors.

Here's how we got there, and why the framework matters more than the model alone.

---

The Enterprise RAG Problem

The toy version of RAG is simple:

1. Embed the documents.
2. Search the vector database.
3. Stuff the top chunks into the prompt.
4. Generate an answer.

That works beautifully in clean demos.

The enterprise version is less polite.

Company knowledge is spread across Slack threads, Jira tickets, GitHub issues, Gmail chains, Fireflies transcripts, HubSpot notes, Linear tasks, Confluence pages, and random Google Docs from 2021 with names like final_final_updated_v3.

Some questions need semantic understanding. Some need exact matching. Some need dates. Some need numbers. Some need the model to realise the evidence isn't there.

A useful enterprise RAG framework has to handle all of this while remaining grounded, auditable, and safe to refuse.

If you're building an AI control layer, you cannot pass the benchmark by quietly cheating. The whole point is to build a system that behaves properly when nobody is watching.

---

The Local Stack

Our stack was designed to run entirely locally:

vLLM                 Model serving
Qwen                 Chat, embeddings, and reranking
Qdrant               Dense vector retrieval
SQLite / BM25        Lexical retrieval
Custom RAG Gateway   Orchestration, retries, context control, and audit traces

The gateway is where most of the real work happens.

The model matters, of course. But the model alone isn't the product.

The gateway decides:

• how to combine dense and lexical search,
• how much context to include,
• how to preserve useful evidence,
• when to retry,
• when to refuse,
• when an answer is too weak to accept,
• and how to expose the trace afterwards.

That's what separates a chatbot with documents from a controlled RAG framework.

---

Why Hybrid Retrieval Was Non-Negotiable

Pure vector search is good at meaning. It's less reliable for exact enterprise details: error codes, ticket names, internal flags, deployment versions, CLI commands, customer names, dates, phone numbers, and obscure configuration fields.

BM25 is old-school, but like a hammer, it's exactly what you want when something needs hitting.

Across the larger benchmark, BM25 consistently found evidence that dense search missed:

Dense hit:      348 / 500
BM25 hit:       416 / 500
Retrieved hit:  432 / 500
Context hit:    395 / 500

The lesson was clear:

For enterprise RAG, lexical search is not a fallback. It is a core component.

Dense retrieval gives you semantic breadth. BM25 gives you exactness. The framework needs both.

---

Verifiable Answers, Not Just Confident Ones

An answer on its own isn't enough.

If the system gives an answer, it must show where that answer came from. That's what makes a tool useful for real work instead of a chatbot that just happens to have access to documents.

For Kubrius Core, every useful answer needs a strict contract:

Answer
The direct response to the user's question.

Sources
The documents, tickets, threads, calls, or notes used to support it.

Confidence & limitations
Whether the answer is strongly supported, partially supported, or missing definitive evidence.

Trace
Retrieval and generation metadata that can be audited later.

This changes how the system handles ambiguity.

When evidence is strong:

Answer: The beta is available to Dedicated and Hosted customers.
Sources:
  - Product rollout note, Google Drive, updated 2026-03-12
  - Jira ticket CORE-1842, comments from Platform PM

When evidence is weak:

Answer: I found related rollout notes mentioning Dedicated and Hosted customers, but I did not find a definitive source for this specific feature flag gate. I do not know.

The second answer is less glamorous, but far more useful.

In enterprise infrastructure, a confident but untraceable answer is a liability.

By default, Kubrius Core surfaces source quality out in the open: not hidden in developer logs, not buried in a trace file, visible right where the user can ask:

Where did this come from?
Is it current?
Is this official policy?
Or is this just a Slack message from Dave?

Dave may be right.

He often is.

But we still need to know it was Dave.

---

Learning When It Doesn't Know

An "I do not know" response shouldn't be treated as failure. In an enterprise system, it can become a useful signal.

Kubrius Core turns unknowns into an improvement loop:

1. A user asks a question.
2. Kubrius searches the knowledge base.
3. The evidence is insufficient.
4. Kubrius says: "I do not know."
5. The user investigates manually.
6. The verified correction is saved as a structured knowledge item.
7. That record is indexed back into the retrieval layer.
8. The next user gets the verified answer.

The important part: Kubrius doesn't blindly learn from raw chat text. Corrections need to become clean, versioned, auditable knowledge records.

Example:

Field	Entry
Question	What customer tiers are allowed into the Console beta?
Canonical Answer	Selected customers on Dedicated and Hosted tiers with opt-in.
Source / Evidence	Product rollout note, March 2026. Confirmed by Platform Team.
Scope & Status	Applies to Console beta feature flag gate. Verified.

A lot of missing company knowledge isn't really missing. It's trapped in someone's head, buried in an old Slack thread, or scattered across three systems.

The moment someone asks the question is the perfect time to capture that knowledge properly.

---

The Golden Rule: Don't Optimise by Making the System Worse

When testing RAG, it's tempting to celebrate whenever unknowns go down.

But fewer unknowns can mean two very different things:

Good:
The system found better evidence.

Bad:
The system became more willing to blag it.

We saw both.

Several experimental builds reduced unknowns but introduced answer drift, accepted weak evidence, or became over-confident on aggregate questions.

We rejected them.

A benchmark improvement only counts if the system remains clean. That became our golden rule:

A better score only matters if the answer path is still honest.

---

What We Improved

The framework improved through a series of controlled changes. Each change had to beat the current baseline without adding hardcoded rescues or benchmark-specific logic.

1. BM25 Preservation

We found cases where BM25 retrieved the right evidence, but the final answer context dropped it.

The fix: preserve a small number of top lexical candidates in the answer context. This helped because BM25 often catches exact phrases, customer names, configuration keys, and error wording that dense retrieval can blur.

This was generic retrieval logic, not a benchmark shortcut.

BM25_PRESERVE_FOR_ANSWER_TOP_K=4

2. Dense Preservation

After BM25 preservation worked, we tested the mirror image: preserving a small number of dense candidates.

Too much dense preservation added noise. Top 4 made things worse. Top 2 helped.

DENSE_PRESERVE_FOR_ANSWER_TOP_K=2

This was an important lesson: more context isn't automatically better. Extra evidence can dilute the answer just as easily as improve it.

3. Answer Context Depth

Next, we increased the amount of text available per answer document.

The earlier setting was too shallow for some questions where the correct span was inside the right document but slightly beyond the truncated section. Increasing answer-context document characters helped without changing retrieval.

ANSWER_CONTEXT_DOC_CHARS=1300

That was a clean win because retrieval numbers stayed mostly stable, but unknowns dropped.

4. Qdrant Fail-Open

One benchmark error came from a dense vector query failure.

The right response wasn't to crash the whole request. If Qdrant times out, BM25 should still be allowed to carry the query.

We added a fail-open path instead:

If dense retrieval fails:
    log warning
    continue with lexical retrieval

This removed a hard error without adding any answer-specific logic.

5. Unknown-Only Deep Retrieval Retry

A global deeper retrieval test looked promising at first. It found more candidate documents.

But it made the system worse overall.

The deeper run improved raw retrieval but degraded clean efficiency, because it introduced noise that broke previously correct answers.

We rejected deeper retrieval as the default because of that. The safer version was an unknown-only retry:

Normal path:
    run current best retrieval
If answer is good:
    return it
If answer is unknown:
    run deeper retrieval
    only accept if strict gates pass

This is a key pattern. Retry logic shouldn't disturb already-good answers. It should only activate after a safe failure.

6. Strict Retry Acceptance

The first retry path recovered useful answers, but it also revealed a new failure mode.

Some retry answers were verbose non-answers:

The provided context does not contain specific pass/fail thresholds...

Those should still count as unknowns.

We tightened the acceptance gate in response: a retry answer now gets rejected if it contains markers such as:

provided context does not contain
cannot determine
not enough information
not specified in the context
insufficient evidence

This kept the system honest.

7. Strict Unknown Scoring

The evaluator needed fixing too.

Previously, the benchmark counted literal "I do not know" responses as unknowns, but verbose refusals could slip through as successful answers. That inflated clean efficiency.

We responded with stricter unknown scoring: these now count as non-answers too:

I do not know
The provided context does not contain...
Cannot determine...
Insufficient context...
Not specified in the context...

The result was a more honest metric. The current promoted baseline is measured using this stricter scoring.

8. Query Rewrite: Powerful, But Not Yet Promoted

One more experiment: unknown-only retrieval-query rewriting.

The idea was simple:

Original question:
In a recent GPU incident in us-east after a driver update, what load balancer network setting change was suspected as contributing to the NCCL watchdog timeouts and Xid errors?

Retrieval query:
recent GPU incident us-east driver update load balancer network setting change suspected contributing NCCL watchdog timeouts Xid errors

This worked extremely well for some misses. It recovered questions that the normal retrieval path failed to find.

But it also introduced risk.

One aggregate Fireflies question asked how many transcripts mentioned data residency requirements. The query-rewrite path produced a confident long answer that appeared to over-count. Another answer looked plausible even though the expected document hadn't been retrieved.

That's exactly the type of behaviour we don't want.

For now, query rewrite stays experimental. It's an important discipline point:

Useful but risky doesn't become production just because the metric looks better.

---

The Current Kubrius Core RAG Pattern

The current promoted pattern looks like this:

[Receive Question]
       │
       ├──► Dense Retrieval via Qdrant
       └──► Lexical Retrieval via SQLite/BM25
       │
[Merge Candidates]
       │
[Preserve Top BM25 + Dense Evidence]
       │
[Rerank]
       │
[Build Answer Context]
       │
[Generate Answer with Sources]
       │
[Strict Answer Evaluation]
       │
       ├──► If Good
       │       └──► Return Answer + Sources + Trace
       │
       └──► If Unknown / Weak
               │
               ├──► Unknown-only Deeper Retrieval Retry
               │
               ├──► Strict Acceptance Gate
               │
               ├──► If Gate Passes
               │       └──► Return Retry Answer + Sources + Trace
               │
               └──► If Gate Fails
                       └──► Return Safe Unknown
                                │
                                └──► Optional Human Correction Loop

The key shift is that Kubrius Core isn't trying to generate text at all costs. It's a controlled loop around company knowledge:

retrieve
answer
verify
refuse when needed
learn from confirmed corrections

---

Current Results

The latest promoted clean baseline is:

500-question strict benchmark
Dense hit:       348 / 500
BM25 hit:        416 / 500
Retrieved hit:   432 / 500
Context hit:     395 / 500
Strict unknowns: 39 / 500
Errors:          0
Extractive:      0
Overall strict clean efficiency: 92.2%

For in-scope sourced questions:

470 in-scope questions
Strict unknowns: 22 / 470
Errors:          0
Extractive:      0
In-scope strict clean efficiency: 95.32%

That distinction matters. Some benchmark questions have no source filter or intentionally missing evidence. For actual enterprise use, the in-scope number is often the more meaningful one.

A system that answers roughly 95% of answerable internal questions, refuses safely on the rest, and exposes its evidence is already useful.

Not autonomous-agent useful.

Not "let it approve payments" useful.

But absolutely useful as an internal knowledge assistant, support copilot, onboarding assistant, sales research layer, or policy lookup tool.

---

When Does RAG Become Useful for an Organisation?

Based on this work, I'd think about it in bands:

<80%
Demo only. Useful for showing the concept, not for relying on it.

80-90%
Useful for assisted search, but still frustrating.

90-95%
Useful for internal copilot workflows with citations and human review.

95-98%
Strong enough for department-level rollout in controlled workflows.

98-99%
Approaching semi-automated workflow territory, depending on risk.

99%+
Needed for high-trust operational automation.

The important thing isn't just the score. It's the failure mode.

A 93% system that safely says "I do not know" is far more useful than a 97% system that guesses.

For Kubrius Core, the priority isn't to remove every unknown. The priority is to make sure every answer is earned.

---

10 Design Principles for Enterprise RAG

If you take anything away from this build, let it be these:

Also published as a standalone, shareable reference: 10 Design Principles for Enterprise RAG.

1. Keep a clean baseline

Don't judge an experiment in isolation. Compare it against a known clean version.

2. Use hybrid retrieval by default

Dense search and BM25 solve different problems. Enterprise RAG needs both.

3. Treat BM25 as first-class infrastructure

It's not legacy. It's exact-match memory.

4. Preserve important evidence into answer context

If retrieval finds the right evidence but context packing drops it, the model never had a fair chance.

5. Retry only after safe failure

Don't disturb already-good answers. Unknown-only retries are much safer than global deeper retrieval.

6. Gate retries strictly

A retry answer shouldn't be accepted just because it isn't literally "I do not know."

7. Count verbose refusals as unknowns

"The provided context does not contain..." is an unknown. Your evaluator should treat it that way.

8. Be suspicious of aggregate questions

"How many?", "which customers?", and "across all" questions often need dedicated aggregation logic, not normal RAG generation.

9. More retrieval isn't always better

Deeper retrieval can find more documents and still make the final answer worse.

10. Never hardcode benchmark answers

If you do, you haven't built a better RAG gateway. You've built a very small theatre production.

---

Frequently Asked Questions

---

Final Thoughts

Local AI doesn't have to mean toy AI.

With the right retrieval framework, local models can deliver controlled, cloud-quality RAG behaviour.

The trick isn't pretending the model is magic. The trick is building the boring but crucial infrastructure around it:

hybrid retrieval
reranking
context control
evidence preservation
typed retries
strict acceptance gates
source attribution
safe refusal
human correction loops
audit traces

That's what Kubrius Core is becoming.

Not another chatbot.

A controlled RAG framework for private AI infrastructure.

None of that matters if there's no reason to run AI locally in the first place. If that's still an open question for your business, see why SMEs need an AI control layer before they need another AI tool.