Building EVA - Part 3: polished implementations and a testing-focused demo

---

Introduction

In Part 2 I added an agent layer with LangGraph (routing, memory, evals). At the end of that part I left a "known limitation" documented: two content files talk about a similar idea, the embedding doesn't separate them, and the bi-encoder's top-k gets it wrong consistently.

As a best practice, I tagged that case in the eval as a known_limitation with the note: "pending with reranking layer in part 3".

Part 3 will cover:

• Two-stage reranking with a cross-encoder
• A double-check step in reformulation, so that rewriting the query with the LLM doesn't degrade it instead of improving it
• The reproducible test

Why reranking?

There were four possible paths:

1. Tune the chunking more
2. Raise the top-k the LLM sees
3. Query expansion before embedding
4. Cross-encoder reranking

Of those:

• Tuning the chunking is the first empirical option, but it doesn't scale.
• Raising top-k to 10 feeds the LLM more noise and pushes cost up linearly.
• Query expansion improves recall (am I finding anything relevant?) but not precision (is the order right?).
• The cross-encoder reorders the top-N candidates using a model that takes (query, chunk) together as input, not separately. Instead of independent vector similarity, there's cross-attention between the query words and the chunk words.

The reranker I picked is cross-encoder/ms-marco-MiniLM-L-6-v2:

• ~90MB in size.
• CPU-friendly.
• Cost per query: ~150ms of CPU over 10 candidates.
• 0 tokens to the LLM.
• The FinOps criteria from Parts 1 and 2 stay intact.

The two-stage architecture

Tap to expand

Changes from the Part 2 graph:

• retrieve_node now brings back CANDIDATES_K = 10 instead of 3.
• New rerank_node between retrieve and grade, filtering down to FINAL_K = 3.
• grade still reasons on the BI-encoder score (not the reranker's).

In code, rerank_node looks like this:

def rerank_node(state: EvaState) -> dict:
    query = state.get("query") or state["question"]
    candidates = state.get("candidates", [])
    reranked = rerank(query, candidates, top_k=FINAL_K)
    return {"retrieved": reranked}

And rerank() is the call to the cross-encoder, with a toggle:

def rerank(query, candidates, top_k=3):
    if not candidates:
        return []
    if not RERANK_ENABLED:
        return candidates[:top_k]

    model = _load_model()
    pairs = [(query, c["text"]) for c in candidates]
    scores = model.predict(pairs)

    reranked = []
    for c, s in zip(candidates, scores):
        item = dict(c)
        item["rerank_score"] = float(s)
        reranked.append(item)

    reranked.sort(key=lambda x: x["rerank_score"], reverse=True)
    return reranked[:top_k]

The EVA_RERANK=0 toggle is intentional by design: you can run the eval with rerank on, then off, and get the empirical A/B without touching code.

Where each chunk's scores live

• bi_score: answers "how close are these two vectors in the embedding space?"
• rerank_score: answers "how relevant is this chunk for this query, considering their word-by-word interaction?"

They're completely different scales.

Comparing scores in reformulation

The reformulate node rewrites the query with an LLM when retrieval scores in the weak band. The intent is that the new query improves retrieval.

But rewriting with an LLM doesn't guarantee improvement. Sometimes the original query was clearer for the retriever, and the reformulation adds synonyms that scatter the embedding. The fix is to snapshot before reformulating and revert if the new score is worse.

Tap to expand

def reformulate_node(state):
    # use LLM
    return {
        "query": new_query,
        "retries": state.get("retries", 0) + 1,
        # Snapshot for the guard in grade()
        "pre_reform_score": state.get("best_score"),
        "pre_reform_hits": state.get("retrieved"),
        "pre_reform_query": query,
    }

def grade(state):
    score = state.get("best_score", 0.0)
    pre_score = state.get("pre_reform_score")

    if pre_score is not None and score < pre_score:
        return {
            "retrieved": state.get("pre_reform_hits"),
            "best_score": pre_score,
            "query": state.get("pre_reform_query"),
            "grade": "answer",
        }

The A/B eval test

Part 2 documented the "known limitation"; here in Part 3, that case is the one that should pass with reranking active.

And since I added the EVA_RERANK toggle, I can run the same test twice by changing a single env var:

python -m evals.run_evals               # rerank ON
EVA_RERANK=0 python -m evals.run_evals  # rerank OFF

That's exactly the effect a cross-encoder produces when the limitation is about precision, not recall.

Enriching the knowledge base

With the infrastructure now complete (RAG + agent + reranking + evals), the next step is content.

What I'm adding before running the final test:

• Certifications: during the Part 2 demo, EVA honestly answered "I don't have information about Kevin's certifications" when I asked. That's the grounding working, but it's also a real content gap. Those go into the KB.
• Technical projects with depth: not just the name and a one-liner, but the key design decisions. The posts on this same blog are natural content for that.
• Blog posts: every post indexed as a source. EVA should be able to answer "what did Kevin write about reranking?" by citing this very Part 3.

With more content and more chunks come more adversarial queries, and that's where reranking pays for itself. I also suspect new known limitations will show up, probably around time ("what's the most recent thing he did?") or cross-topic ("which projects did he use Python 3 in?"), where reranking will be fully justified.

The full code lives at github.com/KevDP/ai-knowledge-blog-assistant.

The live EVA is running in the bottom-right corner of this same page.