Building EVA - Part 2: from pipeline to agent

---

Introduction

In Part 1 I left EVA as a linear RAG pipeline (takes a question, hands it to the workflow, finds top-k matches in DynamoDB, passes through a relevance filter score to decide if it's off-topic, otherwise reaches Claude).

But it was missing four things needed to take it to the next level.

1. No conversation memory

Every query was independent. If you asked "who is kevin?" and then "what did he study?", the system treated the second one in isolation; it had no idea that "he" meant Kevin.

2. No routing

Everything went through the same flow. Diagnostic questions ("what are your sources?") ran through the same pipeline as a content query.

3. No reformulation when retrieval failed

If the best score was 0.56 (right above the threshold), the pipeline proceeded even though retrieval was weak.

4. No automated eval

Every threshold, prompt or model change relied on running some queries by hand and eyeballing whether the result held up.

That said, LangGraph is a convenient tool for this use case: it solves all four problems.

Why LangGraph and not just an if/else?

The clear need is:

• Branching: tell apart a normal question vs a meta-query vs an off-topic one
• Shared state across nodes: keep conversation history, retry counters, or accumulated turn cost
• Ordered composition: reordering nodes shouldn't force a rewrite of the whole flow

Which points at a proper state machine.

The chosen architecture

Tap to expand

Decision 1: memory

The history is designed so the caller passes it on every turn:

def run_turn(question: str, history: list[dict] | None = None):
    initial = {
        "question": question,
        "history": history or [],
        "retries": 0,
    }
    return _graph.invoke(initial)

In my local CLI, the REPL keeps the memory in process. In production, that history would come from the HTTP request or a DynamoDB table. The agent code doesn't change; only the caller does. That's exactly the kind of decision that makes a port to Lambda a matter of hours, not days.

LangGraph's checkpointer with an external backend (DynamoDB) stays on the radar for production, but it's a Part 3 or 4 evolution; it doesn't block the correct separation of responsibilities.

Decision 2: hybrid routing. Heuristic first, LLM only when it hesitates

Every LLM call costs tokens. A router that hits the LLM on every query pays for intelligence where a "research action" was enough.

95% of the queries get routed with regex, in microseconds, for free:

_META_SOURCES_RE = re.compile(
    r"\b(sources?|fuentes?|cita|de d[oó]nde|show.*source)\b",
    re.IGNORECASE,
)
_META_CONCISE_RE = re.compile(
    r"\b(shorter|m[aá]s corto|briefly|concise|res[uú]mel[oa])\b",
    re.IGNORECASE,
)

If it matches, the agent skips retrieval and jumps straight to the meta-query node. Which means it won't burn tokens. The pattern: route cheap when you can.

Decision 3: coreference resolution

This is the problem memory fixes: when a user asks "who is kevin?" and then "what did he study?", the second turn may not match anything useful in the KB.

Solution: heuristic trigger.

_PRONOUN_RE = re.compile(
    r"\b(he|she|it|they|him|her|that|there|"
    r"[ée]l|ella|eso|esa|su|sus)\b",
    re.IGNORECASE,
)

def resolve_context(state):
    q = state["question"]
    history = state.get("history", [])
    needs_resolution = bool(history) and (
        _PRONOUN_RE.search(q) or len(q.split()) <= 4
    )
    if not needs_resolution:
        return {"query": q}  # no cost
    # only here does it pay for LLM
    resolved, cost = _utility_llm(...)
    return {"query": resolved, "utility_cost_usd": cost}

• No history -> never fires.
• No pronouns and a long autonomous query -> never fires.

Coreference resolution only pays when there's something to resolve.

Decision 4: max-retries as a cost guard

The reformulate -> retrieve -> grade -> reformulate loop needs a cap for situations where retrieval never crosses the threshold; otherwise the graph enters an infinite reformulation cycle burning cost forever.

MAX_RETRIES = 1

def grade(state):
    score = state.get("best_score", 0.0)
    retries = state.get("retries", 0)
    if score < OFF_TOPIC_THRESHOLD:
        return {"grade": "off_topic"}
    elif score < WEAK_BAND_CEIL and retries < MAX_RETRIES:
        return {"grade": "reformulate"}
    else:
        # Provide the result
        return {"grade": "answer"}

Two intentional design choices here:

1. Reformulation only kicks in for scores in the murky band between the off-topic gate and a usable score.
2. Once retries are exhausted, it answers with the best score obtained after the retry.

Memory in action (trace results)

Tap to expand

Turn 1. No history, so resolve_context doesn't need to fire; the question is clear on its own.

Turn 2. The query brings the pronoun "he" and history exists. The coref gate fires, the LLM rewrites the query to "what did Kevin study?", and the search now scores 0.770. Utility cost for the turn: $0.000278.

That jump from $0 to $0.000278 between turns is the proof that memory triggered. On a dashboard, that's the first thing I'd check to see whether coref is doing its job.

During the same demo I asked this:

you > qué certificaciones tiene?

I don't have information about Kevin's certifications in the context
available to me. To find out about his certifications, I'd recommend
checking his full resume or CV, or reaching out to him directly.

At the time of writing, my knowledge base doesn't yet include that content (that'll land sometime after Part 3). Instead of inventing a plausible list, it answered "I don't know", which is exactly what the system prompt is meant to enforce.

The automated eval

Every time I adjust a threshold, a prompt, or a reformulation rule, I need success criteria for the different situations the agent might face.

Key decisions:

• Verify agent decisions, not the quality of its answer.
• Decisions (routing, gating, top-k, memory) are deterministic and cheap to verify.
• Judging quality would require an adaptable LLM, which gets expensive to run on tests.

The "testing set" includes:

• Normal queries per topic (skills, education, work, projects).
• Explicit off-topic.
• Multi-turn conversations reproduced in order, to validate the last turn.
• Meta-queries.

"Known limitation"

Running the eval tests, one case failed consistently: the question "what projects has ____ built?" returned different sources than expected. Digging in, two content files overlap semantically and the embedding doesn't separate them.

The right fix isn't to keep tuning the content, but to add a reranking layer.

Coming up in Part 3

• Dynamic reranking: with a larger KB, the eval suite starts failing more, because retrieval degrades with volume. The solution is a cross-encoder reranker that reorders only the top-k, meaning it considers two variables instead of one.
• Reformulation: if the reformulated query scores worse than the original, revert.
• A LangGraph "checkpoint" backed by DynamoDB, for when the agent lives in Lambda with persistent memory.

With Part 3, EVA starts taking shape: a stack that can handle real traffic, scale with volume, and won't have serious cost incidents.

See you in Part 3.

The full code lives at github.com/KevDP/ai-knowledge-blog-assistant. The live EVA is running in the bottom-right corner of this same blog.