Building EVA - Part 1: a RAG assistant over my own portfolio

---

Introduction

If you scroll around my site you'll notice an icon in the bottom-right corner. Tap it and EVA pops up: a chatbot that answers questions about me. EVA is a full RAG (Retrieval-Augmented Generation) pipeline running on AWS that:

• Analyzes and embeds your question with Amazon Titan (multi-language version)
• Looks up the most relevant fragments of my CV, experience and projects in DynamoDB
• Passes those fragments to Claude Haiku 4.5 as context
• Returns an answer that fits the question

All of it at almost zero cost thanks to the free tier.

This first part covers how I designed the system prioritizing cost control and the honest limitations of a simple RAG. Part 2 adds LangGraph orchestration: conversation memory, dynamic routing, and query reformulation when retrieval fails.

Why RAG and not fine-tuning or a bare LLM?

Fine-tuning gets ruled out fast because the information keeps changing (every new blog post, new certification, new project). Retraining a model each time I update my work experience is absurd both financially and in time. On top of that, a fine-tuned model tends to blur the line between facts and "the info suggests this might have happened".

Pure prompt-stuffing isn't an option either. The KB is simple data and pretty small (~20KB). It fits in Haiku's context window easily, but you pay the full context on every query, even when the user only asks for my contact. With RAG I only send the ~800 tokens that are actually relevant.

That said, RAG wins because:

1. The information lives outside the model, in a source I can audit and update with no retrain.
2. Cost per query scales with volume, not with the size of the KB.
3. The pipeline attaches the retrieved sources to the response JSON, and the system prompt forces the model to say "I don't have info on that" rather than hallucinate.
4. The whole pipeline is debuggable; I can inspect which chunks were retrieved and why they scored high.

The chosen architecture

I built EVA in two deliberate phases. Phase 0 was local, with no AWS. Phase 1 took it to production.

Phase 0: local validation

Tap to expand

Stack:

• Python + sentence-transformers (BAAI/bge-small-en-v1.5)
• JSON as the vector store
• Anthropic API directly (no Bedrock yet)
• Interactive CLI with rich

Why BGE-small instead of jumping straight to a cloud embedder?

• Runs on any laptop.
• Solid retrieval benchmarks.
• Zero cost per query.
• Moving to Titan Embeddings later is a client swap, not a pipeline rethink.

If there's one thing I've learned about designing systems, it's: validate the algorithm before paying to make it scale. From simple to complex.

Phase 1: AWS production

Tap to expand

The whole stack lives in us-east-1. Vector search is a linear DDB Scan because at my scale (21 chunks) it's faster and cheaper than any dedicated vector DB.

Contextual retrieval: why I embed metadata, not just the chunk

The problem: my skills.md file has a section like this:

## Languages
- Python - 4 years (primary)
- C++ - 4 years (academic + OOP/DSA work)

When I embed that as-is, the resulting vector has no idea which document it came from or what it represents. When somebody asks "what are Kevin's skills?", the question's embedding lands closer to chunks that literally use the word "skills".

For situations like this, Anthropic documented a solution called "contextual retrieval": enrich the embedded text with metadata, while the text passed to the LLM stays untouched.

def build_embedding_text(chunk, source, topic):

      # EMBEDDINGS:

    return f"Document: {source} | Topic: {topic} | Content: {chunk}"

      # CHUNK IN DYNAMODB:

item = {
    "text": chunk,                          # SENT TO HAIKU
    "embedding": embed(build_embedding_text(chunk, source, topic)),  # RESULT WANTED
    "source": source,
    "topic": topic,
}

Four-layer cost defense (LLM FinOps)

A public LLM endpoint is high risk. A bot could send 1,000 queries/hour, each costing ~$0.001 in Haiku 4.5 tokens. That's $1/hour, $24/day, ~$720/month.

I designed EVA with four defense layers, each at near-zero cost.

Layer 1: Relevance gate

Fires when the question isn't related to anything in the KB, so it isn't worth invoking Haiku at all.

RELEVANCE_THRESHOLD = 0.25
CANNED_OFF_TOPIC = "I can only answer questions about Kevin Delgado..."

def is_off_topic(top_chunks):
    if not top_chunks:
        return True
    return top_chunks[0]["score"] < RELEVANCE_THRESHOLD

# In the handler:
top = retrieve_top_k(query_embedding, chunks, k=3)
if is_off_topic(top):
    return {"answer": CANNED_OFF_TOPIC, "off_topic": True, "cost_usd": 0.000001}

Empirical threshold: 0.25. How did I land there? I measured 7 calibration queries (4 on-topic, 3 off-topic) in Phase 1 with Titan and looked at the distribution:

Tap to expand

Important note on the scale: these scores are from Titan (Phase 1), and the absolute ranges differ from BGE in Phase 0, where on-topic reached ~0.66. Never reuse a threshold across models without recalibrating.

Limitations: the gate does NOT catch domain-adjacent queries. There are topics that slip past it and still consume tokens.

Layer 1 is a probabilistic noise filter, not a security boundary. By design, the idea is to layer several defenses. The real defense against volume lives in Layer 3.

Layer 2: Input length cap

MAX_QUESTION_CHARS = 500  # ~120 tokens
if len(question) > MAX_QUESTION_CHARS:
    return 400

When someone tries to push 8k tokens of prompt injection, by design it gets rejected in microseconds, before touching the embedder.

Layer 3: Rate limiting

resource "aws_apigatewayv2_stage" "default" {
  # ...
  default_route_settings {
    throttling_burst_limit = 20
    throttling_rate_limit  = 10  # req/s
  }
}

Trying 100 requests in one second gets 90 of them back as 429 without ever touching Lambda. Zero tokens consumed. Layer 1 is a semantic filter; Layer 3 is the real defense against volume.

Layer 4: Budget alarms

Three staggered thresholds:

• 50% of the budget
• 80% of the budget
• 100% of the monthly budget (forecasted)

Each of those alerts hits my inbox as an email notification.

Bonus: Response caching in DynamoDB

Cache read (hit):  ~$0.00000025 (DDB read)
Cache miss + LLM:  ~$0.001
Break-even:         >0.03% hit rate (trivial)
Expected hit rate:  30-50% (FAQ pattern on portfolios)

Questions like "who is kevin?" are going to repeat a lot. Cache them with a 24h TTL and SHA256 of the normalized question:

q_hash = hashlib.sha256(question.strip().encode()).hexdigest()
cached = cache_get(q_hash)
if cached and cached["ttl"] > time.time():
    return cached

Important note: the system doesn't cache off-topic responses. The PutItem cost exceeds the embed cost it would save; caching only makes sense when serving without cache costs more than the PutItem.

Real-time cost tracking

Every invocation logs structured JSON to CloudWatch:

{
  "event": "rag_invocation",
  "model": "claude-haiku-4-5",
  "input_tokens": 487,
  "output_tokens": 156,
  "embed_tokens": 14,
  "cost_usd": 0.000927,
  "best_score": 0.652,
  "top_sources": ["experience.md", "about.md", "experience.md"]
}

CloudWatch Insights aggregates it in real time:

fields cost_usd
| filter @message like /rag_invocation/
| stats sum(cost_usd) as total by bin(1h)

When I tweak a threshold or swap a model, I see the impact on the bill without waiting for the monthly report.

Cost ceiling in real scenarios

(Numbers assume ~$0.001 per on-topic query to the LLM; the exact figure depends on real token size and Bedrock Haiku pricing. The point isn't the precise decimal, but the order of magnitude and how each layer caps it.)

Infra as code

The whole stack runs under Terraform:

infra/
   main.tf              # provider, remote backend (S3 + DDB lock)
   lambda.tf            # function + log group + env vars
   api_gateway.tf       # HTTP API + CORS + throttling
   dynamodb.tf          # knowledge + cache tables
   iam.tf               # Lambda role with least-privilege
   budget.tf            # alarms 50/80/100
   variables.tf         # bedrock model id, threshold, etc.

Remote state in S3 + DynamoDB lock

• Terraform state in S3 encrypted and versioned, with locking on DDB. This sits inside the free tier.
• The benefits are huge over running terraform apply with a local terraform.tfstate, which is exposed to corruption or laptop loss.

GitHub Actions OIDC federation

When the workflow runs:

1. Actions requests a JWT from GitHub OIDC
2. AWS verifies the JWT against the role's trust policy (scoped to the specific repo)
3. AWS returns temporary credentials (1h TTL)
4. Terraform runs with those credentials
5. When the workflow ends, the credentials expire on their own

After the npm supply-chain attacks in 2026, this stopped being "nice-to-have" and became non-negotiable.

Why DynamoDB scan instead of a dedicated vector DB?

Numbers on the current system:

• 21 chunks in the KB
• Each chunk weighs ~5 KB
• Full DDB scan: ~30 ms
• Cosine similarity in Python: ~5 ms
• Total retrieval latency: <100 ms

Alternatives and their minimum cost:

Decision: I don't need a dedicated vector index until Scan crosses ~500 ms. With 500 to 1000 chunks I'd reassess the mechanism.

Coming up in Part 2

EVA today is a linear pipeline:

question -> embed -> retrieve -> gate (yes/no) -> LLM/no-LLM -> answer

After several tests in a controlled environment, this project works, but pieces are still missing and there's room to improve:

Conversation memory

If you ask "who is kevin?" and then "what did he study?", the system treats the second question in isolation, with no idea that "he" refers to Kevin.

Routing

Everything goes through the same flow. Diagnostic questions ("show me your sources"), corrections ("no, ask again but shorter"), or multi-hop queries all take the same linear path.

Reformulation when retrieval fails

If the best score is 0.26 (right above the threshold), the pipeline proceeds even though retrieval is weak. An agent would reformulate the query and try again before invoking Haiku.

Automated model evaluation

Every time I change a threshold, prompt or model, I have to run queries manually and judge by eye whether things still look good.

With that in mind, Part 2 adds a LangGraph orchestration layer that solves all four. When that post goes up, EVA will show that it can be:

• RAG with contextual retrieval (Bedrock + Titan embeddings)
• Agent orchestration with LangGraph: conversation memory, dynamic routing, query reformulation when retrieval scores low
• Multi-layer FinOps controls: semantic caching, relevance gating, budget alarms
• Automated eval suite: golden set of queries + expected answers for regression testing
• Fully deployed via Terraform: Lambda + API Gateway + DynamoDB

See you in Part 2.