Delivery and AI transformation leader. 15+ years running large programs; I also build and run the AI systems below.

Most AI work is shown as a demo. This is a personal portfolio of some of the projects I have built, and what happened after.

These are personal projects, built and run on my own to show my skills in applied AI: they are not client work. Each one is documented from its own repository, with the problem, the scope, the architecture, the design decisions and the results.

Photo of Jorge Roberts
3initiatives documented
2in production
433LLM requests in 30 days
2.0Mtokens in 30 days

Selected initiatives

Personal projects, designed, built and run end to end.

Bot · since 2026-07 · In production

HealthBot

Personal health coach on Telegram that logs food, training and sleep from natural language or photos, and produces daily, weekly and monthly plans using LLMs, RAG over cited evidence and deterministic guards on every number.

  • Cascading monthly, weekly and daily reports that adjust nutrition, sleep, training and supplementation targets on their own; 67 reports generated and stored.
  • RAG over 589 videos and 17,588 chunks, with one vector index per source and principles distilled across sources.
  • 70 deterministic guard tests that validate every number the model produces.
  • Food vision where the model identifies and weighs and the code does the math; the same meal gives the same number by photo and by text.
  • End-to-end OpenTelemetry traces (bot, gateway, provider) of every run and every report, in Jaeger and Langfuse.
  • Retrieval quality measured with a frozen eval dataset; hit@5 raised from 0.64 to 0.74.
Scale
589 videos and 17,588 indexed chunks; bot, API and an Android app
LLM, 30 days
120 requests, 350,507 tokens
Verified
2026-10-30
CI
passing (2026-11-20)

rag · vision · structured-extraction · llm-reports · deterministic-guards · tier-escalation · observability · node · sqlite · sqlite-vec · express · telegram · python · docker · opentelemetry

Technical overview
  • Platform: Node.js on Docker; Telegram bot over long polling (node-telegram-bot-api), an Express HTTP API (fail-closed Bearer auth with attempt limits) for the Android app and the Health Connect receiver, and a cron scheduler in local time.
  • Data: SQLite with better-sqlite3 (38 tables) and sqlite-vec for three vector indexes, one per corpus source; migrations at startup.
  • LLM: text goes through the self-hosted gateway with its own key and policy (llm-policy.yaml), via @homelab/llm-client. Tiers: tier-2 for chat, tier3-tools for the daily briefing, tier3-reasoning for weekly and monthly reports, tier-vision for food photos. Embeddings with NVIDIA NIM and backup vision with Google Gemini.
  • RAG: producer-consumer ingestion of YouTube transcripts (Python + Node), embeddings, per-source retrieval and distilled principles; eval with a frozen dataset (hit@5).
  • Integrations: Telegram, Health Connect (through the Android app), YouTube (transcripts), LLM Gateway.
  • Observability: end-to-end OpenTelemetry (bot, gateway, provider) into Jaeger and Langfuse.
  • Quality: 70 deterministic guard tests plus custom calibration tools for photos, macros and RAG.
  • Deploy: a service in the homelab docker-compose, script-only deploys, with a tagged rollback image.
Problem and scope

Problem

Following a nutrition and training plan requires logging a lot (meals, water, loads, weight, sleep) and, above all, cross-checking that data to decide what to adjust. Commercial apps log but do not reason, and a generic LLM chat reasons but does not remember, makes up numbers and has no access to the watch data.

HealthBot is a personal health coach: you talk to it in natural language on Telegram (or send a photo of the plate), it logs what applies, and every day it builds a plan against the real history, with evidence cited from a corpus of science communication. The design premise is that the LLM contributes judgement, but every number that can be calculated or verified is calculated and verified by code.

Scope

  • Logging by natural language, commands and photos: food, water, weight and measurements, workouts (sets, reps, load, RIR, drop sets and supersets), supplements, subjective check-in, lab results.
  • Watch data ingestion through Health Connect (sleep, HRV, heart rate, steps, calories, exercise sessions).
  • Macro estimation from text and from photos (plate or nutrition label).
  • Cascading monthly → weekly → daily reports that adjust targets.
  • Dynamic routine with deterministic load progression and mesocycles.
  • RAG over transcripts from Huberman Lab, Dr. La Rosa and a backup source, with distilled principles.
  • Read and write HTTP API for the app.
  • End-to-end traces of every run.
Architecture and decisions

Diagram

flowchart LR
  user[User] -->|chat and photos| tg[Telegram]
  tg --> bot[Bot and handlers]
  app[HealthApp Android] -->|API with token| api[HTTP API]
  hc[Phone Health Connect] --> app
  bot --> guards[Deterministic guards]
  api --> guards
  cron[Report scheduler] --> guards
  guards --> db[(SQLite + vector indexes)]
  bot --> rag[RAG over video corpus]
  cron --> rag
  api --> rag
  rag --> db
  bot --> gw[LLM Gateway]
  cron --> gw
  api --> gw
  rag -.->|embeddings, direct for now| emb[Embeddings provider]
  bot -.->|backup vision, direct for now| vis[Vision provider]
  bot --> otel[OpenTelemetry traces]
  gw --> models[Local and free models by tier]

Design decisions

  • The prompt asks, the guard guarantees. Everything that can be checked against the user's text or a plausible range is checked in code; the LLM is left with judgement. Every guard has its test.
  • The model identifies and weighs; the code calculates. A photo says what is there, not how much it weighs: vision returns items and grams, and calories come from the same function the text path uses.
  • Machine-readable directives. Reports emit AJUSTE, RUTINA and MESOCICLO as structured blocks, with a narrow extraction as a safety net.
  • A cascade with a single writer per field. Monthly and weekly write the adjustment; the daily report only reads it with COALESCE.
  • Real rotation instead of a calendar. What to train today comes from the workout history, with detection of breaks.
  • One vector index per source. Distances across indexes are not comparable; kept apart, each source can change embedding model without affecting the others.
  • Thin client. The app has no logic and no LLM keys: every number comes from a single function on the server.
  • Only validated output goes out. A report that does not pass validation is regenerated before it is sent.
  • Dates in a single module. The container runs in UTC and the day is always calculated in local time, in one place.
Results
  • In daily use since July 2026, with 67 cascade reports generated and stored (49 daily, 15 weekly, 3 monthly) and target adjustments applied automatically.
  • Corpus of 589 videos and 17,588 chunks indexed in three vector indexes, one per source.
  • 70 automated guard tests that validate every number before it reaches the user.
  • The same meal gives the same number by photo and by text: vision identifies and weighs, and calories come from the same function as text.
  • Food logging by text from the app in under a second.
Evidence

src/shared/macros.js, line 301

export function calcularDesdeItems(parsed, desc, origen = 'texto') {
  // Lo que el usuario conto explicitamente no se estima: se multiplica. Ver
  // cantidadesExplicitas() — "dos huevos" son 2 x 55 g, no lo que el modelo recuerde.
  const explicitas = cantidadesExplicitas(desc);

  const items = (Array.isArray(parsed?.items) ? parsed.items : [])
    .filter(i => Number.isFinite(i?.gramos) && i.gramos > 0)
    // Antes de cualquier cuenta: si el peso es en realidad una cantidad, se corrige.
    .map(i => ({ ...i, gramos: pesoDeUnidades(i?.nombre, i.gramos) }))
    .map(i => {
      const match = explicitas.find(e => referenciasMencionadas(i?.nombre || '')[0]?.id === e.ref.id);
      if (!match) return i;
      const exacto = match.cantidad * match.ref.unit_grams;
      if (Math.abs(exacto - i.gramos) / exacto > 0.1) {
        logger.info({ item: i.nombre, modelo: i.gramos, contado: exacto, cantidad: match.cantidad },
          'el usuario dijo la cantidad: el peso sale de multiplicar, no de estimar');
      }
      return { ...i, gramos: exacto, _contado: true };
    })
The model identifies and estimates weight; the code does the math. If the person stated the amount, the weight comes from multiplying, not from estimating.

Infrastructure · since 2026-06 · In production

LLM Gateway

Self-hosted gateway that routes requests from several apps to local and free models by capability tier, with failover, per-app guardrails and end-to-end tracing in Jaeger and Langfuse.

  • 8 model groups across 5 providers, with no per-call cost.
  • Four model retirements absorbed by changing configuration only, without touching any app.
  • Prompt-injection guardrails, with each app's policy declared and versioned in its own repository.
  • End-to-end traces (app, gateway, provider) with cost, latency and errors per call in Jaeger and Langfuse.
Scale
8 model groups across 5 providers
LLM, 30 days
433 requests, 1,965,099 tokens
Verified
2026-10-30
CI
passing (2026-11-20)

model-routing · failover · guardrails · prompt-injection-defense · observability · python · litellm · postgres · redis · docker · opentelemetry

Technical overview
  • Platform: LiteLLM Proxy (OpenAI-compatible API) on Docker, with a custom Python callback (async_pre_call_hook) that applies each key's policy before routing.
  • Providers and models by tier: local Ollama (qwen2.5:0.5b, llama3.2:1b) for cheap classification; Groq (gpt-oss-20b, gpt-oss-120b, qwen3.6-27b for vision); NVIDIA NIM (llama-3.1-8b, nemotron-3-nano-30b-a3b for function calling, nemotron-3-ultra-550b-a55b for reasoning, llama-3.2-11b-vision, nemotron-3-embed-1b for embeddings); Google Gemini (3.6-flash); OpenRouter (llama-3.3-70b:free).
  • Data: PostgreSQL 16 for virtual keys, policy metadata and spend logs; Redis as response cache (1 h TTL).
  • Governance: one virtual key per app; tools/policy-sync reads the llm-policy.yaml versioned in each repo and applies it to its key (timeout, sanitize, tools, purpose, RPM/TPM). Fallbacks filtered per key; public-only groups return 403 unless explicitly enabled.
  • Observability: OpenTelemetry (OTLP) into a custom collector with a persistent queue, fanning out to Jaeger (local, Badger, 30 days) and Langfuse Cloud.
  • Clients: shared @homelab/llm-client package (Node.js) used by the homelab apps.
  • Quality: guardrail unit tests inside the image and an end-to-end suite with an isolated LiteLLM, Postgres and a custom echo provider, with no quota usage.
  • Deploy: own image built from the repo, a service in the homelab docker-compose; changes are tested on a separate instance and rollback means reverting the config to the previous commit.
Problem and scope

Problem

Several personal bots and apps need LLMs with different requirements (tool use, reasoning, vision, embeddings, sensitive or public data) without paying per call. Talking to each provider directly scatters API keys, makes it impossible to measure usage or quality, and every time a free provider retires a model a different app breaks.

Scope

  • One OpenAI-compatible API for every app.
  • Capability tiers, each with several providers in order and automatic failover.
  • One virtual key per app, with allowed models, limits and its own guardrail policy, declared in the app's repository.
  • Generic prompt-injection guardrails applied at the proxy.
  • Caching, OpenTelemetry traces and usage logging.
Architecture and decisions

Diagram

flowchart LR
  subgraph apps[Apps]
    hb[healthbot]
    ad[ai-digest]
    hub[hub]
    wa[Walle]
  end
  subgraph gw[LLM Gateway]
    g[Guardrails<br/>policy per key]
    r[LiteLLM router<br/>tiers + failover]
    c[(Redis cache)]
    db[(Postgres<br/>keys + usage)]
  end
  hb & ad & hub & wa -->|virtual key| g --> r
  r --- c
  g --- db
  r --> ol[Local Ollama]
  r --> groq[Groq]
  r --> nim[NVIDIA NIM]
  r --> gem[Gemini]
  r --> orr[OpenRouter]
  r -.OTLP.-> otel[OTel Collector] --> jg[Jaeger] & lf[Langfuse]

Design decisions

  • One gateway instead of one SDK per provider: a model change is solved in a single file and the apps do not change.
  • Tiers by capability, not by provider: the app asks for "reasoning", not for "NVIDIA".
  • Each app decides its input length: every project knows the legitimate size of its prompts, so the limit lives in its policy, not in the gateway.
  • The gateway enforces the maximum duration per key: an app cannot ask for more time than its policy allows.
  • Tag escaping by allowlist: every < that could open a tag is escaped, while ->, >99 and <100 pass through.
  • A separate group for public data: allows using providers that log requests without exposing personal data.
  • Guardrails in two layers (ADR 0001): a versioned policy in each app, plus specific controls in each app's code.
Results
  • 8 model groups across 5 providers, all free or local.
  • Four model retirements (NVIDIA, Groq, OpenRouter) absorbed by changing configuration only, without touching the apps.
  • A single end-to-end trace (app → gateway → provider) in Jaeger and Langfuse, with cost and latency per call.
  • Automatic failover across providers, filtered by what each key is allowed to use.
Evidence
Timeline of a trace with spans from two services, the app and the gateway
Real trace of a daily report: the app opens the trace, the gateway joins it and the router recovers on its own from a provider error (red mark) without the app noticing. The 1.8 minutes belong to the reasoning model that writes the report in the background.

litellm/guardrails.py, line 116

def _restrict_fallbacks(fallbacks, key_models):
    """Cadena de fallbacks del router limitada a lo que la key puede usar.

    key_models vacio = la key puede usar cualquier grupo, salvo los solo-publicos.
    Devuelve la lista en el mismo formato del router ([{origen: [destinos]}]).
    """
    allowed = set(key_models or [])
    public_key = bool(allowed & PUBLIC_ONLY_GROUPS)
    ok = (lambda m: m in allowed) if allowed else (lambda m: m not in PUBLIC_ONLY_GROUPS)
    out = []
    for item in fallbacks:
        if not isinstance(item, dict):
            continue
        for src, targets in item.items():
            if not ok(src):
                continue
            keep = [t for t in (targets or []) if ok(t) and (public_key or t not in PUBLIC_ONLY_GROUPS)]
            if keep:
                out.append({src: keep})
    return out
The fallback chain is filtered per request: failover never lands on a model the key is not allowed to use.

Service · since 2026-10 · In development

AI Digest

Weekly AI newsletter produced by a six-stage agentic pipeline, from reading more than 25 sources to an email with up to 20 validated news items.

  • Six-stage pipeline where code validates and the model only decides what requires judgement.
  • From about 260 items a week to 20 at most, with no URL or figure that is not in the source.
  • A model chosen per stage by measuring against a fixed fixture, not by intuition.
  • Every LLM call traced through the gateway in Langfuse and Jaeger, with cost, tokens and latency.
  • Page fetching with SSRF protection, and new sources added only with human approval.
  • 28 network-free tests, including an end-to-end run with a fake LLM.
Scale
25+ sources and up to 20 news items per edition
LLM, 30 days
220 requests, 1,600,463 tokens
Verified
2026-10-29
CI
passing (2026-11-20)

agentic-pipeline · curation · summarization · source-discovery · output-validation · node · nunjucks · readability · docker

Technical overview
  • Platform: Node.js in a short-lived container per run (own image), triggered by cron on Mondays; CLI with a lock, --dry-run, --since and --month.
  • Sources: more than 25 across RSS/Atom feeds, Hugging Face, arXiv and GitHub, declared in a versioned registry (sources.yaml).
  • Processing: Mozilla Readability for full text, downloads with custom SSRF protection (ports 80/443 only, no private IPs, redirects revalidated), date and duplicate filters.
  • LLM: everything through the self-hosted gateway with the app-ai-digest key; tier-2 and tier3-public (a group reserved for public data). Streaming, JSON extraction and schema validation, automatic retries and a 45,000-character cap per request.
  • Prompts: a versioned skill per stage (triage, enrich, compose, discover), with external content always inside tags and never in the system prompt.
  • Deterministic validation: URLs and figures checked against the input; per-section limits.
  • Observability: every LLM call goes through the gateway and is traced in Langfuse and Jaeger (cost, tokens and latency per call).
  • Delivery: HTML rendered with Nunjucks, sent over SMTP to fixed recipients, an archive of every edition and alerts through the homelab notifier (Telegram).
  • Quality: 28 tests with node --test, network-free, including an end-to-end run against a fixture with a fake LLM.
Problem and scope

Problem

Keeping up with AI means reading many scattered sources, and most of what gets published is noise or repetition. The goal is that every Monday morning a newsletter-style email arrives ("Weekly report") with up to 20 news items from the week, each one linking to the original source with a short take on why it matters.

Editorial focus: news with practical value, not "AI news". It prioritizes new models (capabilities and pricing), papers and benchmarks, practices and methods for working with agents, skills, agent security and repositories. It discards funding rounds, people moves, controversies and corporate announcements with no technical substance.

Secondary goal: make the project a practical example of a closed agentic pipeline, with a versioned skill, tier routing in LiteLLM, output validation and observability.

Scope

Includes: the weekly and monthly jobs, a --dry-run mode in both, manual runs for a week (--since) or a month (--month), and an archive of every edition.

Does not include

  • LinkedIn and X. They offer no feed and scraping them goes against their terms.
  • Paid sources or sources behind a login.
  • Republishing full texts.
  • Adding sources without human approval.
Architecture and decisions

Diagram

flowchart LR
  subgraph weekly[Weekly job]
    c[collect<br/>feeds, HF, arXiv, GitHub] --> t[triage<br/>batch prefilter + selection]
    t --> e[enrich<br/>full text + cited facts]
    e --> co[compose<br/>newsletter as JSON]
    co --> v[validate<br/>URLs, figures, limits]
    v --> r[render + send]
  end
  subgraph monthly[Monthly job]
    d[discover<br/>sources cited by the registry]
  end
  reg[(sources.yaml)] --> c
  r -. links and health .-> d
  d -. candidates, human approval .-> reg
  t & e & co & d -->|own key| gw[LLM Gateway]
  r --> mail[Weekly email]
  r -. failures only .-> n[Notifier]

Design decisions

  • A closed six-stage agentic pipeline, with code wherever things can be verified (fetching, filters, validation, rendering) and the model only where judgement is needed (selection, reading, writing).
  • Validation decides, not the model: no URL or figure reaches the email unless it is in the input.
  • Discovery through citations between registry sources, with no web search and with human approval: the registry cannot be poisoned.
  • A model per stage chosen by measuring against a fixed fixture: each stage uses the one that performed best (the reasoning model for prefiltering and writing).
  • Resilient delivery: if a source or a stage is unavailable, the edition still goes out and flags it.
Evidence
First screen of the weekly newsletter with highlights and the models section
Real edition produced by the pipeline: every item with its source, and no URL or figure that is not in the input. It is No. 35 because numbering continues from the reference edition used as a model.

src/validate.js, line 38

function keepSupported(text, pool) {
  let dropped = 0;
  const kept = sentences(text).filter((s) => {
    const ok = significantNumbers(s).every((tok) => [...readings(tok)].some((r) => pool.has(r)));
    if (!ok) dropped++;
    return ok;
  });
  return [kept.join(' '), dropped];
}
Figure validation: a sentence with a number that does not appear in the sources is dropped before sending.

Papers

Methods written for other teams to apply, with their evidence and references.

Methodology paper · v1.0 · October 2026 · 22 pages · EN

From Scrum to Spec-Driven Development

A method for teams already working with Scrum: how to apply it, how to reuse its ceremonies and how to measure the maturity of the adoption.

When an AI agent generates most of the code, the cost of building drops by orders of magnitude, while deciding what to build and validating what was built still depends on people. Scrum was designed on the opposite cost structure. The paper presents Spec-Driven Development in operational terms: the work cycle with its artifacts and gates, how each Scrum ceremony is converted, the metrics that replace story points and velocity, why it is not a return to waterfall, and a three-level maturity model (spec-first, spec-anchored and spec-as-source). It includes a complete example in a mobile app, an eight-week transition plan and a catalog of failure modes.

spec-driven development · Scrum · AI agents · design by contract · quality gates · maturity model

How to cite

Roberts, J. (2026). From Scrum to Spec-Driven Development (version 1.0).

© 2026 Jorge Roberts. All rights reserved. Authorship is also stored in the PDF metadata.

Posts

What I write about delivery with AI agents. Published on LinkedIn, mirrored here automatically. Also via RSS.

Story points measure volume. Measure validation instead.

2026-11-19

Story points measure volume. Measure validation instead.

Full text

Most teams still plan sprints in story points. Once an AI agent writes most of the code, I think that number stops telling you much.

Generating code got cheap. Checking that it's right didn't.

For 15 years I ran delivery where velocity was a decent proxy. Building was the slow, expensive part, so counting what got built made sense.

Now an agent can produce a feature in an afternoon. A team can double its points and, in the same sprint, pile up rework, drift between spec and code, and review debt that never shows up on the board.

So the question I ask a team has changed. How much of what you built was accepted the first time someone reviewed it?

Across the 70+ AI projects I've worked with this year, these are the numbers I look at, with the bar I set: - first-pass acceptance: 80% or more of agent output approved without rework - rework: under 20% of effort going back into work already marked done - validation latency: finished work waits less than 1 day for a person to check it

None of them reward volume. A team that generates faster than it can verify sees them slip within 2 or 3 sprints, long before velocity says anything is wrong.

When building is nearly free, judgement is the scarce resource. That's what the plan should measure.

If you run teams with agents in the loop: what replaced story points in your planning, if anything did?

#SoftwareDelivery #AIAgents #EngineeringManagement

Human review should follow risk, not volume.

2026-11-17

Human review should follow risk, not volume.

Full text

65% to 78%. That's the number Anthropic chose to report about how its team reviews AI-written code.

Not lines reviewed. Not velocity. The share of pull requests that pass after a single review round.

This week the Claude Code team explained how they review code now that agents write a growing share of it. From a delivery point of view, two decisions stood out to me.

First, human review follows risk, not volume. Low-risk changes can merge without human approval. Medium-risk changes need one person. High-risk changes need two, and sensitive areas like CI always get a human.

Second, review loops are capped. After three rounds, the author is asked to rethink the implementation instead of patching it again. PRs that needed five or more rounds dropped from 8.9% to 1.5%.

That second rule is the one I see missing most often. Endless review rounds are where AI speed quietly turns into rework, and nobody's board shows it.

The takeaway for delivery leaders: decide in advance where human judgement is required, and measure whether work passes the first time.

The person directing the agent is still accountable for the code. Governance just decides where that person's attention goes.

How does your team decide which AI-generated changes still need two human approvals?

#SoftwareDelivery #AIAgents #EngineeringManagement