An AI agent that actually does the thing.
The brief.
Two hundred support tickets a week. Most about billing: duplicate charges, refund requests, address changes, the usual. The support team was drowning, the model was already costing more than it saved, and demo-able AI was getting in the way of useful AI.
The brief: build the smallest agent that reliably closes one named class of tickets, refunds, with an eval set you can show the auditor, observability you can show the CFO, and a hand-off path that respects the user.
A graph of five typed nodes.
Each node is its own function. Each transition is a typed contract. Failures route to a human, not the user. The whole thing fits on one page.
What we shipped.
A typed LangGraph agent with five nodes, three tools, and a hand-off node that knows when not to act. The model is small where it can be (classify on gpt-4o-mini) and large only where it must be (the resolve step). Every transition is a typed contract. A node returns a structured object, not free-form text, so the next node knows exactly what it's looking at.
The whole thing fits on one page of code. We'll show you.
A test suite for prompts.
Every change runs the eval set in CI. Regressions don't ship. The pass rate is the only metric we actually trust.
We can finally point at the agent in a board meeting and show the numbers.
— Engineering lead, EU client
Ninety days, seven shipped versions.
- v1.0Day 1First shipped prompt. 71% pass.—
- v1.1Day 6Added Dutch + EN templates for tone.+9.0%
- v1.2Day 14Re-rank with reciprocal rank fusion.+5.4%
- v1.3Day 22Lowered hand-off threshold to 0.78.+2.1%
- v2.0Day 41Switched classify to typed function-calls.+4.8%
- v2.1Day 67Cached fetch_order for hot SKUs.p95 → 1.9s
- v2.2Day 90Added cost ceiling alerter (Slack #ops).ops
Most of the cost is in resolve.
Six steps. One of them does most of the thinking. Classify and check_policy stay cheap on purpose. We want the heavy model to only see context it needs.
- classify18%small model · gpt-4o-mini
- fetch_context12%vector search + RAG
- check_policy9%function call · no llm
- resolve47%drafting · main model
- handoff6%summary for human
- audit_log8%tags · cost · trace
The outcome.
Two hundred tickets a week, zero hallucinations in ninety days. Median resolution time on refunds went from eleven minutes (human) to eleven seconds (agent + audit). The support team is back to handling the calls that actually need a human.
Most importantly: the eval set is the contract. Every prompt change is a PR with a CI run. Regressions don't ship. The team has a real, measurable artefact to point at when the question comes up: “does the AI actually work?” The answer is on the dashboard.
