How to Build an AI Agent That Holds Up in Production

Building an AI agent that demos well has never been easier. Describe a task, wire up a model, give it a couple of functions to call, and within an afternoon you have something that answers questions and takes actions. It looks like magic.

Putting that same agent in front of real users, on real channels, touching real data, is a different discipline entirely. The gap is not the model. The gap is the engineering around it: how the agent is structured, how its tools are built, how its behaviour is measured, and how it fails safely when something goes wrong. This is a practitioner's blueprint for that engineering, the decisions that decide whether an agent survives contact with production.

Separate the brain from the channel

The most important architectural decision comes first: the agent's core engine, the part that manages the conversation, decides what to do next, and calls tools, must be completely decoupled from where the conversation happens.

Users arrive from everywhere. WhatsApp, Telegram, a web chat widget, SMS, maybe a voice line later. Each of those is just a transport. The engine should neither know nor care which one a message came from. Build it as a standalone service with a plain interface: messages come in, responses and actions go out. Every channel is then a thin adapter that translates that channel's format to and from the engine's interface.

The payoff is large. You can add a new channel without touching the agent's logic. You can test the engine with no channel attached at all. And you get the same agent behaving consistently everywhere, instead of three subtly different bots drifting apart.

Keep the model behind an interface too, for the same reason. Good models are released constantly, and they get cheaper and more capable as they do. If switching the model behind your engine is a configuration change rather than a rewrite, you can chase quality and cost freely. Hardcoding one provider's SDK through your business logic is a decision you will regret within a quarter.

That interface also lets you use more than one model at once. Not every turn needs your most capable model: a fast, inexpensive one can handle intent classification, routing, and simple replies, while the frontier model is reserved for the hard reasoning and multi-step tool calls. Routing by difficulty, sometimes called a model cascade, trims both cost and latency without the user ever seeing the seams.

Design for the mess of real channels

Real conversations do not arrive as tidy turns. On WhatsApp, Slack, or SMS, people double-text, edit a message a second after sending it, and fire three more while the agent is still working on the first. Design for that from the start.

Route each user's messages to a consistent session, so their context is not split across parallel workers. Put an idempotency key on any action a tool takes, so a retried or duplicated request books one appointment rather than three. And when a new message lands mid-response, decide deliberately whether to queue it, cancel the in-flight work, or fold it into the current turn. An agent that cheerfully runs the same irreversible action twice because someone got impatient is not ready for production.

Make every tool a first-class, auditable unit

An agent is only as useful as the tools you give it. The engine supplies language and orchestration; the tools do the real work: look something up in a knowledge base, read a calendar, draft an email, search the web, raise a ticket. Give the agent tools that genuinely fit your use-case, and give it good ones.

The mistake is to bury tool logic inline in the agent loop. Treat each tool instead as a configurable, independently testable, audited unit:

• Configurable and manageable. Tools should be registered and switched on or off by configuration, so one use-case can run with a different tool set from another, and adding a tool never means surgery on the engine.
• Independently testable. A tool is a plain function with typed inputs and outputs, so it can be unit-tested on its own, with no model in the loop. This matters more than it sounds: most agent bugs live in the tools, not in the reasoning.
• Audited. Record every tool call, its inputs, its outputs, and its result. You need that record to debug, to prove what the agent did, and to trust it in the first place.

Two details save a lot of pain here. Validate a tool's inputs and outputs against a strict schema (Zod, Pydantic, or your language's equivalent) so a malformed call fails loudly at the boundary instead of corrupting something downstream. And give every tool a timeout, because an external API that hangs should degrade the agent gracefully, not freeze the whole conversation.

Then comes the move that separates reliable agents from flaky ones: find every deterministic point in the conversation and turn it into a tool. Anywhere the correct action is not a matter of judgement, checking an order status, confirming availability, calculating a quote, looking up a policy, do not ask the model to reason its way there. Hand it a tool. The model decides when to call the tool; the tool decides what the answer is. Every deterministic decision you move out of the model and into a tool is one less place the agent can be confidently wrong, and one more piece of behaviour you can reproduce and test.

Give it memory, with a store that expires

A stateless agent forgets the previous message the moment it sends a reply. To hold a real conversation, the engine has to feed the model the history each turn, so it always has the full context of what has been said and done.

Store that history keyed by conversation, and put a time-to-live on it. You rarely want raw chat transcripts living forever; they cost money, they carry privacy risk, and old context stops being relevant. A key-value store with native TTL is the natural home for this. DynamoDB fits it neatly: fast lookups by conversation id, and a TTL attribute that quietly expires stale items with no clean-up job to maintain.

As a conversation grows long, do not simply send everything every turn. Summarise or trim the older turns. Left unchecked, a long history inflates cost and dilutes the model's focus on what actually matters now.

Keep this short-term memory separate from long-term knowledge. The conversation history is working memory: the recent turns the model needs to follow the thread. The facts, documents, and past interactions the agent should be able to recall are something else, and they do not belong in the prompt at all. Store them separately and pull in only the relevant pieces on demand, the retrieval-augmented pattern, so the agent stays grounded without hauling its entire past into every request.

Spend your time on the system prompt

Of every decision here, the system prompt is the single biggest differentiator between an agent that feels sharp and one that feels generic. The model itself is a commodity; everyone has the same one. Your data, your tools, and your system prompt are what make the agent yours.

The system prompt is where you encode the agent's role and personality, its boundaries, how and when to use each tool, what it must never do, and how to handle the messy edges that are specific to your domain. This is not a paragraph you write once. It is a real artefact: version it, change it deliberately, test every change against your evals, and expect to keep refining it for as long as the agent is running. It is never finished, and the time you invest here pays back more than anywhere else.

Cache the parts that do not change

Your system prompt and tool definitions are large, and they are almost identical on every single turn. Sending them in full to the model each time is slow and expensive.

Prompt caching solves this: the provider reuses the unchanged prefix of your request, the system prompt, the tool schemas, the stable early context, instead of reprocessing it on every call. On a chatty agent that can be a dramatic saving in both latency and cost. The rule of thumb is to keep everything static at the front and everything dynamic at the end: the more of your prefix stays byte-for-byte identical between calls, the more the cache can reuse, so the only thing that really changes from turn to turn should be the latest message.

Audit everything, and put guardrails around it

An agent that takes actions raises the stakes above a chatbot that only talks, so the operational discipline has to rise with it.

Log everything: every message in and out, every tool call and its result, every decision the model made, and which model and prompt version produced it. When something goes wrong, and it will, that trace is the difference between a five-minute fix and an afternoon of guesswork.

Then wrap the whole thing in guardrails. Validate tool inputs before they run. Constrain what each tool is allowed to touch, on a least-privilege basis. Put a human in the loop for anything irreversible or high-stakes: moving money, deleting records, sending a customer an email without review. Add rate limits and a hard cost ceiling too, a tripwire that halts the agent if it starts looping or burning tokens far faster than any real conversation would.

Treat every piece of data a tool returns as untrusted, because this is the attack that matters most for tool-using agents. A web page, a customer email, or a document in your own knowledge base can carry instructions aimed at the model, and an agent that follows them has been hijacked into leaking data or calling tools it never should. External content is information to reason about, never commands to obey. Keep it clearly fenced off from your own instructions, and never let text the agent has merely read authorise an action on its own.

Check outputs too, for the failure modes you actually care about, whether that is leaking data, wandering off-topic, or promising something the business cannot deliver. The audit trail and the guardrails together are what let you put an agent in front of real people and trust it there. It is the same discipline that turns any prototype into a product: the rigour goes in deliberately, because nothing surfaces the gaps faster than real users.

Evals are the whole game

This is the part most teams skip, and it is the part that decides whether you have a product or a demo.

You cannot eyeball your way to a reliable agent. "It seemed fine when I tried it" is not a quality bar; it is a hope. What you need is evals: a growing library of test cases, each with a grader that scores the agent's response automatically. Build them from the first day and keep building them as you go. Every bug you find and every strange edge a user hits becomes a new eval case, so that failure can never quietly return.

Graders take a few shapes. Exact-match works for deterministic tool outcomes. Rubric-based grading, often done by a model, works for open-ended responses. Property checks answer specific questions: did the agent call the right tool, did it stay in scope, did it refuse what it was supposed to refuse. Use whichever fits the behaviour you are protecting.

The reason evals matter so much is leverage. With a solid eval suite you can switch the model, rewrite the system prompt, or add a tool, and know within minutes whether the agent got better or worse. Without one, every change is a gamble and every release is a leap of faith.

Wire the suite into your CI/CD pipeline so it runs on every change, exactly as you would run unit tests. A prompt tweak or a model upgrade that drops your scores should fail the build rather than reach users. Evals only protect you if they run without anyone having to remember to run them.

So it is worth being blunt about it: evals, evals, evals. The teams whose agents hold up are the ones who measure, relentlessly.

The shape of a production agent

Put it together and the picture is consistent. A core engine that is completely separate from the channels in front of it. A model that sits behind an interface so it can be swapped at will. A set of configurable, independently testable, fully audited tools that soak up every deterministic decision in the conversation. Conversation memory in a store that expires on its own. A system prompt you treat as your most valuable artefact. Caching to keep it fast and affordable. Audit and guardrails so it fails safely. And evals wrapped around all of it, measuring every change.

The model was the hard part for about a year. It is the easy part now. What separates an agent that impresses in a demo from one you can actually run is the engineering around it, and that engineering is entirely within your control.