Brian Jagger
I want to take you inside an AI agent and show you exactly what happens between the moment a user submits a request and the moment the agent reports a result. Most explanations I read online stop at “the agent uses a large language model and some tools,” which is true but useless. The interesting question is the one that determines whether your system works in production: how does the model actually decide what to do next, and what happens when it gets it wrong?
How AI Agents Work: Inside the Perception-Reasoning-Action Loop
In this article, I will walk you through the perception-reasoning-action-reflection loop that sits at the heart of every modern agentic system. I will cover the planning techniques that make it work (ReAct, Chain of Thought, Tree of Thoughts), the tool-calling mechanics, the three kinds of memory an agent needs, and the feedback loops that turn a one-shot model call into a system that recovers from its own mistakes. Then I will walk through why agents fail (hallucinated tool calls, lost context, infinite loops) and finish with a worked example of a research agent so you can see the entire flow end to end.
This is the article I wish I had when I started building agentic systems. It is the mental model I rely on every time I review an agent design.
The agent loop: the four steps
Every modern agent, regardless of framework, runs some variant of the same four-step loop:
- Perceive: ingest the current state of the world (user request, tool outputs, memory)
- Reason: decide what to do next based on the goal and the perceived state
- Act: execute the chosen action (call a tool, write a file, query an API)
- Reflect: evaluate the result and update the plan
The loop continues until the agent decides the goal is achieved or determines that it cannot be. The loop is the architecture. Everything else, the model choice, the tools, the memory store, the framework, is implementation detail around this loop.
“If you cannot describe how your system implements each of the four steps in one sentence each, your agent does not have a stable architecture. It has a tangle of prompts and hope.”
I have seen sophisticated production systems that boil down to this loop with careful engineering on each step. I have also seen ambitious-looking architectures that turn out to be missing the reflection step, which is why they fail silently in production.
Perception: what the agent sees
Perception is everything the agent knows at a given moment. In a chat agent, perception is the conversation history. In a coding agent, perception is the open files, the test output, the recent commits. In a browser agent, perception is the DOM and the screenshot.
The hard problem in perception is volume. Modern LLMs have large context windows, but they are not infinite, and even when they are large, performance degrades as the context gets longer. The discipline of perception design is deciding what to include and what to leave out.
I push teams to think about perception in three layers:
- The standing context: the agent’s role, the available tools, the high-level goal
- The recent context: the last few steps and their results
- The retrieved context: information pulled from memory or external sources for this specific step
Most failures I see in production agents come from putting too much in the standing context (it crowds out the recent context), too much in the recent context (it crowds out new information), or too little in the retrieved context (the agent makes decisions without the data it needs).
Reasoning: how the agent decides
The reasoning step is where the LLM call happens. The agent assembles a prompt containing the perception, the goal, and the available actions, and asks the model: what should I do next?
The reasoning prompt has a structure that varies by framework but always contains:
- A system message describing the agent’s role and policies
- The goal or task
- The recent history of actions and observations
- The available tools and their schemas
- An instruction to produce either a tool call or a final answer
The model’s job is to read all of that and emit a structured response. In modern systems, this response is a function call (structured JSON specifying which tool to invoke with which arguments) or a “final answer” signal.
The quality of this step depends on three things: the model, the prompt, and the tool schemas. The model is the easy part to pick. The prompt and the schemas are where the engineering happens.
Action: how the agent affects the world
The action step is where the agent’s intent meets reality. The chosen tool is invoked, the result is returned, and the result becomes part of the perception for the next loop iteration.
The action layer is where I see the most production engineering work. It is not glamorous, but it determines whether the agent is reliable. The action layer must handle:
- Authentication and authorisation
- Input validation
- Rate limiting and backoff
- Timeouts and partial failures
- Idempotency and retry safety
- Audit logging
“The model is the brain. The action layer is the spinal cord. A brilliant brain with a broken spinal cord cannot move.”
I tell teams that the quality of their action layer is the single biggest predictor of whether their agent will survive contact with production. A well-engineered action layer can compensate for a mediocre model. A poorly engineered action layer cannot be saved by even the best model.
Reflection: how the agent recovers
Reflection is the step that turns a one-shot model call into a system that learns within a task. After an action, the agent looks at the result, asks “did this work, did it move me toward the goal, do I need to change my plan,” and updates its approach.
Reflection takes several forms in practice:
- Self-critique: the model evaluates its own output and decides whether to retry
- Verifier models: a separate model judges whether the result is correct
- Tool-based verification: a deterministic check (compile, test, validate) confirms the result
- User-in-the-loop verification: a human approves consequential actions
Cheap, deterministic verification is always preferable to model-based reflection when it is available. If you can compile and test the code, that is more reliable than asking the model “is this code correct.”
The agents I have seen survive in production combine several reflection mechanisms. The ones that fail usually have only one (or none), and they fail in exactly the ways the reflection mechanism cannot detect.
ReAct: alternating thought and action
The ReAct paper (2022) gave us the most influential planning pattern in modern agentic AI. The idea is simple: at each step, the model emits a “thought” (free text reasoning about what to do) followed by an “action” (the tool call). The trace looks like:
Thought: The user wants to know about Q3 sales. I need to query the warehouse.
Action: query_warehouse(sql="SELECT SUM(revenue) FROM orders WHERE quarter='Q3'")
Observation: {"result": 4250000}
Thought: I have the answer. I should format it for the user.
Action: respond(text="Q3 sales totalled 4.25 million.")
ReAct works because the thought makes the model’s reasoning visible, debuggable, and (importantly) more accurate. Modern frameworks have evolved beyond the original ReAct template, but the underlying idea, alternating reasoning and action with explicit thoughts, is the foundation of almost every production agent today.
I cover this pattern and its variants in more detail in ReAct, Plan-Execute, Reflection: agent patterns.
Chain of Thought and Tree of Thoughts
Two related techniques are worth knowing because they show up in agent design.
Chain of Thought (CoT) prompting asks the model to reason step by step before producing the answer. It is the conceptual ancestor of ReAct. For pure reasoning problems (math, logic, complex Q&A), CoT often improves accuracy substantially.
Tree of Thoughts (ToT) generalises CoT by asking the model to consider multiple reasoning paths, evaluate them, and pick the best one. It is more expensive but can produce better results on problems with multiple plausible approaches.
In agentic systems, I rarely see pure ToT used as the planning algorithm, because the cost is prohibitive for production. But ToT-inspired techniques (generating multiple candidate plans, running them in parallel, voting on the result) show up in high-value agents where the per-task budget can absorb the cost.
The practical lesson: use ReAct as your default. Reach for ToT-like approaches only when the per-task value justifies the cost and the reliability gap is large enough to matter.
Tool calling: the practical mechanics
Tool calling is the most-engineered part of any production agent. The mechanics are deceptively simple: define a tool with a name, description, and parameter schema, and the model emits a JSON object specifying which tool to call with which arguments.
The hard parts:
| Concern | What to engineer |
|---|---|
| Tool descriptions | Clear, unambiguous, with examples of correct use |
| Parameter schemas | Strictly typed, with enums where possible |
| Error messages | Returned to the model in a format it can act on |
| Tool count | Keep below 10-20 active tools; route to specialised sub-agents above that |
| Tool composition | Prefer small, composable tools over one large multi-purpose tool |
| Idempotency | Make repeat calls safe wherever possible |
| Authorisation | Enforce at the tool boundary, not in the model |
I have seen agents fail catastrophically because they had 50 tools in the active prompt and the model confused similar-sounding ones. I have also seen agents fail because the tools were so coarse-grained that the model had to chain three of them to do anything useful.
The rule I use: a tool should do one thing, do it well, and have an unambiguous name.
Working, episodic, and semantic memory
Memory is what separates a one-shot model call from an agent. There are three distinct kinds of memory you need to think about.
Working memory is the current state inside the task. The scratchpad, the recent observations, the plan in progress. This typically lives in the context window. Designing working memory means deciding what to keep and what to summarise as the task grows.
Episodic memory is the record of past tasks. What the agent did yesterday for this user, what tools it called, what the outcomes were. This typically lives in a structured store (a database table, an event log) and is queried when relevant context is needed for a new task.
Semantic memory is the agent’s general knowledge, often domain-specific facts or organisational information. This typically lives in a vector store and is retrieved via similarity search.
A well-designed agent uses all three, and uses them appropriately. The most common failure I see is teams treating “memory” as a single concept and dumping everything into a vector store. That works poorly because the retrieval semantics for “what did I just do” are different from “what is the company policy on refunds.”
For the architectural pattern, see agentic AI architecture patterns.
Why agents hallucinate tool calls
The first major failure mode: the agent invents tools that do not exist, or calls real tools with parameters that violate the schema.
Why it happens:
- The tool prompt is unclear or contradictory
- Several tools have similar names and descriptions
- The model’s training contains examples of similar but different tools
- The tool schema is permissive enough that the model improvises
How to prevent it:
- Use strict structured outputs (JSON mode, function calling, structured outputs)
- Validate at the tool boundary and return clear error messages
- Reduce the number of tools in the active prompt
- Test with adversarial inputs before going to production
“Every hallucinated tool call is a debugging signal. The model thought the tool existed because something in your prompt suggested it should. Find that something and fix it.”
Why agents lose context
The second major failure mode: the agent forgets information that was relevant earlier in the task, often because the context window filled up with intermediate results.
Why it happens:
- Long tool outputs crowd out important context
- Multi-step plans push earlier reasoning out of the window
- Memory summarisation is too aggressive
- The agent does not know when to retrieve from episodic memory
How to prevent it:
- Summarise tool outputs to fixed sizes when possible
- Use a structured plan that survives across steps
- Implement explicit memory tools the agent can call
- Add reflection prompts that ask “do I still remember the goal”
The cleverest agents I have seen explicitly track their plan as a state object that gets passed forward through every step. The model updates the plan as it learns more, but the plan itself never disappears from the context.
Why agents get stuck in loops
The third major failure mode: the agent calls the same tool over and over, or alternates between two tools indefinitely, without making progress.
Why it happens:
- The reflection step does not detect lack of progress
- The model is overconfident in its current strategy
- The error messages from tools do not give the model enough information to change course
- The budget is unlimited
How to prevent it:
- Hard step budgets per task (with a graceful failure when exceeded)
- Detection logic for repeated tool calls with similar arguments
- Reflection prompts that explicitly ask “am I making progress”
- Escalation paths to a human or a more capable model when stuck
The most important guardrail is a hard step budget. An agent that has run for 50 model calls without producing a result is almost never going to produce a good result with another 50. Cut the loop, log the failure, and let the human take over.
A worked walkthrough: a research agent
Let me walk through a concrete example to make the entire loop concrete.
Task: “Research the top 3 competitors in the European EV charging market and produce a 500-word summary.”
Step 1, perception: the agent reads the task, sees its tools (web_search, fetch_url, summarise, respond), and notes its goal.
Step 2, reasoning: thought: “I need to identify EV charging companies operating in Europe. I should search the web.” Action: web_search(query=“top EV charging companies Europe 2026”).
Step 3, action: the search returns 10 results. The action layer formats them and returns them to the agent.
Step 4, reflection: the agent notes that several names appear repeatedly. It decides on three candidates.
Step 5, next iteration: thought: “I need to fetch the company pages for each.” Action: fetch_url(url=…) for each.
Steps 6 to 10: the agent fetches pages, summarises each, identifies key facts.
Step 11: thought: “I have the data. I should structure the summary.” Action: respond(text=“…”).
Throughout, the agent updates a structured plan (companies found, pages fetched, facts extracted) that persists across steps. Working memory holds the current step’s data. Episodic memory records the search trace. Semantic memory (if any) might hold prior research the agent did on adjacent markets.
The reflection step at each iteration asks: am I making progress, do I have what I need, is there a better next step. The hard step budget (say, 30 model calls) bounds the worst case.
This is what every well-designed agent does. The details vary by framework, but the structure is identical.
The observability stack you need
You cannot operate an agent you cannot see. The observability stack for agents has specific requirements that go beyond standard application monitoring.
What I require for any agent going to production:
- Per-task trace: every model call, every tool call, every observation, every reflection
- Cost tracking: tokens in, tokens out, model used, cost in dollars per task
- Outcome tracking: success vs failure vs human-escalated, with reasons
- Latency: end-to-end per task, broken down by step
- Tool-level metrics: success rate, latency, error rate per tool
- Evaluation harness: a set of representative tasks run on every release
Frameworks like LangSmith, Langfuse, and Arize Phoenix are designed around exactly this requirement. Pick one, instrument from day one, and treat the traces as a first-class asset. The teams that succeed with agents are the teams that look at their traces every day.
