Section I
AI & Agents

How AI Agents Work: Tools, Memory, and Planning Explained

Learn how AI agents combine LLM reasoning, JSON tool calling, short- and long-term memory, and ReAct loops to complete multi-step tasks safely.

MA
Mudasir Ali
13 min read
Editorial illustration of an AI agent reasoning loop connecting planning, tool calling, and memory modules
Overview: Editorial illustration of an AI agent reasoning loop connecting planning, tool calling, and memory modules

An AI agent is a software system that pairs a Large Language Model (LLM) with external tools, working memory, and a repeating control loop (Perceive → Plan → Act → Observe) so the model can decompose goals, execute real actions, and self-correct until a task is finished.

Key Engineering TakeawaysSUMMARY

Unlike a single-turn chatbot, an autonomous agent treats the LLM as a reasoning component inside a runtime loop—trading higher token usage and latency for dynamic problem solving.

4 Pillars

Common Core Modules

LLM + Tools + Memory + Control Loop

5x–18x*

Illustrative Token Range

Depends on depth, context, tools & caching

ReAct

Foundational Loop Pattern

Interleaved Thought & Action traces

Standard Chatbot vs. Workflow vs. Autonomous AI Agent

Teams frequently confuse prompted chatbots, deterministic workflows, and autonomous AI agents. In Anthropic's engineering guide on building effective agents, workflows are defined as systems where LLMs and tools are orchestrated through predefined code paths, whereas agents let the model dynamically direct its own processes and tool usage.

AttributePassive LLM ChatbotWorkflow AutomationAutonomous AI Agent
Control FlowSingle prompt → single replyPredefined DAG or code pipelineDynamic Think → Act → Observe loop
External ActionsNone (text generation only)Fixed API steps defined by developerModel selects tools & JSON arguments at runtime
State & MemoryChat transcript in contextExplicit variables passed between stepsWorking context + optional persistent store
Error RecoveryStops and asks the userRetries or fails via try/catchReads error output, adjusts plan, and retries
Token Cost & Latency1x baseline (hundreds of ms)Predictable (fixed multi-call pipeline)Variable (depends on loop depth, outputs & caching)
Best Use CaseDrafting, Q&A, summarizationInvoices, CI pipelines, fixed ETLDebugging code, deep research, multi-app ops

The 4 Core Pillars of AI Agent Architecture

Most common agent architectures—whether built with raw API calls, LangGraph, or an IDE assistant—assemble the same four core components, though exact implementations vary by framework and workload. In the system diagram below, those four pillars are coordinated by a fifth orchestration layer—the Execution Runtime—which enforces schemas, step limits, and the observation loop.

AI Agent System Architecture5-Component Runtime
USER GOALPrompt & RulesMax steps budgetEXECUTION RUNTIMEControl & Guardrail LoopSchema checks, retries, step capAppends observations to stateLLM REASONING ENGINENext Action or Final ReplyMEMORY STOREContext + Optional StoreMessages, SQL/files, or vectorsTOOL REGISTRYSandboxed ExecutorsAPIs, SQL, files, web searchLeast-privilege permissionsObservation outputOUTPUTFinal ReplyExit loopTask complete
Figure 1: Core runtime topology connecting the LLM reasoning engine, execution runtime, memory store, sandboxed tool registry, and observation loop.

1. Reasoning Engine (The LLM)

The LLM acts as the decision-making kernel. Rather than generating prose for a human reader on every turn, the model evaluates the current state and decides between two outputs: emit a structured tool call or return the final answer. Frontier reasoning models and capable open-weight models (which you can test locally via our guide on running local AI models on your laptop) work well here when they follow JSON schemas reliably.

2. Tool Use (Function Calling)

An LLM cannot query a live PostgreSQL table, read a local file, or check HTTP status codes on its own. Tool calling bridges this gap by passing a registry of JSON Schema tool definitions alongside the prompt. When the model needs external data, it pauses text generation and outputs a structured function name and argument payload; the host runtime executes that function inside a sandbox and returns the result.

3. Short-Term and Long-Term Memory

  • Short-term (working) memory: The active context window containing the system instructions, user goal, prior tool calls, and tool outputs from the current run.
  • Long-term (persistent) memory: Optional external storage retrieved across sessions. Not every agent needs a persistent vector database—many production systems use ordinary SQL tables, flat Markdown or JSON files, rolling conversation summaries, or no long-term memory at all.

4. Planning Loops (ReAct vs. Plan-and-Execute)

Introduced by Yao et al. in ReAct: Synergizing Reasoning and Acting in Language Models (arXiv:2210.03629), the ReAct loop interleaves verbal reasoning (Thought) with task-specific actions (Action) and environment feedback (Observation). In their evaluation across HotpotQA, FEVER, ALFWorld, and WebShop, grounding reasoning steps with external tool observations helped reduce unsupported claims and error propagation compared with ungrounded Chain-of-Thought prompting alone.

For longer tasks, engineers often pair ReAct with Plan-and-Execute: a planner model writes a numbered checklist first, and smaller worker loops execute each step independently before updating the plan.

Inside the Agent Execution Loop (Step-by-Step Trace)

Below is the step-by-step control cycle a runtime executes from the moment a user submits a request until the agent exits.

Autonomous Agent Execution LoopReAct Control Cycle
1. PERCEIVEGoal & ContextPrompt + Memory2. THINK / PLANLLM ReasoningSelect next action3. CALL TOOLJSON Tool CallAPI / DB / Code / Web4. OBSERVE OUTPUTInspect ResultAppend to contextRepeat until done (max N steps)No more tools needed5. FINAL ANSWERVerified Output
Figure 2: The ReAct-style agent control loop interleaves reasoning, structured JSON tool calls, and environment observations until the exit condition is met or a step budget is reached.

Suppose a developer asks an operations agent: "Check why the staging API returned 502 errors in the last hour and open a GitHub issue if a container crashed."

Step 1: Perceive & Initialize State

The runtime constructs the initial prompt payload: system rules, available tool schemas (fetch_logs, inspect_container, create_github_issue), and the user's goal.

Step 2: Select an Action & Emit a Structured Tool Call

The model recognizes it lacks live telemetry, so it outputs a structured JSON tool call rather than guessing. Notice that production applications generally record a brief rationale summary for audit logs rather than exposing a model's private internal reasoning:

{
  "rationale": "Inspect the last 60 minutes of staging API logs for HTTP 502 responses.",
  "tool_call": {
    "id": "call_01H9X",
    "name": "fetch_logs",
    "arguments": {
      "service": "staging-api",
      "status_code": 502,
      "window_minutes": 60
    }
  }
}

Step 3: Execute & Observe Output

Your host application validates the JSON arguments, runs fetch_logs against your logging backend, and appends the response back into the conversation as a tool message (for example: OOMKilled: container exited with code 137 at 10:14 UTC).

Step 4: Iterate or Return Final Answer

On the next loop turn, the LLM observes exit code 137 (OOMKilled), decides it now has enough evidence, calls create_github_issue with the stack trace, observes the created issue URL, and finally exits the loop with a concise summary for the user.

Simplified Python Agent Loop Example

In code, the core control loop looks like a bounded loop over an allowlisted tool registry (MAX_STEPS prevents runaway execution).

Simplified pseudocode: The exact response and tool-call structures depend on the LLM provider's API. Production implementations also need tool allowlisting, exception handling, timeouts, output validation, and execution isolation.

MAX_STEPS = 8
TOOLS = {
    "fetch_logs": fetch_logs,
    "create_github_issue": create_github_issue,
}

def run_agent(user_goal: str) -> str:
    messages = [{"role": "user", "content": user_goal}]

    for step in range(MAX_STEPS):
        response = llm.generate(messages=messages, tools=list(TOOLS.keys()))

        if not response.tool_call:
            return response.final_text  # Exit loop with final answer

        tool_name = response.tool_call.name
        if tool_name not in TOOLS:
            raise ValueError(f"Blocked unknown tool: {tool_name}")

        args = validate_args(tool_name, response.tool_call.arguments)
        observation = TOOLS[tool_name](**args)

        messages.append({"role": "assistant", "tool_call": response.tool_call})
        messages.append({"role": "tool", "name": tool_name, "content": str(observation)})

    raise RuntimeError(f"Agent exceeded MAX_STEPS ({MAX_STEPS}) without finishing.")

Tool execution should also enforce per-tool permissions, timeouts, output-size limits, and approval requirements for sensitive operations.

Why Agent Loops Multiply Token Usage and Latency

Because standard LLM APIs are stateless, each step in an uncached agent loop typically resends the accumulated conversation context (system prompt + tool schemas + prior tool calls and observations). However, prompt caching (which discounts repeated static prefixes) and provider-specific billing models can significantly change actual cost.

Cumulative Token Consumption by Agent Loop DepthIllustrative arithmetic example (1x = 1,200-token single-shot baseline, uncached)
Single-Shot Prompt (0 tools)1x (1,200 tokens)
Fixed 3-Step Workflow (no retries)3.5x (4,200 tokens)
5-Turn ReAct Agent (4 tool/verify steps + synthesis)6x (7,200 tokens)
15-Step Autonomous Coding / Research Loop18x (21,600 tokens)

Here is the exact arithmetic behind the 5-turn (7,200 tokens = 6x) estimate above, assuming a 1,200-token initial request (1,000-token system/tool prefix + 200-token goal) and 50-token tool calls/outputs without prompt caching:

TurnWhat Is Sent & GeneratedTokens Used in TurnCumulative Total
Turn 1Prefix + goal (1,200 in) + Tool Call #1 (50 out)1,2501,250
Turn 2Prior context (1,250) + Tool Output #1 (50) + Tool Call #2 (50 out)1,3502,600
Turn 3Prior context (1,350) + Tool Output #2 (50) + Tool Call #3 (50 out)1,4504,050
Turn 4Prior context (1,450) + Tool Output #3 (50) + Verification (50 out)1,5505,600
Turn 5Prior context (1,550) + Final synthesis (50 out)1,6007,200 (6x)

To control costs in production, enable prompt caching for static system prompts and tool schemas, truncate verbose tool outputs (such as raw HTML or 500-line logs) before appending them to context, and enforce a strict max_steps ceiling. If you are building solo on a tight budget, pair lean agent loops with the zero-cost stack options in our free developer tools for solo builders and free AI coding assistants guides.

Multi-Agent Orchestration Patterns

When a single agent accumulates a large toolset (for example, 15–20+ tools) and competing instructions, splitting work across specialized agents can improve task organization and evaluation—though it adds coordination overhead and does not automatically improve accuracy. As Anthropic's orchestrator-workers pattern demonstrates, delegating focused subtasks keeps each model call narrower and easier to evaluate.

PatternHow It WorksStrengthsTrade-Offs
Supervisor / OrchestratorOne lead agent delegates subtasks to specialized worker agents and synthesizes resultsClear ownership, easy to enforce budgets and permissionsExtra coordination hops add latency
Sequential HandoffAgent A finishes triage and passes state directly to Agent B (e.g., Researcher → Writer)Simple state machine, clean prompt separationEarly mistakes propagate downstream
Parallel SwarmMultiple agents inspect independent files or angles simultaneously and report backFast wall-clock speed on large codebases or auditsHigher peak token burst; requires merge logic

Common Failure Modes & Agent Security Checklist

  1. Infinite Retry Loops: If an API returns 403 Forbidden, an unguarded agent may retry identical arguments repeatedly. Fix: Cap max_iterations (such as 8–15 steps as an illustrative starting cap, depending on task complexity), track repeated identical tool calls, and abort early with a diagnostic message.
  2. Context Window Overflow ("Lost in the Middle"): Dumping raw 50 KB logs into short-term memory degrades reasoning accuracy. Fix: Summarize or slice tool results to the relevant lines before returning them to the model.
  3. Hallucinated Tool Parameters: Models occasionally invent optional flags or malformed IDs. Fix: Keep tool interfaces narrow, use descriptive enum values, and return clear validation errors so the agent can self-correct on the next turn.
  4. Indirect Prompt Injection: Untrusted web pages or tickets read by a tool can contain hidden instructions trying to hijack the agent's next tool call. Fix: Separate data from instructions, sanitize fetched content, and gate sensitive tools behind explicit approval hooks.

Frequently Asked Questions

What is the difference between an AI agent and a normal LLM?

A normal LLM generates text in a single pass from a prompt and stops. An AI agent wraps that LLM in a control loop with tool definitions and memory so it can fetch live data, execute code or API calls, inspect errors, and take multiple steps autonomously.

What is the ReAct framework in AI agents?

ReAct (Reasoning + Acting) is an agent research pattern introduced by Yao et al. where a model interleaves a reasoning trace (Thought), an external tool call (Action), and tool feedback (Observation) before deciding its next move. In production applications, teams typically log concise action summaries (rationale) rather than exposing private internal reasoning.

Do AI agents require LangChain or a heavy framework?

No, especially for prototypes. A small while loop over a model's native tool-calling API is enough for a prototype or narrow internal script, whereas production deployments require retries, structured logging/monitoring, permission boundaries, evaluation tests, and error handling—whether you build them yourself or adopt an orchestration framework.

Can you run AI agents locally without paying API fees?

Yes, depending on your hardware and workload. Runtimes such as Ollama and LM Studio expose local OpenAI-compatible endpoints for open-weight models (like Qwen and Llama), though reliable tool calling depends on both the specific model and runtime support. Memory requirements also vary by parameter count, quantization level (such as Q4_K_M), context length, and GPU/unified memory bandwidth—small 3B–7B quantized models can run on 8–16 GB machines, while larger models need more RAM/VRAM. For a complete installation walkthrough, follow our free guide to running local AI models on your laptop.


Primary References

Was this guide useful?

Quick feedback helps keep our guides accurate and up to date.

Report an Issue

Choose a category to help us route your feedback.

Select Category *
View all in AI & Agents →