Your pricing agent updated 47 SKUs this morning. Except it didn't pull competitor data at all — the API endpoint it depends on changed its response format overnight, and the agent hallucinated numbers to fill the gap. The logs said "complete." The output looked normal. Nobody noticed for six hours.
That's what an AI agent not working actually looks like. Not a crash with a red error screen. A quiet drift where the output passes every automated check, the logs show success, and the damage compounds until a human spots it.
I run over 30 AI agents in production across my ecommerce and advisory businesses. Something breaks at least twice a month. The difference between operators who scale AI and operators who quit after three months is not whether things break — it's how fast you diagnose and fix the failure.
Here's the systematic playbook I use every time.
What Is AI Agent Debugging and Why Is It Different From Normal Software Fixes
AI agent debugging is the process of figuring out why an AI automation is producing wrong, degraded, or missing output — and fixing the root cause so it doesn't happen again.
It matters because AI agents fail differently than regular software. Traditional software is deterministic: same input, same output, same error. You trace a stack trace to the exact broken line. AI agents are nondeterministic: the same prompt can produce different output on different days, with different models, or with slightly different context. When a traditional automation breaks, it throws an error. When an AI agent breaks, it often produces confident-sounding garbage and marks the task complete.
You're not looking for a crash. You're looking for drift.
The Five Ways AI Agents Break in Production
After running agents in production for two years, I've catalogued every failure into five categories. Knowing which one you're dealing with cuts debugging time in half.
1. Context Rot
Your agent's instructions reference a process, file structure, or data format that no longer exists. You updated your CRM fields three weeks ago but never updated the agent's skill file. The agent doesn't error — it improvises around the missing context and produces plausible but wrong output.
How to spot it: Output is structurally correct but factually wrong. It references old field names, deprecated processes, or outdated pricing.
2. Model Drift
Your AI provider updated the model. Maybe a minor version bump, maybe a major release. Your prompt worked perfectly on the old version — the new one interprets the same instructions differently. The agent becomes verbose where it was concise, or ignores constraints it used to follow.
How to spot it: Quality degraded across ALL tasks simultaneously, not just one. Check your provider's changelog — if they shipped a model update recently, that's your culprit.
3. Integration Failures
An API changed its response format. An MCP server timed out. A webhook stopped firing. Your agent depends on external tools, and one of them changed without warning.
How to spot it: The agent runs but produces empty, partial, or obviously stale data. Check your integration logs for 4xx/5xx errors, timeouts, or response format changes.
4. Prompt Fragility
Your prompt worked for 50 runs because the inputs were similar. Run 51 introduces an edge case — a product name with special characters, a competitor with no listed price, a report in an unusual format. The prompt breaks not because it's wrong, but because it's brittle.
How to spot it: The agent works for most inputs but fails on specific ones. The failures cluster around unusual data shapes.
5. Silent Success Failures
The agent runs. Logs say "complete." The output file gets written. But the content is wrong — hallucinated data, skipped steps, or a polite refusal buried in otherwise good output. This is the most dangerous category because every monitoring system says green.
How to spot it: You can't, not without sampling. This is why output sampling is non-negotiable.
The Four-Step Debugging Framework I Use Every Time
When an agent breaks, I follow the same four steps in order. Skip a step and you'll waste hours fixing the wrong thing.
Step 1: Isolate the Failure Window
First question: when did this start? Don't guess — check output logs and find the exact run where quality dropped.
ls -la outputs/pricing-agent/ | tail -20
If the failure started Tuesday at 3 AM, ask: what changed between Monday's good run and Tuesday's bad one? Model update? Dependency change? New data shape?
Step 2: Reproduce in Isolation
Pull the agent out of its automation loop and run it manually with the exact same inputs that failed.
claude -p "$(cat skills/pricing-agent.md)" \
--input "$(cat inputs/failed-run-2026-07-21.json)" \
--output debug-output.json
If it fails manually: the problem is the agent itself (prompt, context, or model). If it succeeds manually: the problem is the orchestration (scheduling, input pipeline, or environment).
Step 3: Diagnose the Category
Using the five categories above, run this checklist before touching anything:
- Context rot? Diff your skill files against the systems they reference. Any mismatches?
- Model drift? Check your provider's status page and changelog. Any model updates in the last week?
- Integration failure? Test each external dependency independently. Any API changes or timeouts?
- Prompt fragility? Run the agent against the last 20 inputs. Does it only fail on specific shapes?
- Silent success? Read the full output word by word. Any hallucinations, hedging, or skipped steps?
Step 4: Fix the Root Cause and Verify
Fix the root cause, not the symptom. If an API changed its response format, don't patch the prompt to handle both formats — update the integration layer. If the model drifted, pin your model version or update the prompt for the new model.
After fixing, run the agent against the last 10 inputs — not just the one that failed. This catches fixes that solve one problem while creating another.
Real Debugging Examples From My Production Stack
The Competitor Monitor That Hallucinated Prices
My competitor pricing agent pulls data from five marketplaces and generates a report. One morning, it showed a competitor at $12.99 — 40% below their actual price. I almost adjusted my own pricing before I caught it.
Root cause: Integration failure. One marketplace changed its HTML structure. The scraping tool returned empty strings. The agent, seeing no data, estimated the price from historical context that was months old.
Fix: Added validation rules to the skill file:
Before generating the report, validate each data point:
- If any price field is empty or null, flag it as MISSING — never estimate
- If any price deviates >25% from the 7-day average, flag for manual review
- Include a data completeness score at the top of every report
Time to diagnose: 20 minutes. Time to fix: 10 minutes. Cost of not catching it: a pricing mistake across 47 SKUs.
The Content Agent That Stopped Following Format
My weekly content drafting agent produces outlines in a specific format: title, 6 H2s, 3 bullet points per section, and a call-to-action. Week 12, it drifted — 8-10 H2s with full paragraphs instead of bullets.
Root cause: Model drift. The underlying model had been updated and now interpreted "outline" more expansively. The prompt said "create an outline" but didn't constrain the format tightly enough.
Fix: Replaced vague instructions with hard constraints:
## Output format (strict)
- Exactly 6 H2 sections
- Each section: 3 bullet points, max 20 words each
- No paragraphs, no sub-sections
- Final section must be a CTA
- If output exceeds these constraints, cut — do not expand
The Daily Briefing That Went Silent
My morning intelligence briefing runs at 5 AM and posts to Slack. One day it just stopped. No error, no output, no alert.
Root cause: The scheduling system was running, but the agent process hung on a stale MCP server connection. It started, waited for the connection, and timed out after the monitoring window closed. The watchdog saw "process started" and marked it healthy.
Fix: Added a completion signal. The agent writes a timestamp file when it finishes. A separate check runs 30 minutes later and verifies the timestamp is fresh:
LAST_RUN=$(cat ~/.briefing-last-run 2>/dev/null || echo "0")
NOW=$(date +%s)
DIFF=$((NOW - LAST_RUN))
if [ $DIFF -gt 7200 ]; then
curl -X POST "$SLACK_WEBHOOK" \
-d '{"text":"Morning briefing did not complete. Check the agent."}'
fi
How to Build Observability That Catches Failures Before Your Customers Do
Debugging is reactive. Observability is proactive. Here's the minimum monitoring I put on every agent that matters:
1. Completion signals. Every agent writes a timestamp and status when it finishes. A separate watchdog checks these signals on a schedule. No signal within the expected window triggers an alert. Setup time: 5 minutes per agent.
2. Output shape validation. Before output reaches its destination — Slack, a spreadsheet, a customer-facing system — a lightweight check validates the structure. Is the JSON valid? Are required fields present? Is the word count within the expected range?
Here's the validation pattern I add to most skill files:
## Self-validation (run before delivering output)
Before sending final output, verify:
1. All required sections are present
2. No section is empty or contains placeholder text
3. No data point says "I don't have" or "unable to find" — flag those as INCOMPLETE
4. Total output is between [MIN] and [MAX] words
If any check fails, prepend "VALIDATION WARNING:" and list the failures
3. Output sampling. Once a week, I manually review 3-5 random outputs per agent. This is the only reliable way to catch silent success failures — the ones where the agent completes its run, reports success, and produces confident-sounding wrong answers. No automation replaces a human reading the actual output.
4. Dependency health checks. For every external API or data source an agent uses, a weekly check confirms the response format hasn't changed. A simple hash comparison on the response structure catches 90% of integration failures before they reach your agents.
When to Rebuild an AI Agent vs. When to Repair It
Not every broken agent deserves a fix. Sometimes rebuilding from scratch is faster and more reliable.
Repair when:
- The failure falls into one clear category from the list above
- The fix is isolated to one layer — prompt, integration, or scheduling
- The agent has been reliable for 20+ runs before this failure
- The fix takes less than 30 minutes
Rebuild when:
- You're patching patches — the skill file has more exceptions than core rules
- The agent's actual task has evolved since you first built it
- You've fixed the same agent three times in two weeks
- The model has changed enough that the original prompt design no longer fits
A rebuild sounds expensive, but a clean rewrite usually takes 45 minutes with a clear spec. A patched agent with six layers of workarounds takes longer to debug each time it breaks, and the patch debt compounds.
The Maintenance Calendar That Prevents Most AI Agent Failures
Most failures are preventable with a lightweight routine.
Weekly (15 minutes):
- Sample 3-5 outputs per agent for quality
- Review flagged validation warnings
- Check completion signal logs for missed or late runs
Monthly (30 minutes):
- Diff skill files against the systems they reference — catch context rot before it causes failures
- Review provider changelogs for model updates
- Test each external integration independently
After any external change (immediately):
- API updates → test the integration, update the agent's handling
- Model updates → run the agent against 10 recent inputs, compare output quality to the previous version
- Process changes → update skill files the same day, don't wait for a failure to remind you
FAQ
How do I know if my AI agent is silently failing?
You can't know from logs alone — a silently failing agent reports success because it completed its run. The only reliable detection method is output sampling: regularly reading actual outputs and comparing them against what you expected. Three to five random outputs per agent per week is enough to catch drift.
Should I pin my AI model version to prevent drift?
Yes, if your provider supports it. Pinning the version prevents unexpected behavior changes from model updates. The trade-off is missing improvements. I pin for production agents and test unpinned versions in a staging environment to preview changes before adopting them.
Why does my AI agent work in testing but fail in production?
Almost always the input data. Testing uses clean, expected inputs. Production introduces edge cases — special characters, missing fields, unusual formats, content that's longer or shorter than expected. Run the agent against 20 real production inputs in your test environment. The failures will cluster around data shapes your test cases didn't cover.
What's the minimum monitoring for every AI agent?
Three things: a completion signal (did it actually finish?), output shape validation (is the structure correct?), and weekly output sampling (is the content actually right?). These three catch 90% of failures. Total setup time is about 15 minutes per agent.
How often do AI agents break in production?
In my experience with 30+ agents: something needs attention roughly twice a month. Most failures are minor — a context update, a reconnected integration. Major failures that require a full rebuild happen about once a quarter. The failure rate drops significantly after the first month once you've handled the initial edge cases.
Your AI Agent Is Not Working — Here Are Your Three Next Actions
If your AI agent is not working right now, do these three things in order:
-
Isolate the failure window. Find the exact run where quality dropped and identify what changed between the last good run and the first bad one. Don't guess. Check logs, timestamps, and provider changelogs.
-
Classify before you fix. Run through the five failure categories — context rot, model drift, integration failure, prompt fragility, silent success — before touching the prompt. Fixing the wrong category wastes time and usually introduces new problems.
-
Build the three minimum monitors on every agent you care about: completion signals, output shape validation, and weekly output sampling. Fifteen minutes of setup per agent saves hours of reactive debugging later.
AI agents that run your business will break. That's not a reason to stop building them — it's a reason to build the muscle of diagnosing failures fast. The operators who win with AI aren't the ones whose agents never fail. They're the ones who detect failures in minutes instead of hours, fix root causes instead of symptoms, and build the monitoring that prevents the same AI agent failure from happening twice.