Heuristic Detectors vs LLM Judges: What We Learned Analyzing 7,000 Agent Traces

Heuristic Detectors vs LLM Judges: What We Learned Analyzing 7,000 Agent Traces The default approach to evaluating AI agents is to use another AI. LLM-as-judge. Feed the trace to a frontier model and ask "what went wrong?" It's intuitive, flexible, and expensive. It also underperforms purpose-built heuristics on most failure categories. We know this because we tested both approaches systematically. Pisama has 18 production-grade heuristic detectors calibrated on 7,212 labeled entries from 13 external data sources. We benchmarked them against LLM judges on two public agent failure benchmarks. The results challenged our assumptions about when you need semantic reasoning and when simple pattern matching is enough. This article presents the data, explains why heuristics outperform LLMs on structural failures, identifies the categories where LLMs are still essential, and describes the tiered architecture we settled on. The Benchmarks TRAIL: Single-Trace Failure Detection TRAIL, released by