Introduction#

Every DeFi exploit leaves a permanent forensic record on-chain. The attacker’s steps, the vulnerable contract, the asset flows, and the call trace are all immutably recorded, available years later from a single transaction hash. The security community has also built curated databases of these incidents, each annotated with a human-written root cause analysis.

That pairing of deterministic inputs, machine-readable evidence, and documented ground truth makes exploit root cause analysis (RCA) a clean benchmark task for AI agents. Unlike static code audits, the evidence is dynamic and on-chain. The ground truth ties back to a specific human-assigned label for a real incident where funds were lost, not a synthetic CTF or a generic vulnerability detection signal.

TxRCA-Bench is, to my knowledge, the first benchmark to evaluate AI agents on this task. We ask one question: given only a transaction hash, can a frontier AI agent identify the root cause of a DeFi exploit using on-chain data alone?

Overview#

TxRCA-Bench consists of:

• 70 real-world exploit transactions spanning Ethereum and BSC, stratified across eight vulnerability categories.
• An automated evaluation harness that builds a per-case workspace with decoded traces, event logs, ABIs, and verified source, then dispatches each agent as an isolated subprocess.
• Seven frontier agent configurations (GPT-5, GPT-5.3-Codex, GPT-5.4 at High/XHigh, Claude Sonnet 4.6, Claude Opus 4.6 at High/Max) running in their native agent runtimes (Codex CLI, Claude Code CLI).
• Two independent LLM judges from different model families (Claude Opus 4.6, GPT-5.4) scoring every run on a 0–7 holistic rubric.

Across 490 runs, the top agent (Claude Opus 4.6 Max) hits 72.9% success under the Opus judge; the weakest (GPT-5 High) reaches 35.7%. Bumping reasoning effort yields +7–9 pp across families. Two-thirds of failures involve taxonomy boundary confusion: the agent describes the mechanism and cites the right evidence, then applies the wrong label. High-profile cases do not beat low-profile ones, which argues against memorization-driven performance.

In the rest of this post, I will discuss the motivation for this task, dataset construction, evaluation pipeline and anti-cheat design, results and failure analysis, and implications for future research.

Motivation#

Three properties make DeFi exploit RCA a useful proxy for agentic security reasoning.

Ground truth is verifiable. Every case has a documented root cause from post-mortem analysis, and the on-chain evidence never changes. You can audit a score, re-run an agent years later, or replay the exact blockchain state.

Success requires multi-hop reasoning. Surface pattern-matching will not solve most cases. An agent has to follow the call trace through multiple contracts, reason about delegatecall and proxy boundaries, cross-reference decoded logs against Solidity source, and demonstrate why a state change leaks value rather than only what it does.

Failure modes are diagnostic. A score of 3 instead of 6 usually reflects a meaningful error: wrong family with correct mechanism, or right function with stale narrative. Those errors tell you where the agent’s reasoning breaks down, which is a lot more useful than a single pass/fail flag.

Dataset#

We sourced exploit cases from the SunWeb3Sec DeFi Security Breach RCA dataset, which indexes 1,753+ incidents with 504 detailed root-cause write-ups. We filtered to cases that satisfy all of:

1. Valid attack transaction hash on Ethereum mainnet or BNB Chain.
2. At least one involved contract has verified source or decompilable bytecode.
3. The root cause label is unambiguous.
4. The exploit date falls within 2021 to 2024.

We split qualifying cases into two disjoint groups: an 11-case pilot set used only for prompt engineering and judge calibration, excluded from all reported results, and a 70-case evaluation set frozen before any final evaluation run began.

Methodology#

Results#

Failure analysis#

We classified the 185 runs scoring ≤ 2 under the Opus judge by analyzing judge rationales:

Contamination analysis#

A key concern for any AI benchmark is whether models succeed by recalling memorized information rather than analyzing the evidence. We compare success rates on high-profile cases (7 well-known incidents covered across security blogs, likely in training data) against low-profile cases (63 smaller incidents).

High-profile cases show lower success rates than low-profile cases on average (Δ = −18.7 pp per-setting), and only 1 of 7 settings has a positive delta. That is a healthy signal for benchmark validity. If agents were pulling from memorized incident reports, high-profile cases would score higher, not lower.

The likely driver of the negative delta is that high-profile exploits tend to involve complex multi-contract protocols (Ronin Network, Curve Finance, AAVE), which make on-chain analysis harder even with memorized knowledge. Low-profile incidents usually center on simpler single-contract vulnerabilities.

Agent self-confidence#

Agents report their own confidence (High/Medium/Low) alongside each analysis, and the self-reports carry signal. High-confidence runs average a score of about 5.0 at a 65.8% success rate. Medium-confidence runs score about 4.0 at 40.1%. Low-confidence runs score about 3.5 at 33.3%. The monotonic pattern supports a cheap triage filter: pull the high-confidence runs forward for manual review.

Per-model reasoning styles#

The models also differ in reasoning style, as you might expect.

GPT-5 High writes the shortest analyses (mean 858 chars in the analysis field). It anchors on the most salient surface feature of the call trace, such as a flash loan or an external call, and calls that feature the root cause without tracing down to the underlying bug.

Claude Sonnet 4.6 High writes the longest analyses (mean 1,909 chars), more than twice the length of Opus outputs. The extra length does not improve accuracy, though.

GPT-5.4 High/XHigh is the most concise and precise OpenAI configuration (mean 1,142 to 1,105 chars). It achieves score 7 on representative Insufficient Validation cases by correctly naming the vulnerable function selector and tracing the full calldata-injection attack path.

Claude Opus 4.6 High/Max combines brevity (mean 1,266 to 1,301 chars) with the highest precision, producing the most score-7 results (23 and 21, against 16 for GPT-5.4 High). Opus reasons bottom-up: it identifies which function accepts attacker-controlled input, traces the call path to the profitable outcome, and only then assigns a root-cause label.

1
root_cause_classes:    Access Control
2
vulnerable_contract:   0x87870b… (Aave V3 Lending Pool core)
3
vulnerable_function:   mintUnbacked()
4
attack_mechanism:      Attacker deploys helper contract and takes a
5
                       multi-asset flash loan from Balancer Vault.
6
                       Using the helper, calls the lending pool at
7
                       0x87870b with function 0x617ba037 to mint
8
                       unbacked aTokens across multiple reserves […]
9
confidence:            Medium

1
root_cause_classes:    Insufficient Validation
2
vulnerable_contract:   0x02e7b8… (Aave V3 ParaSwap Repay Adapter)
3
vulnerable_function:   executeAction (selector 0x4db9dc97)
4
attack_mechanism:      Attacker flash-loans 5 tokens from Balancer
5
                       Vault. Supplies WBTC to Aave V3 as collateral.
6
                       For each subsequent token: supplies to Aave,
7
                       borrows a stablecoin, then calls the adapter's
8
                       executeAction with crafted ParaSwap calldata.
9
                       The adapter burns half the aTokens, withdraws
10
                       the underlying, and attempts a ParaSwap swap;
11
                       however the swap calldata routes funds back to
12
                       the attacker rather than repaying debt. Because
13
                       paraswapData is never validated and the high
14
                       allowance set before the swap is not cleared
15
                       on failure, the adapter effectively transfers
16
                       the withdrawn underlying to the attacker […]
17
confidence:            High

Limitations#

A few things to keep in mind when reading these numbers:

Small Misconfiguration sample. Our evaluation set has only three Misconfiguration cases. The 100% failure rate is notable, but the three cases may share characteristics, such as off-chain root causes (compromised keys, deployment errors) that are harder to observe from a transaction trace.

Verification coverage. 65 of 70 cases have fully verified contracts; 5 required decompilation. That sample is too small to draw strong conclusions about decompilation effectiveness for RCA.

Residual contamination risk. The contamination analysis shows no evidence of memorization-driven inflation, but we cannot rule out training data that included unlabeled on-chain analysis of these specific transactions.

Future work#

Several extensions are natural follow-ups:

• Multi-transaction exploits. Governance attacks, sandwich attacks, and cross-day MEV sequences would push agent reasoning over longer causal chains.
• Automated patch suggestion. Ground-truth patches exist for most cases via associated PoC Foundry tests in DeFiHackLabs. A natural extension is to ask agents to propose a concrete code fix, not just identify the root cause.
• Cross-chain generalization. Extending to Solana, Avalanche, and Arbitrum would test whether on-chain reasoning generalizes across different execution environments and trace formats.
• Tool augmentation. Giving agents access to formal verification tools or symbolic execution may significantly improve precision on complex categories.

Takeaways#

• Frontier agents are quite capable on this task. Claude Opus 4.6 Max succeeds on ~73% of cases under the Opus judge. That’s not sufficient for autonomous deployment, but it is a non-trivial capability level for a task that requires multi-contract trace reasoning with zero off-chain context.

• Reasoning effort matters. Consistent +7–9 pp gains from a single effort-level bump, across two model families and two judges.

• On-chain evidence seems to dominate memorization. High-profile cases do not beat low-profile ones; if anything, they trail. Agents read the trace rather than retrieve the incident.

• The judge gap is measurable. The Opus and GPT-5.4 judges open a 15 to 29 pp success-rate gap across settings on identical outputs, with Cohen’s κ = 0.58. Automated scoring scales, but results are sensitive to judge choice. Anyone building similar pipelines should probably report both.

Open data#

We are releasing the TxRCA-Bench benchmark data: the 70 annotated exploit transactions with ground-truth root cause labels, the per-case workspaces (raw traces, event logs, contracts, ABIs, Solidity sources), all 490 raw agent outputs with both judges’ scores, and the JSON output schema. You can re-score outputs, point your own agent at the same evidence, or extend the benchmark with new cases.

The agent runtime and scoring harness code stay private for now. The on-chain data does not change, and the benchmark is defined by (transaction_hash, chain_id) plus a ground-truth label, so anyone can reproduce it against a new agent: take the inputs, run the agent in any runtime, and score the output against the same rubric.