How much effort does Claude Opus 5.5 need for a Delphi code review? I measured it.

Anthropic’s guide for Claude Opus 5.5 says: test several effort levels against your own tasks, and reserve xhigh and max for work where you have measured a quality gain. Fair enough. So I measured it, on Delphi code.

The test

  • Model: Claude Opus 5.5, in Claude Code.
  • My code review agent. It only reports bugs and changes nothing.
  • 3 units of my LightSaber library, taken from the git history as they were before I fixed 18 known bugs in them.
  • 3 effort levels (medium, high, xhigh). Every unit was reviewed twice at every level: 18 reviews, so 36 known bugs to find per level.
  • A separate agent checked every finding that was not on my list of known bugs against the code. It did not know which effort level the finding came from.

The results

EffortKnown bugs found (of 36)Other real bugsFalse alarmsTokens per reviewMinutes per review
medium335201.5 million6.5
high356302.7 million9.9
xhigh356705.8 million17.2

“Other real bugs” are real bugs that were not on my list. The count is per run, so a bug found by 4 runs counts 4 times. The token count includes cached input, which is cheap, so your bill grows less steeply than the token count.

What I take from it

  • From high to xhigh I see no gain. Two runs at the same level differ more than the two levels do: on one unit, two xhigh runs found 11 and 16 other bugs. But xhigh costs 2.1 times the tokens and 7 more minutes per review.
  • Medium missed one bug, in both of its runs. Both runs read the routine, described its behavior and listed it as “checked and clean”. All four runs at high and xhigh reported it as a bug. The AI that scored the runs counted that as a miss, under a rule it added during the scoring. If you read the original scoring rule literally, it counts as found, and then all three levels find 35 of 36.
  • No false alarms at any level. Here that means: no statement the agent made about the code was false.
  • So my code reviews stay at high.

Limits of this test

  • It is small: 3 units from one library, 2 runs per level and unit. A real difference of 1 or 2 bugs per review cannot show up with so few runs.
  • My known-bug list came from an earlier run of the same review pipeline, so bugs that this pipeline never finds are not on it.
  • The scoring was not fully blind. The AI that scored the runs knew the level of some of them.
  • My review agent can ask a stronger model for advice (the advisor tool), and every run used it once. The effect of the effort level may be smaller with that tool than without it.

Your code may tell a different story. Did you measure this on your own code? I would like to hear what you found.

Leave a Comment

Scroll to Top