LEB-100
AI Benchmark · LEB · the reference instance
The next level, LEB-300, is an exploratory pilot: its aggregate results are published.
The first level of LEB: a legacy PHP application of about 300 lines, solved by every agent below from the same package. How a run works, how it is scored and how this level differs from LEB-300 is on the Benchmark page.
Top 6 so far
LEB-100-A- 1Claude Sonnet 5.5 · xhigh809LEB Gold
- 2Claude Sonnet 5.5 · max807LEB Gold
- 3Claude Sonnet 5.5 · max (ultracode)774LEB Gold
- 4Claude Fable 5.1764LEB Gold
- 5Claude Opus 5.5717LEB Silver
- 6GPT-6-astra · ultra666LEB Silver
Score out of 1000, of 37 agents. Not official until each has three runs. Full leaderboard
Results
Every delivery in a table solved the same package, byte for byte — the same SHA-256 — so the numbers compare like for like.
LEB-100-A v1.1 · Support-ticket panel of an internet provider
The support-ticket panel of an internet provider, written in 2013-style PHP: data-access functions and an index.php that routes, authorizes and builds the HTML. About 300 lines on PHP 8, mysqli and MySQL 8, with 13 planted flaws and 2 decoys.
- mode A · 30 turns
- edition 2026
- evaluated 2026-10-06
- matrix 68088abdb7bc…
Showing 37 of 37 agents. Ranks stay those of the full leaderboard.
-
1
Claude Sonnet 5.5 Anthropic · effort xhigh 3 of 3 runs (825 · 809 · 724)809 of 1000 LEB Gold
- Security250/250
- Architecture25/200
- Bugs139/150
- Performance150/150
- Clean code100/100
- Compatibility100/100
- Explanation45/50
Comment and details
The most complete single-agent result: security at 250 of 250 in its first two runs, performance at full marks, compatibility untouched, and the CSV formula injection fixed the way the answer key expects. Its third run left three security flaws found but unfixed and fell to 724; like every agent, it left the dispatcher whole.
A written reading drawn from the scorecards and verdicts; it is not part of the score.
- Full marks
- Security, Performance, Clean code, Compatibility
- Never fixed, in any run
- A dispatcher that does everything; Magic numbers for status and priority
Run by run
- Run 1 · 82511 of 13 flaws fixedno false positivecontract kept22/22 checks19minUS$ 3.60Tokens: 1,533 in · 138,245 out · 7,176,279 cache · 7,316,057 totalScorecard
- Run 2 · 80911 of 13 flaws fixedno false positivecontract kept22/22 checks23minUS$ 2.96Tokens: 1,092 in · 131,538 out · 4,673,192 cache · 4,805,822 totalScorecard
- Run 3 · 7248 of 13 flaws fixedno false positivecontract kept22/22 checks23minUS$ 2.91Tokens: 1,090 in · 127,242 out · 4,686,490 cache · 4,814,822 totalScorecard
-
2
Claude Sonnet 5.5 Anthropic · effort max 3 of 3 runs (807 · 773 · 820)807 of 1000 LEB Gold
- Security250/250
- Architecture12/200
- Bugs150/150
- Performance150/150
- Clean code100/100
- Compatibility100/100
- Explanation45/50
Comment and details
The most consistent of the top three: 773 to 820 across three runs, bugs and performance at full marks in all of them, no false positive and no contract broken. More effort than xhigh bought nothing measurable, at about one and a half times the cost.
A written reading drawn from the scorecards and verdicts; it is not part of the score.
- Full marks
- Security, Bugs, Performance, Clean code, Compatibility
- Never fixed, in any run
- A dispatcher that does everything; Magic numbers for status and priority
Run by run
- Run 1 · 80711 of 13 flaws fixedno false positivecontract kept22/22 checks27minUS$ 3.86Tokens: 966 in · 214,201 out · 3,527,234 cache · 3,742,401 totalScorecard
- Run 2 · 77310 of 13 flaws fixedno false positivecontract kept22/22 checks29minUS$ 4.75Tokens: 1,078 in · 229,079 out · 6,339,029 cache · 6,569,186 totalScorecard
- Run 3 · 82011 of 13 flaws fixedno false positivecontract kept22/22 checks40minUS$ 6.65Tokens: 5,597 in · 352,423 out · 7,098,658 cache · 7,456,678 totalScorecard
-
3
Claude Sonnet 5.5 Anthropic · effort max (ultracode) 3 of 3 runs (774 · 820 · 759)774 of 1000 LEB Gold
- Security250/250
- Architecture0/200
- Bugs129/150
- Performance150/150
- Clean code100/100
- Compatibility100/100
- Explanation45/50
Comment and details
Claude Code's multi-agent mode: 68 to 145 subagents and 2h to 7h 55min a run, at US$ 119 to 233 each, for 774, 820 and 759. At best it tied the same model working alone in half an hour, with the best-calibrated report of the Claude agents and the same untouched architecture.
A written reading drawn from the scorecards and verdicts; it is not part of the score.
- Full marks
- Security, Performance, Clean code, Compatibility
- Zero
- Architecture
- Never fixed, in any run
- A dispatcher that does everything; Magic numbers for status and priority
Run by run
- Run 1 · 77411 of 13 flaws fixedno false positivecontract kept22/22 checks7h 55minUS$ 233.08Tokens: 8,703 in · 6,819,101 out · 627,624,423 cache · 634,452,227 totalScorecard
- Run 2 · 82011 of 13 flaws fixedno false positivecontract kept22/22 checks2hUS$ 171.18Tokens: 90,631 in · 6,961,681 out · 329,949,168 cache · 337,001,480 totalScorecard
- Run 3 · 75910 of 13 flaws fixedno false positivecontract kept22/22 checks3h 34minUS$ 119.07Tokens: 4,150 in · 3,825,335 out · 270,341,344 cache · 274,170,829 totalScorecard
-
4
Claude Fable 5.1 Anthropic · effort xhigh 2 of 3 runs (781 · 764)764 of 1000 LEB Gold
- Security233/250
- Architecture0/200
- Bugs139/150
- Performance150/150
- Clean code100/100
- Compatibility100/100
- Explanation42/50
Comment and details
Finds as much as Sonnet (11 to 12 of the 13 flaws) and fixes less of it: it explained the CSV formula injection and left it in place on purpose, to protect the file's consumers. Two runs so far; its client switched part of a third attempt to another model, which voided it.
A written reading drawn from the scorecards and verdicts; it is not part of the score.
- Full marks
- Performance, Clean code, Compatibility
- Zero
- Architecture
- Never fixed, in any run
- Formula injection in the CSV export; A dispatcher that does everything; Magic numbers for status and priority
Run by run
- Run 1 · 7819 of 13 flaws fixedno false positivecontract kept22/22 checks16minTokens: 932 in · 71,351 out · 2,551,395 cache · 2,623,678 totalScorecard
- Run 2 · 76410 of 13 flaws fixedno false positivecontract kept22/22 checks22minUS$ 8.04Tokens: 1,630 in · 93,132 out · 2,502,648 cache · 2,597,410 totalScorecard
-
5
Claude Opus 5.5 Anthropic · effort xhigh 3 of 3 runs (711 · 717 · 805)717 of 1000 LEB Silver
- Security233/250
- Architecture50/200
- Bugs139/150
- Performance150/150
- Clean code0/100
- Compatibility100/100
- Explanation45/50
Comment and details
The highest architecture score in an official run (50 of 200), and the widest spread among the Claude models: 711 and 717, then 805. It fixed 9 flaws in every run and never flattened the nested ifs in the run that counts, which is where its gap to Sonnet comes from.
A written reading drawn from the scorecards and verdicts; it is not part of the score.
- Full marks
- Performance, Compatibility
- Zero
- Clean code
- Never fixed, in any run
- Formula injection in the CSV export; A dispatcher that does everything; Magic numbers for status and priority
Run by run
- Run 1 · 7119 of 13 flaws fixedno false positive1 business value changed22/22 checks16minTokens: 40 in · 80,368 out · 1,949,756 cache · 2,030,164 totalScorecard
- Run 2 · 7179 of 13 flaws fixedno false positivecontract kept22/22 checks18minUS$ 3.01Tokens: 962 in · 83,343 out · 1,940,114 cache · 2,024,419 totalScorecard
- Run 3 · 8059 of 13 flaws fixedno false positivecontract kept22/22 checks15minUS$ 2.71Tokens: 1,074 in · 69,811 out · 2,537,939 cache · 2,608,824 totalScorecard
-
6
GPT-6-astra OpenAI · effort ultra 3 of 3 runs (668 · 666 · 651)666 of 1000 LEB Silver
- Security224/250
- Architecture25/200
- Bugs129/150
- Performance150/150
- Clean code25/100
- Compatibility70/100
- Explanation43/50
Comment and details
The strongest GPT agent, with two or three subagents in 10 to 15 minutes a run and three runs within 17 points of each other. It changed one business value in every run (−30 each), and its reports are among the best calibrated here (Brier 0.000).
A written reading drawn from the scorecards and verdicts; it is not part of the score.
- Full marks
- Performance
- Never fixed, in any run
- A dispatcher that does everything; Magic numbers for status and priority; Four levels of nested ifs
Run by run
- Run 1 · 6689 of 13 flaws fixedno false positive1 business value changed22/22 checks11minTokens: 174,060 in · 38,839 out · 3,108,480 cache · 3,321,379 totalScorecard
- Run 2 · 6669 of 13 flaws fixedno false positive1 business value changed22/22 checks15minTokens: 142,970 in · 36,517 out · 2,383,872 cache · 2,563,359 totalScorecard
- Run 3 · 6518 of 13 flaws fixedno false positive1 business value changed22/22 checks10minTokens: 171,561 in · 25,580 out · 1,841,536 cache · 2,038,677 totalScorecard
-
7
GPT-6.1-sol OpenAI · effort xhigh 3 of 3 runs (666 · 653 · 661)661 of 1000 LEB Silver
- Security246/250
- Architecture25/200
- Bugs129/150
- Performance150/150
- Clean code0/100
- Compatibility70/100
- Explanation41/50
Comment and details
Security at 246 of 250, the best outside Claude, with the CSV formula injection and MD5 both fixed in the official run. What holds it below the Claude models is one business value changed in each run (−30) and the nested ifs left as they were.
A written reading drawn from the scorecards and verdicts; it is not part of the score.
- Full marks
- Performance
- Zero
- Clean code
- Never fixed, in any run
- A dispatcher that does everything; Magic numbers for status and priority; Four levels of nested ifs
Run by run
- Run 1 · 6669 of 13 flaws fixedno false positive1 business value changed22/22 checks27minTokens: 73,937 in · 43,294 out · 1,233,792 cache · 1,351,023 totalScorecard
- Run 2 · 6539 of 13 flaws fixedno false positive1 business value changed22/22 checks26minTokens: 73,874 in · 33,443 out · 887,296 cache · 994,613 totalScorecard
- Run 3 · 66110 of 13 flaws fixedno false positive1 business value changed22/22 checks24minTokens: 60,781 in · 34,648 out · 953,728 cache · 1,049,157 totalScorecard
-
8
GPT-6.1-sol pro OpenAI · effort xhigh 1 of 3 runs654 of 1000 LEB Silver
- Security228/250
- Architecture0/200
- Bugs139/150
- Performance150/150
- Clean code25/100
- Compatibility70/100
- Explanation42/50
Comment and details
One run so far, through opencode, at US$ 1.01: 654, close to GPT-6.1-sol in Codex CLI. It left the CSV formula injection in place and changed one business value (−30).
A written reading drawn from the scorecards and verdicts; it is not part of the score.
- Full marks
- Performance
- Zero
- Architecture
- Never fixed, in any run
- Formula injection in the CSV export; A dispatcher that does everything; Magic numbers for status and priority; Four levels of nested ifs
Run by run
- Run 1 · 6549 of 13 flaws fixedno false positive1 business value changed22/22 checks16minUS$ 1.01Tokens: 126,508 in · 41,237 out · 810,414 cache · 1,000,316 totalScorecard
-
9
Grok 4.7 xAI · effort high 3 of 3 runs (638 · 663 · 607)638 of 1000 LEB Silver
- Security207/250
- Architecture0/200
- Bugs139/150
- Performance150/150
- Clean code0/100
- Compatibility100/100
- Explanation42/50
Comment and details
The strongest model from outside Anthropic and OpenAI, with compatibility at 100 in two of three runs and reports at 42 of 50. It kept the unsalted MD5 passwords in every run; in its first it was the only agent to look something up on the web, the PHP manual page for fputcsv.
A written reading drawn from the scorecards and verdicts; it is not part of the score.
- Full marks
- Performance, Compatibility
- Zero
- Architecture, Clean code
- Never fixed, in any run
- Unsalted MD5 passwords; A dispatcher that does everything; Magic numbers for status and priority; Four levels of nested ifs
Run by run
- Run 1 · 6388 of 13 flaws fixedno false positivecontract kept22/22 checks29minUS$ 2.43Tokens: 375,533 in · 19,450 out · 2,298,496 cache · 2,761,606 totalScorecard
- Run 2 · 6638 of 13 flaws fixedno false positivecontract kept22/22 checks24minUS$ 2.88Tokens: 464,772 in · 17,294 out · 2,760,320 cache · 3,320,861 totalScorecard
- Run 3 · 6078 of 13 flaws fixedno false positive1 business value changed22/22 checks26minUS$ 2.57Tokens: 388,897 in · 16,860 out · 2,347,904 cache · 2,839,851 totalScorecard
-
10
Grok 4.6 xAI · effort high 3 of 3 runs (633 · 640 · 620)633 of 1000 LEB Silver
- Security194/250
- Architecture0/200
- Bugs129/150
- Performance150/150
- Clean code25/100
- Compatibility100/100
- Explanation35/50
Comment and details
Nearly Grok 4.7's score for about an eighth of the cost (US$ 0.28 to 0.31 a run), and steady: 620 to 640, 9 flaws found in each run, no contract broken. It never touched MD5 or the CSV formula injection.
A written reading drawn from the scorecards and verdicts; it is not part of the score.
- Full marks
- Performance, Compatibility
- Zero
- Architecture
- Never fixed, in any run
- Formula injection in the CSV export; Unsalted MD5 passwords; A dispatcher that does everything; Magic numbers for status and priority; Four levels of nested ifs
Run by run
- Run 1 · 6337 of 13 flaws fixedno false positivecontract kept22/22 checks6minUS$ 0.31Tokens: 54,906 in · 8,789 out · 158,208 cache · 234,078 totalScorecard
- Run 2 · 6408 of 13 flaws fixedno false positivecontract kept22/22 checks4minUS$ 0.31Tokens: 53,592 in · 8,053 out · 158,720 cache · 232,132 totalScorecard
- Run 3 · 6207 of 13 flaws fixedno false positivecontract kept22/22 checks6minUS$ 0.28Tokens: 38,168 in · 8,547 out · 145,536 cache · 205,923 totalScorecard
-
11
Grok 4.7 xAI · effort xhigh 3 of 3 runs (640 · 617 · 631)631 of 1000 LEB Silver
- Security185/250
- Architecture0/200
- Bugs129/150
- Performance150/150
- Clean code25/100
- Compatibility100/100
- Explanation42/50
Comment and details
More effort scored slightly lower than Grok 4.7 at high, at a higher cost: it fixed 7 or 8 flaws a run and left MD5 and the hardcoded secrets in every one. Its reports, 42 to 44 of 50, are the best explanations from outside Anthropic and OpenAI.
A written reading drawn from the scorecards and verdicts; it is not part of the score.
- Full marks
- Performance, Compatibility
- Zero
- Architecture
- Never fixed, in any run
- Unsalted MD5 passwords; Secrets hardcoded in the config; A dispatcher that does everything; Magic numbers for status and priority; Four levels of nested ifs
Run by run
- Run 1 · 6408 of 13 flaws fixedno false positivecontract kept22/22 checks37minUS$ 3.82Tokens: 630,356 in · 24,515 out · 3,767,424 cache · 4,510,681 totalScorecard
- Run 2 · 6177 of 13 flaws fixedno false positivecontract kept22/22 checks29minUS$ 3.59Tokens: 607,690 in · 20,802 out · 3,405,184 cache · 4,125,277 totalScorecard
- Run 3 · 6317 of 13 flaws fixedno false positivecontract kept22/22 checks33minUS$ 3.31Tokens: 579,862 in · 20,500 out · 2,982,528 cache · 3,672,680 totalScorecard
-
12
Kimi K3 Moonshot AI · effort high 3 of 3 runs (629 · 634 · 574)629 of 1000 LEB Silver
- Security198/250
- Architecture0/200
- Bugs150/150
- Performance150/150
- Clean code25/100
- Compatibility70/100
- Explanation36/50
Comment and details
The strongest Kimi by a wide margin, with bugs and performance at full marks in its official run, for US$ 0.41 to 1.02 a run. Every run changed one business value (−30), and none fixed the CSV formula injection.
A written reading drawn from the scorecards and verdicts; it is not part of the score.
- Full marks
- Bugs, Performance
- Zero
- Architecture
- Never fixed, in any run
- Formula injection in the CSV export; A dispatcher that does everything; Magic numbers for status and priority; Four levels of nested ifs
Run by run
- Run 1 · 6298 of 13 flaws fixedno false positive1 business value changed22/22 checks7minUS$ 0.41Tokens: 28,836 in · 11,322 out · 279,808 cache · 324,940 totalScorecard
- Run 2 · 6349 of 13 flaws fixedno false positive1 business value changed22/22 checks21minUS$ 1.02Tokens: 52,871 in · 21,600 out · 1,249,024 cache · 1,334,129 totalScorecard
- Run 3 · 5747 of 13 flaws fixedno false positive1 business value changed22/22 checks15minUS$ 0.81Tokens: 48,057 in · 18,001 out · 934,144 cache · 1,008,035 totalScorecard
-
13
GPT-6-astra OpenAI · effort xhigh 3 of 3 runs (661 · 596 · 628)628 of 1000 LEB Silver
- Security228/250
- Architecture0/200
- Bugs139/150
- Performance150/150
- Clean code0/100
- Compatibility70/100
- Explanation41/50
Comment and details
Single-agent GPT-6-astra, 38 points below its ultra setting. Its runs disagree on what to fix: the first fixed the CSV formula injection and kept MD5, the official one migrated MD5 to password_hash and left the injection alone.
A written reading drawn from the scorecards and verdicts; it is not part of the score.
- Full marks
- Performance
- Zero
- Architecture, Clean code
- Never fixed, in any run
- A dispatcher that does everything; Magic numbers for status and priority; Four levels of nested ifs
Run by run
- Run 1 · 6619 of 13 flaws fixedno false positive1 business value changed22/22 checks17minTokens: 58,947 in · 31,166 out · 753,408 cache · 843,521 totalScorecard
- Run 2 · 5968 of 13 flaws fixedno false positive1 business value changed22/22 checks12minTokens: 55,599 in · 21,712 out · 549,120 cache · 626,431 totalScorecard
- Run 3 · 6289 of 13 flaws fixedno false positive1 business value changed22/22 checks14minTokens: 69,459 in · 26,327 out · 659,328 cache · 755,114 totalScorecard
-
14
GLM-5.3 Prime Z.AI · effort high 3 of 3 runs (635 · 628 · 541)628 of 1000 LEB Silver
- Security211/250
- Architecture0/200
- Bugs129/150
- Performance150/150
- Clean code0/100
- Compatibility100/100
- Explanation38/50
Comment and details
The strongest GLM, at about US$ 2 a run. Its first run fixed all four flaws the probes cover, the CSV formula injection among them; the third changed two business values and fell to 541.
A written reading drawn from the scorecards and verdicts; it is not part of the score.
- Full marks
- Performance, Compatibility
- Zero
- Architecture, Clean code
- Never fixed, in any run
- Unsalted MD5 passwords; A dispatcher that does everything; Magic numbers for status and priority; Four levels of nested ifs
Run by run
- Run 1 · 6359 of 13 flaws fixed1 false positivecontract kept22/22 checks14minUS$ 1.69Tokens: 66,011 in · 21,625 out · 2,077,696 cache · 2,182,652 totalScorecard
- Run 2 · 6288 of 13 flaws fixedno false positivecontract kept22/22 checks26minUS$ 2.13Tokens: 78,339 in · 24,496 out · 2,647,040 cache · 2,773,658 totalScorecard
- Run 3 · 5417 of 13 flaws fixedno false positive2 business values changed22/22 checks17minUS$ 1.77Tokens: 67,380 in · 20,363 out · 2,154,240 cache · 2,263,862 totalScorecard
-
15
GPT-5.6-terra OpenAI · effort xhigh 3 of 3 runs (625 · 611 · 645)625 of 1000 LEB Silver
- Security211/250
- Architecture0/200
- Bugs129/150
- Performance150/150
- Clean code0/100
- Compatibility100/100
- Explanation35/50
Comment and details
Reports fewer flaws than it fixes: its first run listed 7 and fixed 9, the rest done in passing without a word in the report. Compatibility held in two of three runs, which keeps it 15th despite the short report.
A written reading drawn from the scorecards and verdicts; it is not part of the score.
- Full marks
- Performance, Compatibility
- Zero
- Architecture, Clean code
- Never fixed, in any run
- A dispatcher that does everything; Magic numbers for status and priority; Four levels of nested ifs
Run by run
- Run 1 · 6259 of 13 flaws fixedno false positivecontract kept22/22 checks8minTokens: 54,590 in · 25,866 out · 486,656 cache · 567,112 totalScorecard
- Run 2 · 61110 of 13 flaws fixedno false positive1 business value changed22/22 checks8minTokens: 63,042 in · 25,647 out · 822,016 cache · 910,705 totalScorecard
- Run 3 · 64510 of 13 flaws fixedno false positivecontract kept22/22 checks7minTokens: 52,130 in · 22,262 out · 444,928 cache · 519,320 totalScorecard
-
16
GLM-5.3-Flash Z.AI · effort high 1 of 3 runs624 of 1000 LEB Silver
- Security211/250
- Architecture0/200
- Bugs129/150
- Performance150/150
- Clean code0/100
- Compatibility100/100
- Explanation34/50
Comment and details
One run so far: 624 for US$ 0.08, about a tenth of GLM-5.3's cost for a slightly higher score. It left the CSV formula injection and the hardcoded secrets in place.
A written reading drawn from the scorecards and verdicts; it is not part of the score.
- Full marks
- Performance, Compatibility
- Zero
- Architecture, Clean code
- Never fixed, in any run
- Formula injection in the CSV export; Secrets hardcoded in the config; A dispatcher that does everything; Magic numbers for status and priority; Four levels of nested ifs
Run by run
- Run 1 · 6248 of 13 flaws fixedno false positivecontract kept22/22 checks15minUS$ 0.08Tokens: 68,681 in · 22,218 out · 1,549,184 cache · 1,655,227 totalScorecard
-
17
GLM-5.3 Z.AI · effort high 3 of 3 runs (629 · 621 · 604)621 of 1000 LEB Silver
- Security203/250
- Architecture0/200
- Bugs129/150
- Performance150/150
- Clean code0/100
- Compatibility100/100
- Explanation39/50
Comment and details
Steady (604 to 629) and cheap (US$ 0.47 to 0.73 a run), with compatibility at 100 in all three runs. It never fixed MD5 or the CSV formula injection.
A written reading drawn from the scorecards and verdicts; it is not part of the score.
- Full marks
- Performance, Compatibility
- Zero
- Architecture, Clean code
- Never fixed, in any run
- Formula injection in the CSV export; Unsalted MD5 passwords; A dispatcher that does everything; Magic numbers for status and priority; Four levels of nested ifs
Run by run
- Run 1 · 6298 of 13 flaws fixedno false positivecontract kept22/22 checks17minUS$ 0.73Tokens: 65,504 in · 21,730 out · 1,778,176 cache · 1,883,245 totalScorecard
- Run 2 · 6217 of 13 flaws fixedno false positivecontract kept22/22 checks15minUS$ 0.51Tokens: 376,050 in · 21,002 out · 1,480,512 cache · 1,906,219 totalScorecard
- Run 3 · 6047 of 13 flaws fixedno false positivecontract kept22/22 checks13minUS$ 0.47Tokens: 63,302 in · 22,219 out · 1,457,920 cache · 1,561,006 totalScorecard
-
18
DeepSeek V4 Flash DeepSeek · effort xhigh 1 of 3 runs617 of 1000 LEB Silver
- Security207/250
- Architecture0/200
- Bugs129/150
- Performance150/150
- Clean code0/100
- Compatibility100/100
- Explanation31/50
Comment and details
One run so far, through Novita: 617 for US$ 0.18, against 282 for the same model at high. It fixed 7 flaws and left the CSV injection, MD5 and the hardcoded secrets in place.
A written reading drawn from the scorecards and verdicts; it is not part of the score.
- Full marks
- Performance, Compatibility
- Zero
- Architecture, Clean code
- Never fixed, in any run
- Formula injection in the CSV export; Unsalted MD5 passwords; Secrets hardcoded in the config; A dispatcher that does everything; Magic numbers for status and priority; Four levels of nested ifs
Run by run
- Run 1 · 6177 of 13 flaws fixedno false positivecontract kept22/22 checks28minUS$ 0.18Tokens: 163,821 in · 32,918 out · 4,833,280 cache · 5,068,978 totalScorecard
-
19
GPT-6.1-sol OpenAI · effort ultra 3 of 3 runs (597 · 656 · 616)616 of 1000 LEB Silver
- Security228/250
- Architecture0/200
- Bugs129/150
- Performance150/150
- Clean code0/100
- Compatibility70/100
- Explanation39/50
Comment and details
Ultra effort scored 45 points below GPT-6.1-sol at xhigh, across runs from 597 to 656. With three subagents its first run kept MD5; with two, its second fixed 10 flaws, the CSV formula injection and MD5 among them.
A written reading drawn from the scorecards and verdicts; it is not part of the score.
- Full marks
- Performance
- Zero
- Architecture, Clean code
- Never fixed, in any run
- A dispatcher that does everything; Magic numbers for status and priority; Four levels of nested ifs
Run by run
- Run 1 · 5978 of 13 flaws fixedno false positive1 business value changed22/22 checks24minTokens: 215,626 in · 39,255 out · 3,416,064 cache · 3,670,945 totalScorecard
- Run 2 · 65610 of 13 flaws fixedno false positive1 business value changed22/22 checks13minTokens: 177,994 in · 38,220 out · 2,064,000 cache · 2,280,214 totalScorecard
- Run 3 · 6169 of 13 flaws fixedno false positive1 business value changed22/22 checks11minTokens: 193,606 in · 40,785 out · 3,059,328 cache · 3,293,719 totalScorecard
-
20
GPT-5.6-sol OpenAI · effort xhigh 3 of 3 runs (612 · 612 · 608)612 of 1000 LEB Silver
- Security246/250
- Architecture0/200
- Bugs129/150
- Performance150/150
- Clean code0/100
- Compatibility70/100
- Explanation32/50
Comment and details
Security at 246 of 250 and the CSV formula injection fixed, but the fix also turned the '-' of a ticket with no technician into "'-", a new bug, and every run changed one business value (−30). The most repeatable GPT: 608 to 612.
A written reading drawn from the scorecards and verdicts; it is not part of the score.
- Full marks
- Performance
- Zero
- Architecture, Clean code
- Never fixed, in any run
- A dispatcher that does everything; Magic numbers for status and priority; Four levels of nested ifs
Run by run
- Run 1 · 61210 of 13 flaws fixedno false positive1 business value changed22/22 checks7minTokens: 52,386 in · 18,315 out · 1,163,520 cache · 1,234,221 totalScorecard
- Run 2 · 6129 of 13 flaws fixedno false positive1 business value changed22/22 checks15minTokens: 67,246 in · 32,457 out · 905,216 cache · 1,004,919 totalScorecard
- Run 3 · 6089 of 13 flaws fixedno false positive1 business value changed22/22 checks15minTokens: 63,699 in · 33,962 out · 1,569,920 cache · 1,667,581 totalScorecard
-
21
DeepSeek V4.1 Flash DeepSeek · effort high 3 of 3 runs (625 · 612 · 597)612 of 1000 LEB Silver
- Security203/250
- Architecture0/200
- Bugs129/150
- Performance150/150
- Clean code0/100
- Compatibility100/100
- Explanation30/50
Comment and details
The cheapest agent here, US$ 0.01 to 0.04 a run, two of them under two minutes, for 597 to 625. Compatibility held in all three; the lowest explanation score among the agents above 600 (30 of 50).
A written reading drawn from the scorecards and verdicts; it is not part of the score.
- Full marks
- Performance, Compatibility
- Zero
- Architecture, Clean code
- Never fixed, in any run
- Unsalted MD5 passwords; A dispatcher that does everything; Magic numbers for status and priority; Four levels of nested ifs
Run by run
- Run 1 · 6259 of 13 flaws fixedno false positivecontract kept22/22 checks9minUS$ 0.04Tokens: 30,817 in · 24,049 out · 2,467,712 cache · 2,543,405 totalScorecard
- Run 2 · 6128 of 13 flaws fixedno false positivecontract kept22/22 checks1minUS$ 0.01Tokens: 20,053 in · 12,321 out · 357,504 cache · 397,812 totalScorecard
- Run 3 · 5977 of 13 flaws fixedno false positivecontract kept22/22 checks1minUS$ 0.02Tokens: 27,064 in · 13,385 out · 535,040 cache · 585,035 totalScorecard
-
22
GPT-5.6-luna OpenAI · effort xhigh 3 of 3 runs (599 · 601 · 624)601 of 1000 LEB Silver
- Security220/250
- Architecture0/200
- Bugs129/150
- Performance150/150
- Clean code0/100
- Compatibility70/100
- Explanation32/50
Comment and details
The smallest GPT-5.6, within 24 points of terra and sol across 599 to 624. It changed one business value in each run (−30).
A written reading drawn from the scorecards and verdicts; it is not part of the score.
- Full marks
- Performance
- Zero
- Architecture, Clean code
- Never fixed, in any run
- A dispatcher that does everything; Magic numbers for status and priority; Four levels of nested ifs
Run by run
- Run 1 · 5998 of 13 flaws fixedno false positive1 business value changed22/22 checks12minTokens: 74,979 in · 30,975 out · 886,784 cache · 992,738 totalScorecard
- Run 2 · 6019 of 13 flaws fixedno false positive1 business value changed22/22 checks10minTokens: 65,711 in · 31,899 out · 706,304 cache · 803,914 totalScorecard
- Run 3 · 62410 of 13 flaws fixedno false positive1 business value changed22/22 checks14minTokens: 146,803 in · 42,381 out · 1,674,752 cache · 1,863,936 totalScorecard
-
23
GLM-5.3-FlashX Z.AI · effort high 1 of 3 runs597 of 1000 LEB Bronze
- Security185/250
- Architecture0/200
- Bugs129/150
- Performance150/150
- Clean code0/100
- Compatibility100/100
- Explanation33/50
Comment and details
One run so far: 597 for US$ 0.10, 27 points below GLM-5.3-Flash. It fixed 7 flaws and left the CSV injection, MD5 and the hardcoded secrets in place.
A written reading drawn from the scorecards and verdicts; it is not part of the score.
- Full marks
- Performance, Compatibility
- Zero
- Architecture, Clean code
- Never fixed, in any run
- Formula injection in the CSV export; Unsalted MD5 passwords; Secrets hardcoded in the config; A dispatcher that does everything; Magic numbers for status and priority; Four levels of nested ifs
Run by run
- Run 1 · 5977 of 13 flaws fixedno false positivecontract kept22/22 checks8minUS$ 0.10Tokens: 48,070 in · 13,431 out · 702,784 cache · 771,911 totalScorecard
-
24
Gemini 3.8 Flash Google · effort high 3 of 3 runs (687 · 588 · 100)588 of 1000 LEB Bronze
- Security181/250
- Architecture0/200
- Bugs129/150
- Performance150/150
- Clean code0/100
- Compatibility100/100
- Explanation28/50
Comment and details
Three very different runs: 687 (it would have placed 6th), 588, and 100 from a run that stopped after ten minutes on a database server it started itself. The two complete runs kept MD5 and both secrets and left the CSV export serving every client.
A written reading drawn from the scorecards and verdicts; it is not part of the score.
- Full marks
- Performance, Compatibility
- Zero
- Architecture, Clean code
- Never fixed, in any run
- Formula injection in the CSV export; Unsalted MD5 passwords; Secrets hardcoded in the config; A dispatcher that does everything; Magic numbers for status and priority
Run by run
- Run 1 · 6878 of 13 flaws fixedno false positivecontract kept22/22 checks12minUS$ 1.00Tokens: 447,362 in · 30,814 out · 5,007,269 cache · 5,532,273 totalScorecard
- Run 2 · 5887 of 13 flaws fixedno false positivecontract kept22/22 checks25minUS$ 0.85Tokens: 466,673 in · 14,937 out · 3,198,620 cache · 3,735,479 totalScorecard
- Run 3 · 1000 of 13 flaws fixedno false positivecontract kept22/22 checks10minUS$ 0.28Tokens: 214,747 in · 1,444 out · 704,735 cache · 937,623 totalScorecard
-
25
Gemini 3.7 Flash Google · effort high 3 of 3 runs (587 · 587 · 556)587 of 1000 LEB Bronze
- Security181/250
- Architecture0/200
- Bugs129/150
- Performance150/150
- Clean code0/100
- Compatibility100/100
- Explanation27/50
Comment and details
The generation before Gemini 3.8 Flash, one point behind it and far steadier (556 to 587). Its first run is the only one outside Claude to flatten the nested ifs; its second searched the VM for the answer key, which is not there.
A written reading drawn from the scorecards and verdicts; it is not part of the score.
- Full marks
- Performance, Compatibility
- Zero
- Architecture, Clean code
- Never fixed, in any run
- Formula injection in the CSV export; Unsalted MD5 passwords; Secrets hardcoded in the config; A dispatcher that does everything; Magic numbers for status and priority; Four levels of nested ifs
Run by run
- Run 1 · 5877 of 13 flaws fixedno false positivecontract kept22/22 checks12minUS$ 0.72Tokens: 358,682 in · 22,820 out · 3,161,645 cache · 3,577,678 totalScorecard
- Run 2 · 5877 of 13 flaws fixedno false positivecontract kept22/22 checks16minUS$ 0.60Tokens: 294,559 in · 16,180 out · 2,248,786 cache · 2,599,093 totalScorecard
- Run 3 · 5567 of 13 flaws fixedno false positive1 business value changed22/22 checks7minUS$ 0.49Tokens: 315,568 in · 15,092 out · 1,053,498 cache · 1,416,483 totalScorecard
-
26
GPT-5.5 OpenAI · effort xhigh 3 of 3 runs (558 · 568 · 536)558 of 1000 LEB Bronze
- Security177/250
- Architecture0/200
- Bugs129/150
- Performance150/150
- Clean code0/100
- Compatibility70/100
- Explanation32/50
Comment and details
The previous GPT generation, 43 to 108 points below the GPT-5.6 and GPT-6 models: 8 flaws found and 7 fixed in every run, MD5 and the CSV formula injection always left in place.
A written reading drawn from the scorecards and verdicts; it is not part of the score.
- Full marks
- Performance
- Zero
- Architecture, Clean code
- Never fixed, in any run
- Formula injection in the CSV export; Unsalted MD5 passwords; A dispatcher that does everything; Magic numbers for status and priority; Four levels of nested ifs
Run by run
- Run 2 · 5587 of 13 flaws fixedno false positive1 business value changed22/22 checks6minTokens: 54,617 in · 18,745 out · 374,272 cache · 447,634 totalScorecard
- Run 3 · 5687 of 13 flaws fixedno false positivecontract kept22/22 checks5minTokens: 47,993 in · 16,748 out · 387,584 cache · 452,325 totalScorecard
- Run 4 · 5367 of 13 flaws fixedno false positive1 business value changed22/22 checks5minTokens: 50,817 in · 16,437 out · 435,456 cache · 502,710 totalScorecard
-
27
Gemini 3.8 Flash Google · effort medium 1 of 3 runs550 of 1000 LEB Bronze
- Security147/250
- Architecture0/200
- Bugs129/150
- Performance150/150
- Clean code0/100
- Compatibility100/100
- Explanation24/50
Comment and details
One run so far, in 4min 34s for US$ 0.19: 550. It fixed 6 flaws and claimed SQL injection in two functions that only take integers.
A written reading drawn from the scorecards and verdicts; it is not part of the score.
- Full marks
- Performance, Compatibility
- Zero
- Architecture, Clean code
- Never fixed, in any run
- Formula injection in the CSV export; Session fixation at login; Unsalted MD5 passwords; Secrets hardcoded in the config; A dispatcher that does everything; Magic numbers for status and priority; Four levels of nested ifs
Run by run
- Run 1 · 5506 of 13 flaws fixed2 false positivescontract kept22/22 checks5minUS$ 0.19Tokens: 153,206 in · 11,176 out · 464,128 cache · 628,510 totalScorecard
-
28
Qwen3 Coder Next Alibaba (Qwen team) · default effort (not configurable) 1 of 3 runs507 of 1000 LEB Bronze
- Security125/250
- Architecture0/200
- Bugs129/150
- Performance150/150
- Clean code0/100
- Compatibility100/100
- Explanation18/50
Comment and details
The weakest explanation (18 of 50) and the worst calibration here: three flaws that cannot exist reported at confidence 100, and no tests run. It is, on the other hand, the only one of the bottom nine that fixed the N+1 query.
A written reading drawn from the scorecards and verdicts; it is not part of the score.
- Full marks
- Performance, Compatibility
- Zero
- Architecture, Clean code
- Never fixed, in any run
- Reflected XSS in the search; Formula injection in the CSV export; Unsalted MD5 passwords; Secrets hardcoded in the config; Any ticket readable by id (IDOR); A dispatcher that does everything; Magic numbers for status and priority; Four levels of nested ifs
Run by run
- Run 1 · 5075 of 13 flaws fixed3 false positivescontract kept22/22 checks16minUS$ 1.53Tokens: 7,393,841 in · 32,724 out · 0 cache · 7,426,565 totalScorecard
-
29
DeepSeek V4 Pro DeepSeek · effort high 3 of 3 runs (604 · 496 · 432)496 of 1000 LEB Bronze
- Security181/250
- Architecture0/200
- Bugs129/150
- Performance56/150
- Clean code0/100
- Compatibility100/100
- Explanation30/50
Comment and details
The larger DeepSeek: all three runs (604, 496 and 432) score below DeepSeek V4.1 Flash's official 612. The first rated SQL injection at confidence 100 in two functions that only take integers; the other two left the N+1 query in place.
A written reading drawn from the scorecards and verdicts; it is not part of the score.
- Full marks
- Compatibility
- Zero
- Architecture, Clean code
- Never fixed, in any run
- Formula injection in the CSV export; Unsalted MD5 passwords; A dispatcher that does everything; Magic numbers for status and priority; Four levels of nested ifs
Run by run
- Run 1 · 6048 of 13 flaws fixed2 false positivescontract kept22/22 checks8minUS$ 0.03Tokens: 68,936 in · 13,219 out · 322,560 cache · 412,383 totalScorecard
- Run 2 · 4966 of 13 flaws fixedno false positivecontract kept22/22 checks23minUS$ 0.42Tokens: 28,203 in · 15,517 out · 1,835,520 cache · 1,904,131 totalScorecard
- Run 3 · 4326 of 13 flaws fixed1 false positivecontract kept22/22 checks12minUS$ 0.04Tokens: 103,562 in · 15,859 out · 580,864 cache · 708,898 totalScorecard
-
30
MiniMax-M3 MiniMax · effort thinking 3 of 3 runs (462 · 616 · 434)462 of 1000 LEB Bronze
- Security220/250
- Architecture0/200
- Bugs150/150
- Performance0/150
- Clean code25/100
- Compatibility40/100
- Explanation27/50
Comment and details
Fixed 8 flaws in every run, with bugs at full marks, but changed business values (compatibility at 40 in its official run) and left the N+1 query. Its second run scored 616; its third broke four of the 22 characterization checks.
A written reading drawn from the scorecards and verdicts; it is not part of the score.
- Full marks
- Bugs
- Zero
- Architecture, Performance
- Never fixed, in any run
- Formula injection in the CSV export; A dispatcher that does everything; Magic numbers for status and priority
Run by run
- Run 1 · 4628 of 13 flaws fixedno false positive1 business value changed22/22 checks4minUS$ 0.12Tokens: 55,144 in · 19,131 out · 993,940 cache · 1,085,756 totalScorecard
- Run 2 · 6168 of 13 flaws fixedno false positive1 business value changed22/22 checks6minUS$ 0.10Tokens: 69,677 in · 14,753 out · 601,806 cache · 707,582 totalScorecard
- Run 3 · 4348 of 13 flaws fixed1 false positive1 business value changed18/22 checks5minUS$ 0.13Tokens: 63,000 in · 22,731 out · 1,089,762 cache · 1,188,744 totalScorecard
-
31
Kimi K2.7 Code Moonshot AI · default effort (not configurable) 3 of 3 runs (415 · 415 · 512)415 of 1000 LEB Bronze
- Security203/250
- Architecture0/200
- Bugs86/150
- Performance0/150
- Clean code0/100
- Compatibility100/100
- Explanation26/50
Comment and details
Left the N+1 query in every run and reported flaws that do not exist in all three, SQL injection in functions that only take integers among them. Its third run, 512, fixed one flaw more than the other two.
A written reading drawn from the scorecards and verdicts; it is not part of the score.
- Full marks
- Compatibility
- Zero
- Architecture, Performance, Clean code
- Never fixed, in any run
- Formula injection in the CSV export; Unsalted MD5 passwords; One query per ticket for the technician (N+1); A dispatcher that does everything; Magic numbers for status and priority; Four levels of nested ifs
Run by run
- Run 1 · 4156 of 13 flaws fixed2 false positivescontract kept22/22 checks14minUS$ 0.48Tokens: 55,678 in · 20,809 out · 1,616,640 cache · 1,702,051 totalScorecard
- Run 2 · 4156 of 13 flaws fixed1 false positivecontract kept22/22 checks4minUS$ 0.25Tokens: 43,066 in · 10,167 out · 766,976 cache · 826,182 totalScorecard
- Run 3 · 5127 of 13 flaws fixed2 false positivescontract kept22/22 checks15minUS$ 0.81Tokens: 80,899 in · 20,334 out · 2,925,824 cache · 3,051,753 totalScorecard
-
32
GPT-5.3-Codex OpenAI · effort xhigh 1 of 3 runs403 of 1000 LEB Bronze
- Security147/250
- Architecture0/200
- Bugs129/150
- Performance0/150
- Clean code0/100
- Compatibility100/100
- Explanation27/50
Comment and details
One run so far, through opencode: 403, the lowest GPT. It found 6 flaws and fixed 5, leaving the N+1 query, the session fixation at login and MD5 in place, with no false positive.
A written reading drawn from the scorecards and verdicts; it is not part of the score.
- Full marks
- Compatibility
- Zero
- Architecture, Performance, Clean code
- Never fixed, in any run
- Formula injection in the CSV export; Session fixation at login; Unsalted MD5 passwords; Secrets hardcoded in the config; One query per ticket for the technician (N+1); A dispatcher that does everything; Magic numbers for status and priority; Four levels of nested ifs
Run by run
- Run 1 · 4035 of 13 flaws fixedno false positivecontract kept22/22 checks7minUS$ 0.54Tokens: 42,455 in · 7,374 out · 351,744 cache · 422,872 totalScorecard
-
33
Kimi K2.7 Code (highspeed) Moonshot AI · default effort (not configurable) 3 of 3 runs (402 · 363 · 402)402 of 1000 LEB Bronze
- Security155/250
- Architecture0/200
- Bugs129/150
- Performance0/150
- Clean code25/100
- Compatibility70/100
- Explanation23/50
Comment and details
The fast variant of Kimi K2.7 Code, 13 points below it: 5 flaws fixed in every run, two flaws that do not exist reported in each, and the N+1 query always left in place.
A written reading drawn from the scorecards and verdicts; it is not part of the score.
- Zero
- Architecture, Performance
- Never fixed, in any run
- Formula injection in the CSV export; Session fixation at login; Unsalted MD5 passwords; Secrets hardcoded in the config; One query per ticket for the technician (N+1); A dispatcher that does everything; Magic numbers for status and priority; Four levels of nested ifs
Run by run
- Run 1 · 4025 of 13 flaws fixed2 false positives1 business value changed22/22 checks2minUS$ 0.38Tokens: 31,371 in · 10,254 out · 506,880 cache · 554,104 totalScorecard
- Run 2 · 3635 of 13 flaws fixed2 false positives1 business value changed22/22 checks6minUS$ 0.68Tokens: 48,822 in · 12,842 out · 960,722 cache · 1,037,685 totalScorecard
- Run 3 · 4025 of 13 flaws fixed2 false positivescontract kept22/22 checks10minUS$ 1.16Tokens: 73,637 in · 20,792 out · 2,034,944 cache · 2,139,219 totalScorecard
-
34
GLM-5.2 Z.AI · effort high 1 of 3 runs388 of 1000 Failed
- Security172/250
- Architecture0/200
- Bugs86/150
- Performance0/150
- Clean code0/100
- Compatibility100/100
- Explanation30/50
Comment and details
One run so far: 388, below the pass line and 233 points under GLM-5.3. It left the N+1 query, the leaked file handle and the session fixation in place; what it did report was accurate, with no false positive.
A written reading drawn from the scorecards and verdicts; it is not part of the score.
- Full marks
- Compatibility
- Zero
- Architecture, Performance, Clean code
- Never fixed, in any run
- Formula injection in the CSV export; Session fixation at login; Unsalted MD5 passwords; File handle leaked on the error path; One query per ticket for the technician (N+1); A dispatcher that does everything; Magic numbers for status and priority; Four levels of nested ifs
Run by run
- Run 1 · 3885 of 13 flaws fixedno false positivecontract kept22/22 checks6minUS$ 0.35Tokens: 52,899 in · 13,001 out · 716,992 cache · 790,478 totalScorecard
-
35
Nex N2.5 Pro Nex AGI · effort high 2 of 3 runs (317 · 428)317 of 1000 Failed
- Security194/250
- Architecture0/200
- Bugs107/150
- Performance75/150
- Clean code0/100
- Compatibility70/100
- Explanation31/50
Comment and details
Too slow to finish three runs: 10h 03min and 19h 35min, and a third voided after more than 21 hours when the operator's connection failed, so it stays at two and publishes the lower. Both runs were strong on security and both broke the contract: the first made every function return nothing without a logged-in user (8 checks, −160, 317), the second made the CSV export throw for a caller that has already printed output (3 checks, 428).
A written reading drawn from the scorecards and verdicts; it is not part of the score.
- Zero
- Architecture, Clean code
- Never fixed, in any run
- A dispatcher that does everything; Magic numbers for status and priority; Four levels of nested ifs
Run by run
- Run 1 · 3179 of 13 flaws fixedno false positive1 business value changed14/22 checks10h 3minUS$ 2.80Tokens: 29,260,714 in · 117,982 out · 20,215,616 cache · 49,594,312 totalScorecard
- Run 2 · 42810 of 13 flaws fixedno false positive2 business values changed19/22 checks19h 35minUS$ 4.87Tokens: 50,033,318 in · 254,132 out · 33,604,416 cache · 83,891,866 totalScorecard
-
36
Claude Haiku 4.5 Anthropic · default effort (not configurable) 3 of 3 runs (317 · 369 · 232)317 of 1000 Failed
- Security138/250
- Architecture0/200
- Bugs86/150
- Performance0/150
- Clean code0/100
- Compatibility70/100
- Explanation23/50
Comment and details
The small Claude, at US$ 0.21 to 0.40 a run, below the pass line in all three. Its first run's visibility fix hides a client's own tickets, comparing an integer with the string mysqli returns; its third broke three characterization checks.
A written reading drawn from the scorecards and verdicts; it is not part of the score.
- Zero
- Architecture, Performance, Clean code
- Never fixed, in any run
- Formula injection in the CSV export; Session fixation at login; Unsalted MD5 passwords; File handle leaked on the error path; One query per ticket for the technician (N+1); A dispatcher that does everything; Magic numbers for status and priority; Four levels of nested ifs
Run by run
- Run 1 · 3174 of 13 flaws fixed2 false positives1 business value changed22/22 checks2minUS$ 0.21Tokens: 1,062 in · 13,699 out · 819,787 cache · 834,548 totalScorecard
- Run 2 · 3695 of 13 flaws fixed2 false positivescontract kept22/22 checks7minUS$ 0.40Tokens: 1,198 in · 20,213 out · 2,111,822 cache · 2,133,233 totalScorecard
- Run 3 · 2325 of 13 flaws fixed1 false positive1 business value changed19/22 checks10minUS$ 0.27Tokens: 1,213 in · 17,741 out · 1,063,175 cache · 1,082,129 totalScorecard
-
37
DeepSeek V4 Flash DeepSeek · effort high 2 of 3 runs (612 · 282)282 of 1000 Failed
- Security99/250
- Architecture0/200
- Bugs96/150
- Performance0/150
- Clean code0/100
- Compatibility100/100
- Explanation22/50
Comment and details
Two runs that could hardly differ more: the first fixed 9 flaws for 612, the second's SQL-injection fix throws on every search and its client CSV comes out on one line, 282. It publishes the lower; which weights the hosts served under this name is not certain.
A written reading drawn from the scorecards and verdicts; it is not part of the score.
- Full marks
- Compatibility
- Zero
- Architecture, Performance, Clean code
- Never fixed, in any run
- Formula injection in the CSV export; A dispatcher that does everything; Magic numbers for status and priority; Four levels of nested ifs
Run by run
- Run 1 · 6129 of 13 flaws fixedno false positive1 business value changed22/22 checks9minUS$ 0.07Tokens: 85,100 in · 16,008 out · 1,518,080 cache · 1,646,803 totalScorecard
- Run 2 · 2823 of 13 flaws fixed1 false positivecontract kept21/22 checks7minUS$ 0.00Tokens: 43,443 in · 8,858 out · 445,696 cache · 510,802 totalScorecard
Flaw by flaw
What each agent found and fixed among the planted flaws. The hard ones are flaws of absence — a missing authorization check, a session never regenerated, a file left open on the error path.
The table shows the top 10 of 37 agents. Every agent's result, flaw by flaw, is in its scorecard.
- fixed
- found, not fixed
- missed
| Flaw | Claude Sonnet 5.5 · xhigh | Claude Sonnet 5.5 · max | Claude Sonnet 5.5 · max (ultracode) | Claude Fable 5.1 | Claude Opus 5.5 | GPT-6-astra · ultra | GPT-6.1-sol · xhigh | GPT-6.1-sol pro | Grok 4.7 · high | Grok 4.6 |
|---|---|---|---|---|---|---|---|---|---|---|
| SQL injection in the search | 10/10fixed | 10/10fixed | 10/10fixed | 10/10fixed | 10/10fixed | 10/10fixed | 10/10fixed | 10/10fixed | 10/10fixed | 10/10fixed |
| Reflected XSS in the search | 8/8fixed | 8/8fixed | 8/8fixed | 8/8fixed | 8/8fixed | 8/8fixed | 8/8fixed | 8/8fixed | 8/8fixed | 8/8fixed |
| Formula injection in the CSV export | 6/6fixed | 6/6fixed | 6/6fixed | 2/6found, not fixed | 2/6found, not fixed | 6/6fixed | 6/6fixed | 2/6found, not fixed | 6/6fixed | 0/6missed |
| Session fixation at login | 8/8fixed | 8/8fixed | 8/8fixed | 8/8fixed | 8/8fixed | 8/8fixed | 8/8fixed | 8/8fixed | 8/8fixed | 8/8fixed |
| Unsalted MD5 passwords | 8/8fixed | 8/8fixed | 8/8fixed | 8/8fixed | 8/8fixed | 3/8found, not fixed | 8/8fixed | 8/8fixed | 3/8found, not fixed | 3/8found, not fixed |
| Secrets hardcoded in the config | 8/8fixed | 8/8fixed | 8/8fixed | 8/8fixed | 8/8fixed | 8/8fixed | 8/8fixed | 8/8fixed | 3/8found, not fixed | 6/8found, not fixed |
| Any ticket readable by id (IDOR) | 10/10fixed | 10/10fixed | 10/10fixed | 10/10fixed | 10/10fixed | 9/10fixed | 9/10fixed | 9/10fixed | 10/10fixed | 10/10fixed |
| Division by zero in the SLA average | 8/8fixed | 8/8fixed | 8/8fixed | 8/8fixed | 8/8fixed | 8/8fixed | 8/8fixed | 8/8fixed | 8/8fixed | 8/8fixed |
| File handle leaked on the error path | 5/6fixed | 6/6fixed | 4/6fixed | 5/6fixed | 5/6fixed | 4/6fixed | 4/6fixed | 5/6fixed | 5/6fixed | 4/6fixed |
| One query per ticket for the technician (N+1) | 8/8fixed | 8/8fixed | 8/8fixed | 8/8fixed | 8/8fixed | 8/8fixed | 8/8fixed | 8/8fixed | 8/8fixed | 8/8fixed |
| A dispatcher that does everything | 2/10found, not fixed | 1/10found, not fixed | 0/10missed | 0/10missed | 4/10found, not fixed | 2/10found, not fixed | 2/10found, not fixed | 0/10missed | 0/10missed | 0/10missed |
| Magic numbers for status and priority | 0/6missed | 0/6missed | 0/6missed | 0/6missed | 0/6missed | 0/6missed | 0/6missed | 0/6missed | 0/6missed | 0/6missed |
| Four levels of nested ifs | 8/8fixed | 8/8fixed | 8/8fixed | 8/8fixed | 0/8missed | 2/8found, not fixed | 0/8missed | 2/8found, not fixed | 0/8missed | 2/8found, not fixed |
What stood out
- Six agents fixed the formula injection in the CSV (SEC-008) — Sonnet 5.5 in all three of its settings, GPT-6.1-sol, GPT-5.6-sol and Grok 4.7 — with the fix the answer key expects; Sonnet 5.5 is the only model with security at 250 of 250, in all three of its settings. GPT-5.6-sol also turned the
-of a ticket with no technician into'-, a new bug (−15). - Architecture was the weakest category for everyone: 50 of 200 for Opus 5.5, 25 for Sonnet 5.5 at xhigh and for GPT-6.1-sol, 12 for Sonnet 5.5 at max, and 0 for the other twenty-eight. Nobody split the dispatcher that does everything; those three named it and declined to restructure it.
- Two published runs broke the contract mechanically. All thirty-seven stayed on mysqli and reported no decoy, and thirty-five kept the 22 characterization checks green; DeepSeek V4 Flash publishes a run whose search throws, and Nex N2.5 Pro one whose functions return nothing without a logged-in user (8 checks broken, −160). Judgement is what separated the rest: twenty-four kept compatibility at 100, while the other thirteen each changed at least one business value (−30 each), most often the SLA average, scoped to each client.
- Twenty-six scores are official: Claude Sonnet 5.5 at xhigh, 809 (runs of 825, 809 and 724), Claude Sonnet 5.5 at max, 807 (807, 773 and 820), Claude Sonnet 5.5 at max in multi-agent mode, 774 (774, 820 and 759), Claude Opus 5.5, 717 (711, 717 and 805), GPT-6.1-sol, 661 (666, 653 and 661), GPT-6.1-sol at ultra, 616 (597, 656 and 616), Grok 4.7, 638 (638, 663 and 607), Grok 4.6, 633 (633, 640 and 620), Grok 4.7 at xhigh, 631 (640, 617 and 631), GPT-6-astra at ultra, 666 (668, 666 and 651), Kimi K3 at high, 629 (629, 634 and 574), GPT-6-astra, 628 (661, 596 and 628), GPT-5.6-terra, 625 (625, 611 and 645), GLM-5.3 Prime, 628 (635, 628 and 541), GLM-5.3, 621 (629, 621 and 604), DeepSeek V4.1 Flash, 612 (625, 612 and 597), GPT-5.6-sol, 612 (612, 612 and 608), GPT-5.6-luna, 601 (599, 601 and 624), Gemini 3.8 Flash at high, 588 (687, 588 and 100), GPT-5.5, 558 (558, 568 and 536), DeepSeek V4 Pro, 496 (604, 496 and 432), MiniMax-M3 with the thinking variant, 462 (462, 616 and 434), Kimi K2.7 Code, 415 (415, 415 and 512), Kimi K2.7 Code highspeed, 402 (402, 363 and 402), Gemini 3.7 Flash at high, 587 (587, 587 and 556), and Claude Haiku 4.5, 317 (317, 369 and 232), each the median of three runs. A single run can sit more than 100 points from the median: Opus's third scored 805, Sonnet's third 724, DeepSeek V4 Pro's first 604, Gemini 3.8 Flash's third, which stopped after ten minutes, 100, and astra's first, 661, had placed it 5th, from judgement calls such as which flaws to leave unfixed. Claude Fable 5.1 has two runs, 781 and 764, and publishes the lower, 4th
- Gemini 3.8 Flash at high is official at 588, 24th, Bronze, the median of three very different runs (687, 588 and 100). Gemini 3.7 Flash, the generation before, is official at 587 (587, 587 and 556), 25th. Its first run, 6th, fixed 8 of the 13 planted flaws and is the only one outside Claude to flatten the nested ifs (CLN-007); its second fixed 7, and searched the VM for the answer key, by name and by the hash quoted in the task. The key is not on the VM, and the search turned up nothing the run did not already have. Its third stopped after ten minutes, on a database server it started that kept its own shell command from ever returning: no report, the code untouched, scored as delivered. The two complete runs kept MD5 and both secrets and left the CSV export serving every client. At medium effort the same model scored 550 (27th) in 4min 34s: it fixed 6 flaws and claimed SQL injection in two functions that only take integers.
- GPT-6-astra at ultra is the strongest GPT agent here, official at 666 (runs of 668, 666 and 651), 6th, with two or three subagents in 10 to 15 minutes. GPT-6.1-sol follows, official at 661 (666, 653 and 661), 7th, and GPT-6.1-sol pro at 654, 8th, from one run. GPT-6-astra at xhigh, official at 628, is 13th. Its first run fixed the CSV injection and kept MD5; its official run did the opposite, migrating MD5 to
password_hashand leaving the CSV injection alone; GPT-6.1-sol's official run fixed both - Multi-agent work matched one agent at best, at many times the cost. Claude Sonnet 5.5 in Claude Code's multi-agent mode, at max effort, scored 774 in its first run (7 workflows, 68 subagents, 7h 55min, US$ 233), 820 in its second (2 workflows, 105 subagents, 2h, US$ 171) and 759 in its third (3 workflows, 145 subagents, 3h 34min, US$ 119). It is official at 774, 3rd, Gold. The same model at the same max effort as a single agent took about half an hour a run and is official at 807 (runs of 807, 773 and 820), 2nd; at xhigh it is official at 809. Its third report scored 47 of 50, the best explanation here, and no run touched the architecture. In each run a few subagents (nine, eight, then seven) fell back to an older Sonnet after a safety classifier stopped them; none of any delivery came from them.
- GPT-6.1-sol at ultra is official 45 points below itself at xhigh (616 against 661), the median of three runs spread over 59 points (597, 656 and 616). With three subagents, its first found 9 of the 13 planted flaws, kept MD5 and did not name the dispatcher; with two, its second fixed 10, the CSV injection and MD5 among them. Each took under half an hour, not the 8 hours of the multi-agent Sonnet.
- Grok 4.7 is the strongest model here from outside Anthropic and OpenAI (638, 9th, Silver, official over three runs of 638, 663 and 607), with an explanation at the level of the best GPT reports (42 of 50). In its first run it is also the only agent that used the web, to read the PHP manual page for fputcsv; the VM blocks GitHub, not the web. At xhigh, Grok 4.7 is official at 631 (640, 617 and 631), 11th, below its own high effort, with reports scored 42 to 44 of 50, the best explanations from outside Anthropic and OpenAI. Grok 4.6 follows, official at 633 (10th; runs of 633, 640 and 620), for about an eighth of the cost (US$ 0.31 against 2.43).
- GLM-5.3 Prime is the strongest GLM model (628, 14th, Silver), official over three runs through two hosts (635, 628 and 541): the first fixed all four probe-covered flaws, the CSV injection among them, for US$ 1.69; the second left the injection alone. GLM-5.3 is official at 621 (17th), the median of three runs (629, 621 and 604), the last two served by another host; GLM-5.3-Flash scores 624 for about a tenth of the cost, and GLM-5.3-FlashX 597
- DeepSeek V4.1 Flash is the cheapest agent here (US$ 0.01 to 0.04 a run; its last two took under two minutes each) and is official at 612 (runs of 625, 612 and 597), 21st. DeepSeek V4 Flash at xhigh, through Novita, scored 617 in its first run (18th). At high, run through two other hosts, it publishes 282, the lower of its two runs (612 and 282), last: the first fixed eight flaws fully, while the second's SQL-injection fix throws on every search and its client CSV comes out on one line. DeepSeek now routes the V4 Flash name to V4.1 on its own API, so which weights those hosts served is not certain. DeepSeek V4 Pro, the larger model, is official at 496, the median of three runs through two hosts (604, 496 and 432), all below DeepSeek V4.1 Flash's official 612: the first rates SQL injection at confidence 100 in two functions that only take integers, and the other two left the N+1 query in place.
- Twenty-four agents did not run at xhigh. Kimi K2.7 Code (in both of Moonshot's ids), Qwen3 Coder Next and Claude Haiku 4.5 have no effort setting and ran at their model's default; the two Grok models, the five GLM models, the three DeepSeek models and Gemini 3.8 and 3.7 Flash and Nex N2.5 Pro ran at high, in opencode, and Gemini 3.8 Flash also at medium; MiniMax-M3 ran in opencode with its thinking variant and Kimi K3 with its high variant; Sonnet 5.5 ran twice at max, once in multi-agent mode, and GPT-6.1-sol and GPT-6-astra once each at ultra. The rest ran at xhigh.
- Qwen3 Coder Next, DeepSeek V4 Pro, MiniMax-M3 with the thinking variant, Kimi K2.7 Code, GPT-5.3-Codex, Kimi K2.7 Code highspeed, GLM-5.2, Nex N2.5 Pro, Claude Haiku 4.5 and DeepSeek V4 Flash close the table (507, 496, 462, 415, 403, 402, 388, 317, 317 and 282; the last four are below the pass line). Nex N2.5 Pro is too slow to finish three runs: 10h 03min and 19h 35min, the longest single-agent runs here, and a third voided after more than 21 hours, so it stays at two; both fixed most of the security flaws, both broke the contract, and it publishes the lower, 317 (317 and 428). Claude Haiku 4.5 is official at 317 (runs of 317, 369 and 232); in the first, 2.5 minutes and US$ 0.21, its visibility fix hides a client's own tickets on the main page, comparing an integer with the string mysqli returns. All but Qwen3 Coder Next left the N+1 query in place. Qwen3 Coder Next has the weakest explanation (18 of 50) and the worst calibration (Brier 0.301): it reported three flaws that cannot exist at confidence 100, SQL injection in two functions that only take integers among them, as Kimi K2.7 Code did. It also ran no tests.
- Places 9 to 23 sit within 41 points (638 to 597), inside the noise of a single run. GPT-5.6-terra reported the fewest planted flaws and still ranks 15th, on compatibility and on what it did fix.
- The GPT models are among the best calibrated (Brier 0.015 or less): fewer findings, each stated with high confidence and nearly all real.
Finding is not fixing
Side by side, the three Claude models find nearly the same flaws and fix different numbers of them. One total hides which of the two it is measuring.
- They find nearly the same. In the run that counts for each, Sonnet 5.5 at xhigh reported 12 of the 13 planted flaws and Fable 5.1 and Opus 5.5 11; across their eight runs, each found 11 or 12
- Sonnet 5.5 fixes more, in the run that counts. Its official run fixed 11 of the 13, against 9 for Opus's official run and 10 for Fable's. Run by run it varies: Sonnet fixed 11, 11 and 8, Opus 9 in each of its three, Fable 9 and 10. Fable and Opus found and explained the formula injection in the CSV and left it in place, to protect the file's consumers; Sonnet fixed it in two of its three runs, prefixing only the cells that start a formula. Its 92-point lead over Opus's official 717 comes from clean code (+100), not from security (+17); its 45-point lead over Fable's 764 is security (+17) and architecture (+25)
- More compute did not change the pattern. The single-agent Claude runs took 16 to 27 minutes each, and the three at the same max effort scored 807, 773 and 820. Sonnet 5.5 in multi-agent mode took 7h 55min, 2h and 3h 34min and US$ 233, 171 and 119 for 774, 820 and 759, and is official at 774 against the single agent's 807: at best it tied the single-agent 820, with the same fixes and a deeper check of its own code, and in none of its three runs did the dispatcher reach its report.
- How to read it. On this task the three Claude models see the same problems; what separates them is how far they go in fixing them within the contract. In the median, Sonnet went further; a single run can reverse it, as Sonnet's third did. One task, two or three runs per model: a pattern worth testing, not a verdict
Read this before quoting a number
- Up to three runs per agent. An official LEB score is the median of three independent runs. Until an agent has three, its score is the lower of its totals so far, the totals are listed next to it, and everything shown with the score comes from that same run.
- The judge is an AI. Claude Opus 5.5 applied the published rubric to every delivery without knowing which model wrote it — each was anonymised — and the explanation was scored by a separate judge that saw neither the answer key nor the other scores.
- The judge is also a contestant. Claude Opus 5.5 is one of the agents evaluated, and six of the thirty-seven are Claude models, Sonnet 5.5 three times: they hold the top five places and the second to last. Anonymity limits that bias; it does not remove it, since a model can recognise its own style. Every verdict is published with its rationale, flaw by flaw, and the four verdicts changed in review say why; one of them adds 41 points to GPT-5.6-terra's total.
- Twenty-one attempts were left unscored, each before any judge saw it. Twice, Claude Fable 5.1's safeguards stopped one of its responses while it worked on the security flaws, and its client handed the rest of the run to Claude Opus 4.8, which wrote the whole report; a run must come from one model, so both are void, and Fable 5.1 shows its two complete runs. Four ended on the provider's side before the delivery was complete: GPT-5.6-sol pro when its gateway ran out of credit, and Gemini 3.8 Flash three times, on a gateway timeout and twice on a rate limit. Five were set up wrong: the first multi-agent Sonnet 5.5 run was stopped to give the machine more processors, the second found its client logged out and never reached the model, as did Claude Fable 5.1's third, Claude Haiku 4.5's first attempt had its client mode changed mid-run, and a Gemini attempt was started outside the task's folder. One finished, but GPT-5.6-terra ran in a different client from its first run, and was repeated in the same one. Six finished after their agent already had its three runs, two of GPT-6.1-sol at xhigh, three at ultra and one of Gemini 3.7 Flash; a fourth run would let the published score be chosen. Two finished, a Grok 4.7 run at xhigh and a Kimi K3 run at high, but their VMs were restored before the deliveries were copied off them, so nothing was left to score. One was cut off by the operator's own connection: Nex N2.5 Pro's third run, after more than 21 hours, which the operator chose not to repeat because the model is too slow. All twenty-one are kept in the repository.
- The answer key is public. The failure matrix of LEB-100-A has been in the public repository since 13 July 2026. The runs of 29 September had the VM's name block only, so no agent could fetch it by accident; the runs of 30 September also had GitHub's addresses blocked, and the session logs of every run that left one show no request to GitHub at all. What a model saw in training depends on how far its training data reaches: each run records the cutoff its provider publishes. Without one, the provider's release date bounds it, since a model cannot train on data from after its release. A model whose cutoff or release is later, or that has neither, is marked in the leaderboard.
- The benchmark's own tests were fixed. Scoring these runs exposed two defects in the evaluation tooling: an SQL loader that split a statement on a semicolon inside a comment, and CSV checks that read a temporary file the contract never promised. Both were fixed before scoring, the same way for every agent, and are recorded in the repository.
- The machine changed on 30 September. Until then the agents ran as the VM's administrator, with sudo, on a machine that kept earlier runs' leftovers. Their session logs show no agent reading another's work; the opencode agents shared a leftover test database that held only the seed rows. From Qwen3 Coder Next on, runs are made as an unprivileged user on a machine cleaned before each run.
- Not every run parameter was recorded. The exact model version and the temperature were not, and cost and model time only where the client kept them; the three runs that left no session log at all (GPT-5.5's first, MiniMax-M3's at its default and Kimi K3's) were withdrawn on 4 October 2026 and are not published. MiniMax-M3 at its default is gone from the table; Kimi K3 is back with a recorded run at high. Each run marks what is missing instead of guessing.
Audit it
Every delivery, mechanical report, verdict and scorecard is in the repository, next to the specification that produced them. The same data is here as two spreadsheets: one row per run, and one per run and planted flaw.
Results on GitHub Specification Runs (CSV) Flaw by flaw (CSV) What each column means
The numbers on this page are not typed by hand: they are rendered from the published results of the ai-benchmark repository, the same data samirhv.com.br shows.