LEB-300
AI Benchmark · LEB · exploratory pilot
The first level, LEB-100, is the reference instance: every agent evaluated so far has run on it, and every run is published.
The second level of LEB: an application of about 3,000 lines, in multiple files, whose flaws cross files. The instance is active, so only the aggregate of each agent's runs is published. Its answer key is public on GitHub with the instance, and the machines that run the agents cannot reach GitHub. How a run works, how it is scored and how this level differs from LEB-100 is on the Benchmark page.
Top 6 so far
LEB-300-A- 1Claude Sonnet 5.5 · max932LEB Platinum
- 2Claude Sonnet 5.5 · xhigh860LEB Gold
- 3Claude Sonnet 5.5 · high855LEB Gold
- 4Claude Opus 5.5 · high848LEB Gold
- 5Claude Opus 5.5 · xhigh845LEB Gold
- 6Claude Fable 5.1836LEB Gold
Score out of 1000, of 27 agents. Not official until each has three runs. Full leaderboard
The results so far
One line per agent: the total, the grade, the score in each category, how many runs it rests on, and the cost and the time. Every agent solved the same package, byte for byte, so the numbers compare like for like.
LEB-300-A v1.0 · An application of about 3,000 lines
An exploratory pilot. An official score is the median of three runs of an agent. A line that rests on fewer runs is not official: it says how many it has, and is listed all the same, as on the LEB-100 page. The instance itself is still a pilot: its difficulty has not been homologated. Only the aggregate is published; each run's delivery, verdict and flaw-by-flaw results stay in a private archive. The answer key is public on GitHub with the instance, since 12 October 2026, after the first runs; the machines that ran the agents cannot reach GitHub, and every run records its model's training cutoff.
- mode A · 60 turns
- edition 2026
- key public
- matrix c42c82878d2c…
Showing 27 of 27 agents. Ranks stay those of the full leaderboard.
-
1
Claude Sonnet 5.5 Anthropic · effort max 1 of 3 runs932 of 1000 LEB Platinum
- Security250/250
- Architecture178/200
- Bugs134/150
- Performance140/150
- Clean code89/100
- Compatibility100/100
- Explanation41/50
Comment and details
On a single run: security 250 of 250, architecture 178 of 200, bugs 134 of 150, performance 140 of 150 and clean code 89 of 100, with the contract untouched (compatibility 100) and no penalty. It is the highest score of any run so far, above the 911 of the same model at xhigh effort, whose published score is 860 over two runs. It fixed 34 of the 36 planted flaws and reported 31, with one false positive, and its report was rated 41 of 50. One check of the verifier did not register a fix that reads correct; it counts as not fixed, which costs 8 points. The session took 93 minutes, without the 16 minutes the operator took between the two stages, for US$ 10.75. A second run was voided: in it the agent launched four reviewer subagents on Claude Opus 5.5 of its own accord, and they wrote 122 of its 543 responses, so it does not count as a run of Claude Sonnet 5.5. One run of three: it is not official and could move.
A written reading of the aggregate; it is not part of the score, and it names no flaw.
- Full marks
- Security, Compatibility
Run by run
- Run 1 · 9321h 32minUS$ 10.75Tokens: 1,390 in · 550,255 out · 15,124,592 cache · 15,676,237 total
-
2
Claude Sonnet 5.5 Anthropic · effort xhigh 2 of 3 runs (860 · 911)860 of 1000 LEB Gold
- Security250/250
- Architecture156/200
- Bugs139/150
- Performance140/150
- Clean code57/100
- Compatibility75/100
- Explanation43/50
Comment and details
Two runs 51 points apart (860 and 911). The second is the highest single run of any agent so far: security 232 of 250, architecture 183 of 200, bugs 147 of 150, performance 144 of 150 and clean code 89 of 100, against 250, 156, 139, 140 and 57 in the first. In both runs one change went beyond what the manifest contracts and cost 25 compatibility points (75). The second run fixed 34 of the 36 planted flaws and reported 33, with no false positive, and its report was rated 41 of 50. The sessions took 42 and 43 minutes for US$ 6.58 and US$ 6.12. The same model at high effort has a median of 855 over three runs. Two runs of three: not official, and the published score is the lower of the two.
A written reading of the aggregate; it is not part of the score, and it names no flaw.
- Full marks
- Security
Run by run
- Run 1 · 86042minUS$ 6.58Tokens: 1,030 in · 244,526 out · 12,893,196 cache · 13,138,752 total
- Run 2 · 91143minUS$ 6.12Tokens: 3,602 in · 262,629 out · 9,421,853 cache · 9,688,084 total
-
3
Claude Sonnet 5.5 Anthropic · effort high 2 of 3 runs (897 · 855)855 of 1000 LEB Gold
- Security243/250
- Architecture153/200
- Bugs145/150
- Performance115/150
- Clean code79/100
- Compatibility75/100
- Explanation45/50
Comment and details
Two runs, 897 and 855, so the published score is the lower one, 855; with two of three runs it is not official and could move. A run filed in between, as run 2, was withdrawn because its session log shows it ran at medium effort and not at high. Security was 250 of 250 in the 897 run and 243 in the 855 run, bugs 147 and 145, architecture 178 and 153, performance 128 and 115, clean code 89 and 79. The 855 run, the published one, fixed 33 of the 36 planted flaws and reported 27, with no false positive, and its report was rated 45 of 50; the 897 run fixed 34, reported 32 and was rated 45 too. Both lost 25 compatibility points (75): the 897 run to a login lockout that answers 429, the 855 run to a change beyond what the manifest contracts; only the 897 run also paid 15 points for a new bug. The 855 run took 42 minutes and US$ 5.30, the 897 run 43 minutes and US$ 5.18. The same model at xhigh effort scored 860 and 911 in two runs.
A written reading of the aggregate; it is not part of the score, and it names no flaw.
Run by run
- Run 1 · 89743minUS$ 5.18Tokens: 1,918 in · 165,030 out · 11,795,003 cache · 11,961,951 total
- Run 3 · 85542minUS$ 5.30Tokens: 3,519 in · 174,950 out · 11,635,966 cache · 11,814,435 total
-
4
Claude Opus 5.5 Anthropic · effort high 2 of 3 runs (848 · 866)848 of 1000 LEB Gold
- Security232/250
- Architecture153/200
- Bugs137/150
- Performance134/150
- Clean code71/100
- Compatibility75/100
- Explanation46/50
Comment and details
Two runs, 18 points apart (848 and 866), with the same shape: 232 and 229 of 250 in security, 153 of 200 in architecture in both, 137 and 147 of 150 in bugs, 134 and 124 of 150 in performance and 71 of 100 in clean code in both, with no penalty. The first lost 25 compatibility points for one change that contradicts the manifest: a repeated idempotency key from a different user got a 422 where the manifest says the original response comes back. The second kept the contract whole (compatibility 100). The first cost US$ 8.77 in 56 minutes, the second US$ 7.57 in 50, without the operator's wait between the two stages. The same model at xhigh effort scored 886 and 822. Two runs of three: not official, and the published score is the lower of the two.
A written reading of the aggregate; it is not part of the score, and it names no flaw.
Run by run
- Run 1 · 84856minUS$ 8.77Tokens: 1,450 in · 148,633 out · 16,974,538 cache · 17,124,621 total
- Run 2 · 86650minUS$ 7.57Tokens: 1,832 in · 135,000 out · 13,004,371 cache · 13,141,203 total
-
5
Claude Opus 5.5 Anthropic · effort xhigh 3 of 3 runs (886 · 822 · 845)845 of 1000 LEB Gold
- Security232/250
- Architecture153/200
- Bugs139/150
- Performance131/150
- Clean code71/100
- Compatibility75/100
- Explanation44/50
Comment and details
Three runs (886, 822 and 845), so the published score is their median, 845. The shape is the same in all three: 229 to 246 of 250 in security, 144 to 164 of 200 in architecture, 139 to 147 of 150 in bugs, 131 to 144 of 150 in performance and 64 to 71 of 100 in clean code. The first two took the same 30-point compatibility deduction (COMP-003): the idempotency fingerprint now includes the user, so a repeated key from another user of the same company no longer returns the original response, which the manifest requires. The third did not repeat that, but lost 25 points (COMP-008) for a login lockout that answers 429, a status the manifest does not list, even to the right password. It fixed 32 of the 36 planted flaws and reported 28, with no false positive, and its report was rated 44 of 50. The sessions took 82, 48 and 74 minutes for US$ 13.03, US$ 6.77 and US$ 13.69.
A written reading of the aggregate; it is not part of the score, and it names no flaw.
Run by run
- Run 1 · 8861h 22minUS$ 13.03Tokens: 1,104 in · 247,375 out · 23,558,840 cache · 23,807,319 total
- Run 2 · 82248minUS$ 6.77Tokens: 1,294 in · 143,108 out · 8,673,335 cache · 8,817,737 total
- Run 3 · 8451h 14minUS$ 13.69Tokens: 1,384 in · 340,120 out · 14,941,510 cache · 15,283,014 total
-
6
Claude Fable 5.1 Anthropic · effort high 1 of 3 runs836 of 1000 LEB Gold
- Security215/250
- Architecture156/200
- Bugs147/150
- Performance128/150
- Clean code71/100
- Compatibility75/100
- Explanation44/50
Comment and details
On a single run: security 215 of 250, architecture 156 of 200, bugs 147 of 150, performance 128 of 150 and 71 of 100 in clean code, with no penalty. It fixed 33 of the 36 planted flaws and its report was rated 44 of 50. It lost 25 compatibility points for one security change that the manifest does not allow: after 10 failed logins for an e-mail, the login answers 429 even to the right password, a status the manifest does not list, and anyone who knows an e-mail can lock that user out. The run cost US$ 15.81 and took 52 minutes of the agent's own time, without the 18 minutes the operator took between the two stages. One run of three: it is not official and could move.
A written reading of the aggregate; it is not part of the score, and it names no flaw.
Run by run
- Run 1 · 83652minUS$ 15.81Tokens: 2,426 in · 152,548 out · 8,470,393 cache · 8,625,367 total
-
7
GPT-6.1-sol OpenAI · effort xhigh 2 of 3 runs (800 · 830)800 of 1000 LEB Gold
- Security232/250
- Architecture142/200
- Bugs147/150
- Performance105/150
- Clean code29/100
- Compatibility100/100
- Explanation45/50
Comment and details
Two runs (800 and 830), so the published score is the lower, 800. Security 232, architecture 142, bugs 147 and performance 105 were the same in both; the second run took clean code from 29 to 64 of 100 and its report from 45 to 40 of 50, weakest on prioritisation. It fixed 33 of the 36 planted flaws and reported 22, with no false positive (the first run fixed 31 and had one), and kept the contract untouched (compatibility 100) with no penalty. The sessions took 50 and 49 minutes, without the 4 and 3 minutes the operator took between the two stages, and used about 7.4 and 5.1 million tokens; Codex keeps no cost. The second run was judged under the written rule that one finding is credited to one row only, which left five flaws named inside two bundled findings as fixes without a report. Two runs of three: not official, and the published score is the lower of the two.
A written reading of the aggregate; it is not part of the score, and it names no flaw.
- Full marks
- Compatibility
Run by run
- Run 1 · 80050minTokens: 310,106 in · 100,213 out · 6,995,200 cache · 7,405,519 total
- Run 2 · 83049minTokens: 217,725 in · 77,782 out · 4,814,592 cache · 5,110,099 total
-
8
GPT-6.1-sol OpenAI · effort ultra 1 of 3 runs797 of 1000 LEB Gold
- Security232/250
- Architecture111/200
- Bugs145/150
- Performance121/150
- Clean code46/100
- Compatibility100/100
- Explanation42/50
Comment and details
On a single run: security 232 of 250, bugs 145 of 150, performance 121 of 150, architecture 111 of 200 and 46 of 100 in clean code, with the contract untouched (compatibility 100) and no penalty. It fixed 29 of the 36 planted flaws and reported 25, with no false positive; the report was rated 42 of 50, weakest on prioritisation, because effort and risk are not weighed. The agent spread the work over three subagent threads of its own. The session took 55 minutes, without the 3 minutes the operator took between the two stages, and the four threads together used about 28.4 million tokens (10.4 million in the main one); Codex keeps no cost. One run of three: it is not official and could move.
A written reading of the aggregate; it is not part of the score, and it names no flaw.
- Full marks
- Compatibility
Run by run
- Run 1 · 79755minTokens: 712,563 in · 166,543 out · 27,510,912 cache · 28,390,018 total
-
9
GPT-6.1-sol OpenAI · effort high 1 of 3 runs779 of 1000 LEB Gold
- Security232/250
- Architecture142/200
- Bugs147/150
- Performance70/150
- Clean code46/100
- Compatibility100/100
- Explanation42/50
Comment and details
On a single run: security 232 of 250, bugs 147 of 150, performance 70 of 150, architecture 142 of 200 and 46 of 100 in clean code, with the contract untouched (compatibility 100) and no penalty. It fixed 30 of the 36 planted flaws and reported 21 of them, with no false positive; ten of the fixes were never reported as findings of their own (several sat bundled inside another finding), so they count only for the code. The report was rated 42 of 50, weakest on prioritisation, because effort is not weighed. The session took 33 minutes, without the 127 minutes the operator took between the two stages, and used about 7.1 million tokens; Codex keeps no cost. The same model at xhigh effort scored 800 and at ultra 797. One run of three: it is not official and could move.
A written reading of the aggregate; it is not part of the score, and it names no flaw.
- Full marks
- Compatibility
Run by run
- Run 1 · 77933minTokens: 354,573 in · 63,518 out · 6,653,440 cache · 7,071,531 total
-
10
Claude Haiku 5.5 Anthropic · effort xhigh 3 of 3 runs (708 · 732 · 824)732 of 1000 LEB Silver
- Security250/250
- Architecture100/200
- Bugs121/150
- Performance102/150
- Clean code43/100
- Compatibility75/100
- Explanation41/50
Comment and details
Three runs (708, 732 and 824), so the published score is their median, 732. The third is the best, and the one that differs most: architecture 172 of 200 against 92 and 100, bugs 137 of 150 against 121 and 121, performance 115 of 150 against 96 and 102; security (243 of 250) and clean code (43 of 100) are as in the first run. In every run one security change altered a behaviour the manifest contracts (compatibility 70, 75 and 75). The third run fixed 30 of the 36 planted flaws and reported 30, against 25 fixed in both earlier ones, with no false positive. It used one subagent and took 99 minutes for US$ 5.16, where the first two took 36 and 39 minutes for US$ 1.72 and US$ 1.61. Claude Haiku 4.5 scored 242 on the same task.
A written reading of the aggregate; it is not part of the score, and it names no flaw.
- Full marks
- Security
Run by run
- Run 1 · 70836minUS$ 1.72Tokens: 1,438 in · 252,651 out · 16,402,385 cache · 16,656,474 total
- Run 2 · 73239minUS$ 1.61Tokens: 1,378 in · 326,646 out · 9,253,810 cache · 9,581,834 total
- Run 3 · 8241h 40minUS$ 5.16Tokens: 69,441 in · 578,399 out · 62,347,246 cache · 62,995,086 total
-
11
Claude Sonnet 5.5 Anthropic · effort medium 3 of 3 runs (659 · 717 · 734)717 of 1000 LEB Silver
- Security236/250
- Architecture100/200
- Bugs121/150
- Performance99/150
- Clean code50/100
- Compatibility70/100
- Explanation41/50
Comment and details
Three runs (659, 717 and 734), so the published score is their median, 717. Security stayed between 222 and 236 of 250 and bugs between 121 and 137 of 150; performance rose from 77 to 99 to 115 of 150, architecture was 61, 100 and 94 of 200 and clean code 21, 50 and 21 of 100. The runs fixed 23, 28 and 27 of the 36 planted flaws and reported 21, 22 and 25, none with a false positive, and the reports were rated 41, 41 and 42 of 50. In the second run, the median one, a change went beyond what the manifest contracts (the idempotency key now includes the user, so another user of the same company repeating the request gets a 422 instead of the original answer) and cost 30 compatibility points (70, against 100 in the other two). The sessions took 14, 29 and 21 minutes, without the wait between the two stages, for US$ 1.98, US$ 2.96 and US$ 2.25. The same model at high effort has a median of 855 and at xhigh effort a published score of 860 over two runs.
A written reading of the aggregate; it is not part of the score, and it names no flaw.
Run by run
- Run 1 · 65914minUS$ 1.98Tokens: 1,364 in · 34,340 out · 3,425,232 cache · 3,460,936 total
- Run 2 · 71729minUS$ 2.96Tokens: 1,794 in · 59,634 out · 5,841,812 cache · 5,903,240 total
- Run 3 · 73421minUS$ 2.25Tokens: 1,272 in · 61,955 out · 4,759,376 cache · 4,822,603 total
-
12
GPT-6-astra OpenAI · effort xhigh 2 of 3 runs (717 · 729)717 of 1000 LEB Silver
- Security194/250
- Architecture142/200
- Bugs124/150
- Performance86/150
- Clean code50/100
- Compatibility100/100
- Explanation41/50
Comment and details
Two runs, 717 and 729, with the same shape: 194 of 250 in security and 124 of 150 in bugs in both, architecture 142 and 136 of 200, performance 86 and 89 of 150, clean code 50 and 64 of 100, with the contract untouched (compatibility 100). Both took the 20-point penalty for the same characterization test, which broke after the refactor: the customer's ticket history no longer came most recent first. It fixed 32 and 33 of the 36 planted flaws and reported 22 and 23 of them, with no false positive; the reports were rated 41 and 42 of 50, weakest on prioritisation. The sessions took 28 and 43 minutes, without the operator's wait between the two stages, and used about 7.3 and 7.7 million tokens; Codex keeps no cost. Two runs of three: not official, and the published score is the lower of the two.
A written reading of the aggregate; it is not part of the score, and it names no flaw.
- Full marks
- Compatibility
Run by run
- Run 1 · 71728minTokens: 213,017 in · 62,221 out · 6,991,616 cache · 7,266,854 total
- Run 2 · 72942minTokens: 284,263 in · 74,645 out · 7,340,800 cache · 7,699,708 total
-
13
Kimi K3 Moonshot AI · effort high 2 of 3 runs (737 · 712)712 of 1000 LEB Silver
- Security229/250
- Architecture92/200
- Bugs137/150
- Performance96/150
- Clean code18/100
- Compatibility100/100
- Explanation40/50
Comment and details
Two runs, 25 points apart (737 and 712), on two different hosts: the first through Novita AI, the second through Moonshot's own API, as in LEB-100. Same shape in both: security 232 and 229 of 250, architecture 92 of 200, performance 96 of 150 and the contract untouched (compatibility 100). The difference is clean code, 43 against 18 of 100. In the first run one finding described a conflict the code cannot produce. The first session took 69 minutes without the 10 the operator took between the stages and cost US$ 4.86; the second took 71 and cost US$ 4.67. Two runs of three: not official, and the published score is the lower of the two.
A written reading of the aggregate; it is not part of the score, and it names no flaw.
- Full marks
- Compatibility
Run by run
- Run 1 · 7371h 9minUS$ 4.86Tokens: 331,676 in · 56,450 out · 7,277,568 cache · 7,721,456 total
- Run 2 · 7121h 11minUS$ 4.67Tokens: 206,664 in · 57,396 out · 8,800,256 cache · 9,100,849 total
-
14
GLM-5.3-Flash Z.AI · effort max 1 of 3 runs693 of 1000 LEB Silver
- Security211/250
- Architecture106/200
- Bugs147/150
- Performance89/150
- Clean code0/100
- Compatibility100/100
- Explanation40/50
Comment and details
On a single run: security 211 of 250, bugs 147 of 150, architecture 106 of 200, performance 89 of 150 and nothing in clean code, with the contract untouched (compatibility 100) and no penalty. It fixed 25 of the 36 planted flaws and reported 23, with no false positive; the report was rated 40 of 50. The same model at high effort scored 568 on a run through Z.ai's own API; this one ran at max effort through Novita AI, so the gap of 125 points mixes the effort and the host. The session took 53 minutes, without the 13 minutes the operator took between the two stages, for US$ 0.54 and about 13.4 million tokens. One run of three: it is not official and could move.
A written reading of the aggregate; it is not part of the score, and it names no flaw.
- Full marks
- Compatibility
- Zero
- Clean code
Run by run
- Run 1 · 69353minUS$ 0.54Tokens: 622,450 in · 63,956 out · 12,599,424 cache · 13,354,809 total
-
15
GLM-5.3 Z.AI · effort high 1 of 3 runs688 of 1000 LEB Silver
- Security222/250
- Architecture69/200
- Bugs147/150
- Performance96/150
- Clean code14/100
- Compatibility100/100
- Explanation40/50
Comment and details
On a single run, level with the best in bugs (147 of 150): security 222 of 250, performance 96 of 150, architecture 69 of 200 and 14 of 100 in clean code, with the contract untouched. Several of its security and architecture findings were filed as plain bugs or as quality issues, and each costs half of its credit. It ran through OpenRouter, in a 75-minute session for US$ 1.62. One run of three: it is not official and could move.
A written reading of the aggregate; it is not part of the score, and it names no flaw.
- Full marks
- Compatibility
Run by run
- Run 1 · 6881h 15minUS$ 1.62Tokens: 127,024 in · 66,405 out · 17,751,296 cache · 17,994,214 total
-
16
DeepSeek V4.1 Flash DeepSeek · effort high 1 of 3 runs649 of 1000 LEB Silver
- Security229/250
- Architecture47/200
- Bugs137/150
- Performance96/150
- Clean code0/100
- Compatibility100/100
- Explanation40/50
Comment and details
On a single run: security at 229 of 250 and bugs at 137 of 150, with the contract untouched (compatibility 100). It left architecture largely alone (47 of 200) and earned nothing in clean code, which is where its points ran out. One of its findings described a problem the code cannot have. Both stages ran in one OpenCode session of 36 minutes for US$ 0.12, by far the cheapest line here. One run of three: it is not official and could move.
A written reading of the aggregate; it is not part of the score, and it names no flaw.
- Full marks
- Compatibility
- Zero
- Clean code
Run by run
- Run 1 · 64936minUS$ 0.12Tokens: 123,372 in · 47,279 out · 14,006,016 cache · 14,226,586 total
-
17
GLM-5.3-FlashX Z.AI · effort high 1 of 3 runs648 of 1000 LEB Silver
- Security187/250
- Architecture92/200
- Bugs145/150
- Performance77/150
- Clean code32/100
- Compatibility75/100
- Explanation40/50
Comment and details
On a single run: bugs at 145 of 150, security 187 of 250, architecture 92 of 200, performance 77 of 150 and clean code 32 of 100. It added a login lockout (10 failed attempts for an e-mail in 15 minutes, from anyone) that answers 401 even to the right password, which the judge charged as a change of contracted behaviour (compatibility 75 of 100). It ran through Z.ai's own API. The session took 47 minutes and cost US$ 0.96. One run of three: it is not official and could move.
A written reading of the aggregate; it is not part of the score, and it names no flaw.
Run by run
- Run 1 · 64847minUS$ 0.96Tokens: 198,120 in · 54,593 out · 10,246,592 cache · 10,540,100 total
-
18
GPT-5.6-terra OpenAI · effort xhigh 3 of 3 runs (625 · 574 · 657)625 of 1000 LEB Silver
- Security225/250
- Architecture47/200
- Bugs134/150
- Performance96/150
- Clean code0/100
- Compatibility100/100
- Explanation38/50
Comment and details
Three runs from 574 to 657, and the official 625 is the middle one. Security was 225, 225 and 246 of 250, architecture 47 of 200 in all three, clean code 0, 0 and 18 of 100, bugs 134, 137 and 142 of 150 and performance 96, 41 and 64 of 150, with the contract untouched in every run (compatibility 100). The first two took the 15-point penalty for a fix that introduced a new bug; the third took none. The sessions took 32, 28 and 32 minutes, without the operator's wait between the two stages. Codex keeps no cost.
A written reading of the aggregate; it is not part of the score, and it names no flaw.
- Full marks
- Compatibility
- Zero
- Clean code
Run by run
- Run 1 · 62532minTokens: 229,711 in · 55,406 out · 8,839,936 cache · 9,125,053 total
- Run 2 · 57428minTokens: 238,473 in · 55,402 out · 7,062,528 cache · 7,356,403 total
- Run 3 · 65732minTokens: 214,606 in · 52,932 out · 7,931,904 cache · 8,199,442 total
-
19
GLM-5.3-Flash Z.AI · effort high 1 of 3 runs568 of 1000 LEB Bronze
- Security204/250
- Architecture47/200
- Bugs126/150
- Performance51/150
- Clean code0/100
- Compatibility100/100
- Explanation40/50
Comment and details
On a single run: security 204 of 250, bugs 126 of 150, performance 51 of 150, architecture 47 of 200 and nothing in clean code, with the contract untouched (compatibility 100) and no penalty. It fixed 17 of the 36 planted flaws and reported 20, with no false positive; the report was rated 40 of 50, weakest on prioritisation, because the order of the findings contradicts its own severity labels. It ran through Z.ai's own API; the session took 46 minutes, without the nearly two hours the operator took between the two stages, for US$ 0.31 and about 7.7 million tokens. One run of three: it is not official and could move.
A written reading of the aggregate; it is not part of the score, and it names no flaw.
- Full marks
- Compatibility
- Zero
- Clean code
Run by run
- Run 1 · 56846minUS$ 0.31Tokens: 333,659 in · 52,805 out · 7,231,040 cache · 7,652,156 total
-
20
GPT-5.6-luna OpenAI · effort xhigh 2 of 3 runs (638 · 512)512 of 1000 LEB Bronze
- Security173/250
- Architecture67/200
- Bugs121/150
- Performance57/150
- Clean code0/100
- Compatibility75/100
- Explanation39/50
Comment and details
Two runs, 126 points apart (638 and 512), so this agent is not steady yet. The first had no penalty and kept the contract whole (compatibility 100): security 211 of 250, bugs 147 of 150, performance 77 of 150, architecture 47 of 200 and 18 of 100 in clean code. The second fell in every category except architecture (67): security 173, bugs 121, performance 57 and nothing in clean code. It lost 25 compatibility points and took a 20-point penalty for one edit to the company routes: the platform role lost the write access the route table gives it, and the company administrator gained the right to switch a company on and off, which broke a public characterization test. It fixed 22 and 23 of the 36 planted flaws and reported 20 and 18 of them, with one false positive in the second; the reports were rated 38 and 39 of 50, weakest on prioritisation. The sessions took 42 and 38 minutes, without the operator's wait between the two stages; Codex keeps no cost. Two runs of three: not official, and the published score is the lower of the two.
A written reading of the aggregate; it is not part of the score, and it names no flaw.
- Zero
- Clean code
Run by run
- Run 1 · 63842minTokens: 449,393 in · 80,165 out · 19,209,472 cache · 19,739,030 total
- Run 2 · 51238minTokens: 392,754 in · 73,502 out · 13,548,032 cache · 14,014,288 total
-
21
GPT-6-luna OpenAI · effort xhigh 1 of 3 runs471 of 1000 LEB Bronze
- Security180/250
- Architecture47/200
- Bugs124/150
- Performance70/150
- Clean code0/100
- Compatibility45/100
- Explanation40/50
Comment and details
On a single run: security 180 of 250, bugs 124 of 150, performance 70 of 150, architecture 47 of 200 and nothing in clean code. Compatibility fell to 45 of 100, with two violations kept as judged (the company update route now refuses the platform role, and the technician productivity report lists deactivated technicians), and two penalties took 35 points: one characterization test broke (an admin could no longer change a company's active flag) and a login limiter it added keyed its block on the connection address. It fixed 22 of the 36 planted flaws and reported 21, with two false positives, neither a decoy; the report was rated 40 of 50, weakest on prioritisation. The session took 35 minutes, without the 1 minute the operator took between the two stages, and used about 9.4 million tokens; Codex keeps no cost. One run of three: it is not official and could move.
A written reading of the aggregate; it is not part of the score, and it names no flaw.
- Zero
- Clean code
Run by run
- Run 1 · 47134minTokens: 325,721 in · 99,556 out · 8,990,976 cache · 9,416,253 total
-
22
GLM-5.2 Z.AI · effort high 1 of 3 runs390 of 1000 Failed
- Security151/250
- Architecture39/200
- Bugs63/150
- Performance51/150
- Clean code0/100
- Compatibility45/100
- Explanation41/50
Comment and details
Explanation scored 41 of 50, but the code lost ground: security 151 of 250, bugs 63 of 150, performance 51 of 150 and nothing in clean code. Two of its changes broke contracts of the manifest, which cost 55 of the 100 compatibility points, the lowest on the board. It ran through Novita AI, in a 21-minute session for US$ 3.14. One run of three: it is not official and could move.
A written reading of the aggregate; it is not part of the score, and it names no flaw.
- Zero
- Clean code
Run by run
- Run 1 · 39021minUS$ 3.14Tokens: 625,652 in · 33,738 out · 7,828,480 cache · 8,505,204 total
-
23
MiniMax-M3 MiniMax · effort thinking 2 of 3 runs (388 · 398)388 of 1000 Failed
- Security113/250
- Architecture56/200
- Bugs47/150
- Performance26/150
- Clean code7/100
- Compatibility100/100
- Explanation39/50
Comment and details
Two runs within ten points of each other (388 and 398), so the shape is steady: compatibility whole, security at 113 of 250, bugs at 47 of 150 and little in performance or clean code. In both, the first-stage report filed several security and architecture problems as plain bugs and never moved them, which cost points in the categories that mattered. Its runs took 42 and 53 minutes, at US$ 1.21 and 0.81. Two runs of three: not official, and the published score is the lower of the two.
A written reading of the aggregate; it is not part of the score, and it names no flaw.
- Full marks
- Compatibility
Run by run
- Run 1 · 38842minUS$ 1.21Tokens: 128,455 in · 48,678 out · 15,344,274 cache · 15,575,334 total
- Run 2 · 39853minUS$ 0.81Tokens: 282,942 in · 41,745 out · 10,646,261 cache · 11,004,186 total
-
24
MiniMax-M2.7-highspeed MiniMax · default effort (not configurable) 1 of 3 runs291 of 1000 Failed
- Security77/250
- Architecture22/200
- Bugs11/150
- Performance51/150
- Clean code0/100
- Compatibility100/100
- Explanation30/50
Comment and details
On a single run: security 77 of 250, bugs 11 of 150, performance 51 of 150, architecture 22 of 200 and nothing in clean code, with the contract untouched (compatibility 100) and no penalty. It made a few small, targeted edits and fixed 7 of the 36 planted flaws; it reported 8 of them (one of those left unfixed) and one false positive, a counting bug the code does not have. The report was rated 30 of 50, weakest on prioritisation and trade-offs: its two critical findings come 7th and 8th in the list, and its table of decisions rarely compares alternatives. MiniMax-M2.7-highspeed is the M2.7 series served faster (MiniMax: "same performance, faster"), and OpenCode offers no effort setting for it. The agent wrote RELATORIO.md and achados.json inside code/ instead of the project root, and the delivery was accepted as made. Both stages ran in one OpenCode session of 10 minutes, without the 7 minutes the operator took between the two stages, through MiniMax's own API, for US$ 0.35. One run of three: it is not official and could move.
A written reading of the aggregate; it is not part of the score, and it names no flaw.
- Full marks
- Compatibility
- Zero
- Clean code
Run by run
- Run 1 · 29110minUS$ 0.35Tokens: 59,631 in · 15,697 out · 3,198,034 cache · 3,280,847 total
-
25
Claude Haiku 4.5 Anthropic · default effort (not configurable) 3 of 3 runs (242 · 218 · 272)242 of 1000 Failed
- Security56/250
- Architecture25/200
- Bugs29/150
- Performance0/150
- Clean code0/100
- Compatibility100/100
- Explanation32/50
Comment and details
Three runs from 218 to 272, and the published 242 is the middle one. Two of the three changed behaviour that the manifest contracts and lost 30 compatibility points; the published run did not. Its security score was 56 of 250 in the published run and above 100 in the other two, while performance and clean code stayed near zero throughout. The published run was the fastest line here, 12 minutes; the other two took 28 and 44, and the second cost US$ 2.22. All three ran in Claude Code.
A written reading of the aggregate; it is not part of the score, and it names no flaw.
- Full marks
- Compatibility
- Zero
- Performance, Clean code
Run by run
- Run 1 · 24212minUS$ 0.96Tokens: 1,476 in · 32,117 out · 6,064,686 cache · 6,098,279 total
- Run 2 · 21844minUS$ 2.22Tokens: 16,438 in · 88,702 out · 13,017,862 cache · 13,123,002 total
- Run 3 · 27228minUS$ 1.36Tokens: 2,073 in · 43,806 out · 9,106,328 cache · 9,152,207 total
-
26
Gemma 4 26B A4B Google DeepMind · effort high 1 of 3 runs232 of 1000 Failed
- Security63/250
- Architecture14/200
- Bugs8/150
- Performance16/150
- Clean code0/100
- Compatibility100/100
- Explanation31/50
Comment and details
On a single run: the contract stayed whole (compatibility 100), but little else moved: security 63 of 250, architecture 14 of 200, bugs 8 of 150, performance 16 of 150 and nothing in clean code. It fixed 3 of the 36 planted flaws and reported 5, all of them real and none a false positive, each with full confidence; the report was rated 31 of 50, weakest on prioritisation and on trade-offs. The run cost US$ 0.36 and took 14 minutes of the agent's own time, without the 7 minutes the operator took between the two stages. One run of three: it is not official and could move.
A written reading of the aggregate; it is not part of the score, and it names no flaw.
- Full marks
- Compatibility
- Zero
- Clean code
Run by run
- Run 1 · 23214minUS$ 0.36Tokens: 2,731,666 in · 20,301 out · 0 cache · 2,752,166 total
-
27
MiniMax-M2.7 MiniMax · default effort (not configurable) 1 of 3 runs202 of 1000 Failed
- Security85/250
- Architecture0/200
- Bugs8/150
- Performance0/150
- Clean code0/100
- Compatibility75/100
- Explanation34/50
Comment and details
On a single run: security 85 of 250, bugs 8 of 150, and nothing in architecture, performance or clean code, with compatibility at 75 of 100 for one violation kept as judged: the login rate limiter it added answers 401 even to the right password after five attempts in 15 minutes from the same non-local address (COMP-008). It fixed 3 of the 36 planted flaws and reported 5 (two of those left unfixed), with one false positive, a counting bug the code does not have, and one extra finding. The report was rated 34 of 50, weakest on precision, because it contradicts itself in places (a token valid "until expiry" and then "indefinitely") and overstates how long a brute-force attack would take. MiniMax-M2.7 is the standard-speed model of the series, and OpenCode offers no effort setting for it. Both stages ran in one OpenCode session of 15 minutes, without the 7 minutes the operator took between the two stages, through MiniMax's own API, for US$ 0.44. One run of three: it is not official and could move.
A written reading of the aggregate; it is not part of the score, and it names no flaw.
- Zero
- Architecture, Performance, Clean code
Run by run
- Run 1 · 20215minUS$ 0.44Tokens: 76,199 in · 15,764 out · 5,219,356 cache · 5,319,203 total
What the record does not have
- No checkpoint was taken between the two stages of the task in any of the runs, so it is not on record that the model stayed the same through the first stage. The transcripts show a single model.
- The task text the agent read names the previous scoring-matrix hash. The matrix this run is scored against differs from it only by a header field that marks the instance as active. The scoring is the same.
- The agents did not all run in the same client, Claude Code or OpenCode, each with its own tools. The client of every run is in its record.
What is published
This page shows one line per agent: the total, the grade, the score in each category, how many runs it rests on, and the cost and the time. It never shows a run's delivery, its verdict or its flaw-by-flaw results, which stay in a private archive. The instance and its answer key are public on GitHub; the machines that run the agents cannot reach GitHub.
The instance is in an exploratory pilot. The results above are the ones published so far, and other agents follow. Its first runs also served to check the procedure.
The protocol, the scoring and the tools are public in the ai-benchmark repository. How a run works and how the two levels differ is on the Benchmark page, and the results of the first level are on the LEB-100 page.