LEB Default

One run of a case of five scored 831 (Gold). It completed inside the 90-minute budget, at about US$ 7.47 and 37 minutes, and it did not break a behavior the public tests protect (compatibility 100). Bug fixing was the strongest area (195 of 213), then performance (186 of 213), architecture (214 of 283) and clean code (91 of 141). The report was rated 45 of 50. It is one system, so how the configuration does across systems is not measured yet. The blind judges agreed on almost every item, and a person settled the rest. The other four runs are still to be made, so the figure will move.

• 1 of 1 scheduled attempt completed with the requested model. Systems on which every attempt completed: 1 of 1.
• Expected score per scheduled attempt: 831.0 (831.0 to 831.0).
• Blocked: 0 (at the start 0, during the work 0). Budget exhausted: 0. Switched: 0. Lost to the lab or the operator, rerun: 0.

Answer key (matrix) SHA-256 9bd0c169e420927d8e0aadd87f30789b4cf6e60a2db49999c450c7f86f2207ce · package SHA-256 3370984d863313e524b7e976eca416d043489fa8f67858e70dabd5998f4dba4e.

Five runs between 752 and 836, mean 785 with a 95% interval of 764 to 812, all Gold. All five completed inside the 90-minute budget, at about US$ 5.78 and 35 minutes each, and none broke a behavior the public tests protect (compatibility 100 in every run). Bug fixing was the strongest area (202 of 213 on average), then performance (176 of 213), clean code (88 of 141) and architecture (176 of 283). The reports were rated 45 of 50. Penalties: run 3 lost 5 points for reporting something that is not a flaw. It is one system, so the interval only reflects the variation among the runs on it: how the configuration does across systems is not measured yet. The blind judges agreed on almost every item, and a person settled the rest.

• 5 of 5 scheduled attempts completed with the requested model. Systems on which every attempt completed: 1 of 1.
• Expected score per scheduled attempt: 785.2 (764.0 to 812.2).
• Blocked: 0 (at the start 0, during the work 0). Budget exhausted: 0. Switched: 0. Lost to the lab or the operator, rerun: 1.

Answer key (matrix) SHA-256 9bd0c169e420927d8e0aadd87f30789b4cf6e60a2db49999c450c7f86f2207ce · package SHA-256 3370984d863313e524b7e976eca416d043489fa8f67858e70dabd5998f4dba4e.

Five runs between 625 and 812, mean 732 with a 95% interval of 677 to 789, two Gold and three Silver. All five completed inside the 90-minute budget, at about US$ 3.97 and 22 minutes each, and none broke a behavior the public tests protect (compatibility 100 in every run). Bug fixing was the strongest area (173 of 213 on average), then performance (173 of 213), architecture (166 of 283) and clean code (79 of 141). The reports were rated 43 to 44 of 50. Penalties: run 2 lost 15 points for a new defect introduced by its own change. It is one system, so the interval only reflects the variation among the runs on it: how the configuration does across systems is not measured yet. The blind judges agreed on almost every item, and a person settled the rest.

• 5 of 5 scheduled attempts completed with the requested model. Systems on which every attempt completed: 1 of 1.
• Expected score per scheduled attempt: 731.6 (676.8 to 789.0).
• Blocked: 0 (at the start 0, during the work 0). Budget exhausted: 0. Switched: 0. Lost to the lab or the operator, rerun: 0.

Answer key (matrix) SHA-256 9bd0c169e420927d8e0aadd87f30789b4cf6e60a2db49999c450c7f86f2207ce · package SHA-256 3370984d863313e524b7e976eca416d043489fa8f67858e70dabd5998f4dba4e.

Five runs between 672 and 776, mean 726 with a 95% interval of 698 to 754, one Gold and four Silver. All five completed inside the 90-minute budget, about 4.6 million tokens and 25 minutes each (the log keeps no cost), and none broke a behavior the public tests protect (compatibility 100 in every run). Bug fixing was the strongest area (209 of 213 on average), then performance (183 of 213), architecture (150 of 283) and clean code (46 of 141). The reports were rated 42 to 43 of 50. Penalties: run 2 lost 15 points for a new defect introduced by its own change; runs 4 and 5 lost 5 points each for reporting something that is not a flaw. It is one system, so the interval only reflects the variation among the runs on it: how the configuration does across systems is not measured yet. The blind judges agreed on almost every item, and a person settled the rest. It ran with the client's approvals and sandbox bypassed, an exception to the series' rule against it (see the notes).

• 5 of 5 scheduled attempts completed with the requested model. Systems on which every attempt completed: 1 of 1.
• Expected score per scheduled attempt: 725.6 (698.0 to 753.8).
• Blocked: 0 (at the start 0, during the work 0). Budget exhausted: 0. Switched: 0. Lost to the lab or the operator, rerun: 0.

Answer key (matrix) SHA-256 9bd0c169e420927d8e0aadd87f30789b4cf6e60a2db49999c450c7f86f2207ce · package SHA-256 3370984d863313e524b7e976eca416d043489fa8f67858e70dabd5998f4dba4e.

Five runs between 694 and 734, mean 713 with a 95% interval of 700 to 728, all Silver. All five completed inside the 90-minute budget, at about US$ 2.29 and 20 minutes each, and none broke a behavior the public tests protect (compatibility 100 in every run). Bug fixing was the strongest area (199 of 213 on average), then performance (173 of 213), architecture (138 of 283) and clean code (66 of 141). The reports were rated 40 to 43 of 50. Penalties: runs 1, 2, 3 and 5 lost 5 points each for reporting something that is not a flaw. It is one system, so the interval only reflects the variation among the runs on it: how the configuration does across systems is not measured yet. The blind judges agreed on almost every item, and a person settled the rest.

• 5 of 5 scheduled attempts completed with the requested model. Systems on which every attempt completed: 1 of 1.
• Expected score per scheduled attempt: 713.4 (699.8 to 727.8).
• Blocked: 0 (at the start 0, during the work 0). Budget exhausted: 0. Switched: 0. Lost to the lab or the operator, rerun: 0.

Answer key (matrix) SHA-256 9bd0c169e420927d8e0aadd87f30789b4cf6e60a2db49999c450c7f86f2207ce · package SHA-256 3370984d863313e524b7e976eca416d043489fa8f67858e70dabd5998f4dba4e.

Three runs of a case of five, between 685 and 718, mean 699 (the interval, 685 to 718, is the range of the three), all Silver. All three completed inside the 90-minute budget, about 6.7 million tokens and 25 minutes each (the log keeps no cost), and none broke a behavior the public tests protect (compatibility 100 in every run). Bug fixing was the strongest area (209 of 213 on average), then performance (183 of 213), architecture (140 of 283) and clean code (35 of 141). The reports were rated 41 to 43 of 50. Penalties: run 1 lost 15 points for a new defect introduced by its own change; runs 1, 2 and 3 lost 5 points each for reporting something that is not a flaw. It is one system, so the interval only reflects the variation among the runs on it: how the configuration does across systems is not measured yet. The blind judges agreed on almost every item, and a person settled the rest. The other two are still to be made, so the figures will move. It ran with the client's approvals and sandbox bypassed, an exception to the series' rule against it (see the notes).

• 3 of 3 scheduled attempts completed with the requested model. Systems on which every attempt completed: 1 of 1.
• Expected score per scheduled attempt: 699.3 (685.0 to 718.0).
• Blocked: 0 (at the start 0, during the work 0). Budget exhausted: 0. Switched: 0. Lost to the lab or the operator, rerun: 0.

Answer key (matrix) SHA-256 9bd0c169e420927d8e0aadd87f30789b4cf6e60a2db49999c450c7f86f2207ce · package SHA-256 3370984d863313e524b7e976eca416d043489fa8f67858e70dabd5998f4dba4e.

One run of a case of five scored 690 (Silver). It completed inside the 90-minute budget, about 10.9 million tokens and 15 minutes (the log keeps no cost), and it did not break a behavior the public tests protect (compatibility 100). Bug fixing was the strongest area (209 of 213), then performance (199 of 213), architecture (146 of 283) and clean code (0 of 141). The report was rated 41 of 50. Penalties: run 1 lost 5 points for reporting something that is not a flaw. It is one system, so how the configuration does across systems is not measured yet. The blind judges agreed on almost every item, and a person settled the rest. The other four runs are still to be made, so the figure will move. It ran with the client's approvals and sandbox bypassed, an exception to the series' rule against it (see the notes). Ultra is the client's multi-agent effort: a main agent and subagents on the same model, and the tokens counted are those of all of them.

• 1 of 1 scheduled attempt completed with the requested model. Systems on which every attempt completed: 1 of 1.
• Expected score per scheduled attempt: 690.0 (690.0 to 690.0).
• Blocked: 0 (at the start 0, during the work 0). Budget exhausted: 0. Switched: 0. Lost to the lab or the operator, rerun: 0.

Answer key (matrix) SHA-256 9bd0c169e420927d8e0aadd87f30789b4cf6e60a2db49999c450c7f86f2207ce · package SHA-256 3370984d863313e524b7e976eca416d043489fa8f67858e70dabd5998f4dba4e.

Two runs of a case of five, between 622 and 648, mean 635 (the interval, 622 to 648, is the range of the two), all Silver. Both completed inside the 90-minute budget, about 17.8 million tokens and 16 minutes each (the log keeps no cost), and none broke a behavior the public tests protect (compatibility 100 in every run). Bug fixing was the strongest area (209 of 213 on average), then performance (199 of 213), architecture (89 of 283) and clean code (0 of 141). The reports were rated 42 to 44 of 50. Penalties: runs 1 and 2 lost 5 points each for reporting something that is not a flaw. It is one system, so the interval only reflects the variation among the runs on it: how the configuration does across systems is not measured yet. The blind judges agreed on almost every item, and a person settled the rest. The other three are still to be made, so the figures will move. It ran with the client's approvals and sandbox bypassed, an exception to the series' rule against it (see the notes). Ultra is the client's multi-agent effort: a main agent and subagents on the same model, and the tokens counted are those of all of them.

• 2 of 2 scheduled attempts completed with the requested model. Systems on which every attempt completed: 1 of 1.
• Expected score per scheduled attempt: 635.0 (622.0 to 648.0).
• Blocked: 0 (at the start 0, during the work 0). Budget exhausted: 0. Switched: 0. Lost to the lab or the operator, rerun: 0.

Answer key (matrix) SHA-256 9bd0c169e420927d8e0aadd87f30789b4cf6e60a2db49999c450c7f86f2207ce · package SHA-256 3370984d863313e524b7e976eca416d043489fa8f67858e70dabd5998f4dba4e.

Two runs of a case of five, between 579 and 627, mean 603 (the interval, 579 to 627, is the range of the two), one Silver and one Bronze. Both completed inside the 90-minute budget, at about US$ 7.71 (the client's own estimate) and 38 minutes each, and none broke a behavior the public tests protect (compatibility 100 in every run). Bug fixing was the strongest area (202 of 213 on average), then performance (140 of 213), architecture (121 of 283) and clean code (0 of 141). The reports were rated 40 to 44 of 50. Penalties: run 2 lost 5 points for reporting something that is not a flaw. It is one system, so the interval only reflects the variation among the runs on it: how the configuration does across systems is not measured yet. The blind judges agreed on almost every item, and a person settled the rest. The other three are still to be made, so the figures will move. It ran with the client's auto-approval flag, an exception to the series' rule against bypassing approvals (see the notes). It was reached through OpenRouter, whose only upstream for this model is xAI's own.

• 2 of 2 scheduled attempts completed with the requested model. Systems on which every attempt completed: 1 of 1.
• Expected score per scheduled attempt: 603.0 (579.0 to 627.0).
• Blocked: 0 (at the start 0, during the work 0). Budget exhausted: 0. Switched: 0. Lost to the lab or the operator, rerun: 0.

Answer key (matrix) SHA-256 9bd0c169e420927d8e0aadd87f30789b4cf6e60a2db49999c450c7f86f2207ce · package SHA-256 3370984d863313e524b7e976eca416d043489fa8f67858e70dabd5998f4dba4e.

Three runs of a case of five, between 519 and 606, mean 552 (the interval, 519 to 606, is the range of the three), one Silver and two Bronze. All three completed inside the 90-minute budget, about 5.7 million tokens and 17 minutes each (the log keeps no cost), and none broke a behavior the public tests protect (compatibility 100 in every run). Bug fixing was the strongest area (203 of 213 on average), then performance (149 of 213), architecture (57 of 283) and clean code (0 of 141). The reports were rated 42 to 44 of 50. It is one system, so the interval only reflects the variation among the runs on it: how the configuration does across systems is not measured yet. The blind judges agreed on almost every item, and a person settled the rest. The other two are still to be made, so the figures will move. It ran with the client's approvals and sandbox bypassed, an exception to the series' rule against it (see the notes).

• 3 of 3 scheduled attempts completed with the requested model. Systems on which every attempt completed: 1 of 1.
• Expected score per scheduled attempt: 552.0 (519.0 to 606.0).
• Blocked: 0 (at the start 0, during the work 0). Budget exhausted: 0. Switched: 0. Lost to the lab or the operator, rerun: 0.

Answer key (matrix) SHA-256 9bd0c169e420927d8e0aadd87f30789b4cf6e60a2db49999c450c7f86f2207ce · package SHA-256 3370984d863313e524b7e976eca416d043489fa8f67858e70dabd5998f4dba4e.

Five runs between 524 and 580, mean 547 with a 95% interval of 530 to 565, all Bronze. All five completed inside the 90-minute budget, at about US$ 1.13 and 27 minutes each, and none broke a behavior the public tests protect (compatibility 100 in every run). Bug fixing was the strongest area (173 of 213 on average), then performance (125 of 213) and architecture (96 of 283); clean code scored 10 of 141 on average, with two runs at 0. The reports were rated 41 to 43 of 50. It is one system, so the interval only reflects the variation among the runs on it: how the configuration does across systems is not measured yet. The blind judges agreed on almost every item, and a person settled the rest.

• 5 of 5 scheduled attempts completed with the requested model. Systems on which every attempt completed: 1 of 1.
• Expected score per scheduled attempt: 546.8 (530.0 to 565.0).
• Blocked: 0 (at the start 0, during the work 0). Budget exhausted: 0. Switched: 0. Lost to the lab or the operator, rerun: 1.

Answer key (matrix) SHA-256 9bd0c169e420927d8e0aadd87f30789b4cf6e60a2db49999c450c7f86f2207ce · package SHA-256 71dfeb7c3f82c75a96f3134e4e050b4476436116e14402d483e9c5f5510955e4.

One run of a case of five scored 541 (Bronze). It completed inside the 90-minute budget, at about US$ 2.64 (the client's own estimate) and 36 minutes, and it did not break a behavior the public tests protect (compatibility 100). Bug fixing was the strongest area (159 of 213), then performance (131 of 213), architecture (89 of 283) and clean code (20 of 141). The report was rated 42 of 50. It is one system, so how the configuration does across systems is not measured yet. The blind judges agreed on almost every item, and a person settled the rest. The other four runs are still to be made, so the figure will move. It ran with the client's auto-approval flag, an exception to the series' rule against bypassing approvals (see the notes).

• 1 of 1 scheduled attempt completed with the requested model. Systems on which every attempt completed: 1 of 1.
• Expected score per scheduled attempt: 541.0 (541.0 to 541.0).
• Blocked: 0 (at the start 0, during the work 0). Budget exhausted: 0. Switched: 0. Lost to the lab or the operator, rerun: 0.

Answer key (matrix) SHA-256 9bd0c169e420927d8e0aadd87f30789b4cf6e60a2db49999c450c7f86f2207ce · package SHA-256 3370984d863313e524b7e976eca416d043489fa8f67858e70dabd5998f4dba4e.

Two runs of a case of five, between 499 and 578, mean 538 (the interval, 499 to 578, is the range of the two), all Bronze. Both completed inside the 90-minute budget, at about US$ 3.01 (the client's own estimate) and 37 minutes each, and none broke a behavior the public tests protect (compatibility 100 in every run). Bug fixing was the strongest area (166 of 213 on average), then performance (108 of 213), architecture (111 of 283) and clean code (10 of 141). The reports were rated 42 to 44 of 50. It is one system, so the interval only reflects the variation among the runs on it: how the configuration does across systems is not measured yet. The blind judges agreed on almost every item, and a person settled the rest. The other three are still to be made, so the figures will move. It ran with the client's auto-approval flag, an exception to the series' rule against bypassing approvals (see the notes).

• 2 of 2 scheduled attempts completed with the requested model. Systems on which every attempt completed: 1 of 1.
• Expected score per scheduled attempt: 538.5 (499.0 to 578.0).
• Blocked: 0 (at the start 0, during the work 0). Budget exhausted: 0. Switched: 0. Lost to the lab or the operator, rerun: 2.

Answer key (matrix) SHA-256 9bd0c169e420927d8e0aadd87f30789b4cf6e60a2db49999c450c7f86f2207ce · package SHA-256 3370984d863313e524b7e976eca416d043489fa8f67858e70dabd5998f4dba4e.

Two runs of a case of five, between 499 and 526, mean 512 (the interval, 499 to 526, is the range of the two), all Bronze. Both completed inside the 90-minute budget, at about US$ 4.09 (the client's own estimate) and 52 minutes each, and none broke a behavior the public tests protect (compatibility 100 in every run). Bug fixing was the strongest area (166 of 213 on average), then performance (104 of 213), architecture (101 of 283) and clean code (0 of 141). The reports were rated 41 to 42 of 50. It is one system, so the interval only reflects the variation among the runs on it: how the configuration does across systems is not measured yet. The blind judges agreed on almost every item, and a person settled the rest. The other three are still to be made, so the figures will move. It ran with the client's auto-approval flag, an exception to the series' rule against bypassing approvals (see the notes).

• 2 of 2 scheduled attempts completed with the requested model. Systems on which every attempt completed: 1 of 1.
• Expected score per scheduled attempt: 512.5 (499.0 to 526.0).
• Blocked: 0 (at the start 0, during the work 0). Budget exhausted: 0. Switched: 0. Lost to the lab or the operator, rerun: 0.

Answer key (matrix) SHA-256 9bd0c169e420927d8e0aadd87f30789b4cf6e60a2db49999c450c7f86f2207ce · package SHA-256 3370984d863313e524b7e976eca416d043489fa8f67858e70dabd5998f4dba4e.

Two runs of a case of five, between 487 and 512, mean 500 (the interval, 487 to 512, is the range of the two), all Bronze. Both completed inside the 90-minute budget, at about US$ 0.08 (the client's own estimate) and 30 minutes each, and none broke a behavior the public tests protect (compatibility 100 in every run). Bug fixing was the strongest area (144 of 213 on average), then performance (134 of 213), architecture (85 of 283) and clean code (0 of 141). The reports were rated 43 to 44 of 50. Penalties: run 2 lost 15 points for a new defect introduced by its own change. It is one system, so the interval only reflects the variation among the runs on it: how the configuration does across systems is not measured yet. The blind judges agreed on almost every item, and a person settled the rest. The other three are still to be made, so the figures will move. It ran with the client's auto-approval flag, an exception to the series' rule against bypassing approvals (see the notes).

• 2 of 2 scheduled attempts completed with the requested model. Systems on which every attempt completed: 1 of 1.
• Expected score per scheduled attempt: 499.5 (487.0 to 512.0).
• Blocked: 0 (at the start 0, during the work 0). Budget exhausted: 0. Switched: 0. Lost to the lab or the operator, rerun: 0.

Answer key (matrix) SHA-256 9bd0c169e420927d8e0aadd87f30789b4cf6e60a2db49999c450c7f86f2207ce · package SHA-256 3370984d863313e524b7e976eca416d043489fa8f67858e70dabd5998f4dba4e.

One run of a case of five scored 498 (Bronze). It completed inside the 90-minute budget, at about US$ 0.30 (the client's own estimate) and 32 minutes, and it did not break a behavior the public tests protect (compatibility 100). Bug fixing was the strongest area (173 of 213), then performance (95 of 213), architecture (89 of 283) and clean code (0 of 141). The report was rated 41 of 50. It is one system, so how the configuration does across systems is not measured yet. The blind judges agreed on almost every item, and a person settled the rest. The other four runs are still to be made, so the figure will move. It ran with the client's auto-approval flag, an exception to the series' rule against bypassing approvals (see the notes).

• 1 of 1 scheduled attempt completed with the requested model. Systems on which every attempt completed: 1 of 1.
• Expected score per scheduled attempt: 498.0 (498.0 to 498.0).
• Blocked: 0 (at the start 0, during the work 0). Budget exhausted: 0. Switched: 0. Lost to the lab or the operator, rerun: 0.

Answer key (matrix) SHA-256 9bd0c169e420927d8e0aadd87f30789b4cf6e60a2db49999c450c7f86f2207ce · package SHA-256 3370984d863313e524b7e976eca416d043489fa8f67858e70dabd5998f4dba4e.

Two runs of a case of five, between 432 and 494, mean 463 (the interval, 432 to 494, is the range of the two), all Bronze. Both completed inside the 90-minute budget, at about US$ 0.55 (the client's own estimate) and 46 minutes each, and compatibility was between 70 and 100. Bug fixing was the strongest area (134 of 213 on average), then performance (126 of 213), architecture (77 of 283) and clean code (0 of 141). The reports were rated 40 to 42 of 50. It is one system, so the interval only reflects the variation among the runs on it: how the configuration does across systems is not measured yet. The blind judges agreed on almost every item, and a person settled the rest. The other three are still to be made, so the figures will move. It ran with the client's auto-approval flag, an exception to the series' rule against bypassing approvals (see the notes). It was served by Novita AI, the host the first series used for this model, in its 0731 version; the client names no xhigh for it, and DeepSeek treats xhigh as max.

• 2 of 2 scheduled attempts completed with the requested model. Systems on which every attempt completed: 1 of 1.
• Expected score per scheduled attempt: 463.0 (432.0 to 494.0).
• Blocked: 0 (at the start 0, during the work 0). Budget exhausted: 0. Switched: 0. Lost to the lab or the operator, rerun: 1.

Answer key (matrix) SHA-256 9bd0c169e420927d8e0aadd87f30789b4cf6e60a2db49999c450c7f86f2207ce · package SHA-256 3370984d863313e524b7e976eca416d043489fa8f67858e70dabd5998f4dba4e.

Two runs of a case of five, between 446 and 464, mean 455 (the interval, 446 to 464, is the range of the two), all Bronze. Both completed inside the 90-minute budget, at about US$ 2.06 (the client's own estimate) and 35 minutes each, and none broke a behavior the public tests protect (compatibility 100 in every run). Bug fixing was the strongest area (146 of 213 on average), then performance (122 of 213), architecture (50 of 283) and clean code (0 of 141). The reports were rated 36 to 37 of 50. It is one system, so the interval only reflects the variation among the runs on it: how the configuration does across systems is not measured yet. The blind judges agreed on almost every item, and a person settled the rest. The other three are still to be made, so the figures will move. It ran with the client's auto-approval flag, an exception to the series' rule against bypassing approvals (see the notes). It was reached through OpenRouter, whose only upstreams for this model are Google's own.

• 2 of 2 scheduled attempts completed with the requested model. Systems on which every attempt completed: 1 of 1.
• Expected score per scheduled attempt: 455.0 (446.0 to 464.0).
• Blocked: 0 (at the start 0, during the work 0). Budget exhausted: 0. Switched: 0. Lost to the lab or the operator, rerun: 0.

Answer key (matrix) SHA-256 9bd0c169e420927d8e0aadd87f30789b4cf6e60a2db49999c450c7f86f2207ce · package SHA-256 3370984d863313e524b7e976eca416d043489fa8f67858e70dabd5998f4dba4e.

Three runs of a case of five, between 408 and 499, mean 455 (the interval, 408 to 499, is the range of the three), all Bronze. All three completed inside the 90-minute budget, about 2.4 million tokens and 7 minutes each (the log keeps no cost), and none broke a behavior the public tests protect (compatibility 100 in every run). Bug fixing was the strongest area (187 of 213 on average), then performance (78 of 213), architecture (54 of 283) and clean code (0 of 141). The reports were rated 38 to 41 of 50. Penalties: runs 2 and 3 lost 5 points each for reporting something that is not a flaw. It is one system, so the interval only reflects the variation among the runs on it: how the configuration does across systems is not measured yet. The blind judges agreed on almost every item, and a person settled the rest. The other two are still to be made, so the figures will move. It ran with the client's approvals and sandbox bypassed, an exception to the series' rule against it (see the notes).

• 3 of 3 scheduled attempts completed with the requested model. Systems on which every attempt completed: 1 of 1.
• Expected score per scheduled attempt: 455.0 (408.0 to 499.0).
• Blocked: 0 (at the start 0, during the work 0). Budget exhausted: 0. Switched: 0. Lost to the lab or the operator, rerun: 0.

Answer key (matrix) SHA-256 9bd0c169e420927d8e0aadd87f30789b4cf6e60a2db49999c450c7f86f2207ce · package SHA-256 3370984d863313e524b7e976eca416d043489fa8f67858e70dabd5998f4dba4e.

Two runs of a case of five, between 422 and 483, mean 452 (the interval, 422 to 483, is the range of the two), all Bronze. Both completed inside the 90-minute budget, at about US$ 0.07 (the client's own estimate) and 25 minutes each, and none broke a behavior the public tests protect (compatibility 100 in every run). Bug fixing was the strongest area (148 of 213 on average), then performance (98 of 213), architecture (61 of 283) and clean code (5 of 141). The reports were rated 40 to 42 of 50. It is one system, so the interval only reflects the variation among the runs on it: how the configuration does across systems is not measured yet. The blind judges agreed on almost every item, and a person settled the rest. The other three are still to be made, so the figures will move.

• 2 of 2 scheduled attempts completed with the requested model. Systems on which every attempt completed: 1 of 1.
• Expected score per scheduled attempt: 452.5 (422.0 to 483.0).
• Blocked: 0 (at the start 0, during the work 0). Budget exhausted: 0. Switched: 0. Lost to the lab or the operator, rerun: 0.

Answer key (matrix) SHA-256 9bd0c169e420927d8e0aadd87f30789b4cf6e60a2db49999c450c7f86f2207ce · package SHA-256 3370984d863313e524b7e976eca416d043489fa8f67858e70dabd5998f4dba4e.

Five runs between 384 and 457, mean 421 with a 95% interval of 398 to 442, four Bronze and one Reprovada. All five completed inside the 90-minute budget, about 3.8 million tokens and 15 minutes each (the log keeps no cost), and none broke a behavior the public tests protect (compatibility 100 in every run). Bug fixing was the strongest area (162 of 213 on average), then performance (67 of 213), architecture (53 of 283) and clean code (0 of 141). The reports were rated 39 to 41 of 50. It is one system, so the interval only reflects the variation among the runs on it: how the configuration does across systems is not measured yet. The blind judges agreed on almost every item, and a person settled the rest. It ran with the client's approvals and sandbox bypassed, an exception to the series' rule against it (see the notes).

• 5 of 5 scheduled attempts completed with the requested model. Systems on which every attempt completed: 1 of 1.
• Expected score per scheduled attempt: 421.2 (397.6 to 442.4).
• Blocked: 0 (at the start 0, during the work 0). Budget exhausted: 0. Switched: 0. Lost to the lab or the operator, rerun: 0.

Answer key (matrix) SHA-256 9bd0c169e420927d8e0aadd87f30789b4cf6e60a2db49999c450c7f86f2207ce · package SHA-256 3370984d863313e524b7e976eca416d043489fa8f67858e70dabd5998f4dba4e.

One run of a case of five scored 357 (Reprovada). It completed inside the 90-minute budget, at about US$ 0.28 (the client's own estimate) and 23 minutes, and it did not break a behavior the public tests protect (compatibility 100). Bug fixing was the strongest area (101 of 213), then performance (95 of 213), architecture (20 of 283) and clean code (0 of 141). The report was rated 41 of 50. It is one system, so how the configuration does across systems is not measured yet. The blind judges agreed on almost every item, and a person settled the rest. The other four runs are still to be made, so the figure will move. It ran with the client's auto-approval flag, an exception to the series' rule against bypassing approvals (see the notes). It was served by Novita AI, the host the first series used for this model, in its 0731 version.

• 1 of 1 scheduled attempt completed with the requested model. Systems on which every attempt completed: 1 of 1.
• Expected score per scheduled attempt: 357.0 (357.0 to 357.0).
• Blocked: 0 (at the start 0, during the work 0). Budget exhausted: 0. Switched: 0. Lost to the lab or the operator, rerun: 0.

Answer key (matrix) SHA-256 9bd0c169e420927d8e0aadd87f30789b4cf6e60a2db49999c450c7f86f2207ce · package SHA-256 3370984d863313e524b7e976eca416d043489fa8f67858e70dabd5998f4dba4e.

Two runs of a case of five, between 275 and 291, mean 283 (the interval, 275 to 291, is the range of the two), all Reprovada. Both completed inside the 90-minute budget, at about US$ 0.79 (the client's own estimate) and 53 minutes each, and compatibility was between 70 and 100. Bug fixing was the strongest area (80 of 213 on average), then performance (54 of 213), clean code (20 of 141) and architecture (16 of 283). The reports were rated 35 to 36 of 50. Penalties: run 3 lost 15 points for a new defect introduced by its own change. It is one system, so the interval only reflects the variation among the runs on it: how the configuration does across systems is not measured yet. The blind judges agreed on almost every item, and a person settled the rest. The other three are still to be made, so the figures will move. Some runs ran with the client's default permissions and some with its auto-approval flag (see the notes).

• 2 of 2 scheduled attempts completed with the requested model. Systems on which every attempt completed: 1 of 1.
• Expected score per scheduled attempt: 283.0 (275.0 to 291.0).
• Blocked: 0 (at the start 0, during the work 0). Budget exhausted: 0. Switched: 0. Lost to the lab or the operator, rerun: 1.

Answer key (matrix) SHA-256 9bd0c169e420927d8e0aadd87f30789b4cf6e60a2db49999c450c7f86f2207ce · package SHA-256 3370984d863313e524b7e976eca416d043489fa8f67858e70dabd5998f4dba4e.