Repository navigation
Conversation
…hted_score SuiteRollup.average_weighted_score averaged only the rows with a score, so an ERROR, BUILD_FAILED or unscored TIMEOUT row left the mean and a suite with errors scored higher than a clean one. It now uses measured_scores, the rule run.md's Avg Reliability Score and VariantAggregate.average_score already use: ungraded rows leave the mean, any other unscored row counts as 0.0. It is None when nothing_was_measured holds, as pass_rate is.
uipreliga
left a comment
There was a problem hiding this comment.
Review: coder_eval — pr:265 (5 files) axis:1,2,3,4,5,6,7,8
Scope: pr:265 (5 files) axis:1,2,3,4,5,6,7,8 · branch fix/suite-json-avg-score-counts-errors (PR #265 head) · c7397c7 · 2026-11-21T08:36Z · workflow variant
Change class: complex — changes the formula of a serialized suite.json metric (average_weighted_score) consumed downstream; correctness needs reasoning over status classification
PR #265 correctly makes suite.json's average_weighted_score count unscored ERROR and TIMEOUT rows as 0.0, using the shared measured_scores filter, and it has no type, security, error-handling or architecture defects; the real risks are a REPORT_SCHEMA.md row that tells downstream consumers to expect null for an all-ERROR suite that now publishes 0.0, and a gate-plus-mean expression copied into a second report surface, which is the same per-surface drift that caused this bug; bottom line: merge after you correct the doc row, and add the shared helper and the two missing tests as follow-ups or in this PR.
Summary
| Axis | Score | 🔴 | 🟠 | 🟡 | 🔵 | Top Issue |
|---|---|---|---|---|---|---|
| 1. Code Quality & Style | 9.3 / 10 | 0 | 0 | 1 | 2 | Gated suite headline-mean composition duplicated inline in markdown.py (second copy of run.md gate-plus-mean; nothing_was_measured called twice) |
| 2. Type Safety | 10 / 10 | 0 | 0 | 0 | 0 | — |
| 3. Test Health | 9.8 / 10 | 0 | 0 | 0 | 2 | suite.md now shows a 0.000 average for an all-errored suite, but no render test checks it; the only render test for this behaviour was moved to NOT_GRADED |
| 4. Security | 10 / 10 | 0 | 0 | 0 | 0 | — |
| 5. Architecture & Design | 10 / 10 | 0 | 0 | 0 | 0 | — |
| 6. Error Handling & Resilience | 10 / 10 | 0 | 0 | 0 | 0 | — |
| 7. API Surface & Maintainability | 10 / 10 | 0 | 0 | 0 | 0 | — |
| 8. Evaluation Harness Quality | 9.9 / 10 | 0 | 0 | 0 | 1 | REPORT_SCHEMA null condition for suite.json average_weighted_score contradicts the all-ERROR (0.0) case |
Overall Score: 9.9 / 10 · Weakest Axis: Code Quality & Style at 9.3 / 10
Totals: 🔴 0 · 🟠 0 · 🟡 1 · 🔵 5 across 8 axes.
Blockers
None.
Non-blocking, but please consider before merge
- [Axis 1] Gated suite headline-mean composition duplicated inline in markdown.py (second copy of run.md gate-plus-mean; nothing_was_measured called twice) (
src/coder_eval/reports/markdown.py:962-967) — The PR makes suite.json use the same rule as run.md by pasting run.md's expression a second time. run.md, markdown.py:474/481:scores = measured_scores(...)/if scores and not nothing_was_measured(not_graded=summary.tasks_not_graded, measured=summary.tasks_measured):thensum(scores) / len(scores). suite.json, markdown.py:962-967:scores = measured_scores((r.result.final_status, r.result.weighted_score) for r in rows)/average_weighted_score = (sum(scores) / len(scores) if scores and not nothing_was_measured(not_graded=rows_not_graded, measured=rows_measured) else None). The notes now say the rule "lives once" (orchestration.md:936). That is true of the filter, but the visibility gate and the mean are written out at each surface. This is the same per-surface drift that caused the bug being fixed. The same function also calls the predicate again at markdown.py:1084 (if nothing_was_measured(not_graded=rows_not_graded, measured=rows_measured)). The routed radon result shows_compute_suite_rollupgoing from CC 38 to CC 40 (grade E). Fix: add onemeasured_mean(rows, *, not_graded, measured) -> float | Nonetoresult_metrics.pynext tomeasured_scores, and call it from both report surfaces. In_compute_suite_rollup, computeunmeasured = nothing_was_measured(...)once and use it for bothpass_rateandaverage_weighted_score. Duplication across 2 sites is the Axis 1 Medium anchor.
Nits
- [Axis 1] The new local
scoreshas the same name as the comprehension variable a few lines below (src/coder_eval/reports/markdown.py:962) — The PR renames the old localscoredtoscores(line 962:scores = measured_scores(...)). The per-criterion block at lines 978-986 then readsrows_evaluated=len(scores)andaverage_score=sum(scores) / len(scores) if scores else 0.0insidefor ctype, scores in sorted(by_type.items()). Comprehension scoping keeps the two names separate, so the behaviour is correct. But a reader seeslen(scores)and can take it to mean the suite-level list declared 16 lines above. Rename one of them, e.g. keepsuite_scoresfor the rollup mean, ortype_scoresin the comprehension. - [Axis 1] The new suite.json parity test copies the previous test's row fixture word for word (
tests/test_ungraded_reporting.py:154-164) —test_suite_json_and_the_variant_average_publish_one_mean_for_one_set_of_rowsre-declaresrows: list[tuple[FinalStatus, float | None]] = [(FinalStatus.SUCCESS, 1.0), (FinalStatus.FAILURE, 0.0), (FinalStatus.ERROR, None), (FinalStatus.TIMEOUT, None), (FinalStatus.NOT_GRADED, None)]and thevariant_rows = [VariantResult(...) for i, (status, score) in enumerate(rows)]comprehension. Both are already at lines 121-131 intest_run_md_and_the_variant_average_publish_one_mean_for_one_set_of_rows. Both tests assert the same parity (== _mean_graded_score(variant_rows) == 0.25). Move the fixture to a module-level constant, or merge the two into one test that checks run.md, suite.json and the variant mean against one set of rows. That merged test states the 'one mean, every surface' contract once. - [Axis 3] suite.md now shows a 0.000 average for an all-errored suite, but no render test checks it; the only render test for this behaviour was moved to NOT_GRADED (
tests/test_suite_rollup.py:487) — Before this PR,test_omits_avg_score_when_nonebuilt a singleFinalStatus.ERRORrow (witherror_message="boom") and checked that suite.md left out the average. The PR changes that row tofinal_status=FinalStatus.NOT_GRADED, which is right, because an errored suite now scores 0.0. But no test replaced the old one for the new suite.md output. The parametrized caseid="all-error-suite-scores-zero"(line 152) checks only the rollup value, and the one test that checks the rendered line (line 388,assert "Average weighted score**:" in md) uses a 2/3 SUCCESS suite with a non-zero mean. If someone changes the render guard at markdown.py:1112 (if rollup.average_weighted_score is not None:) to a truthiness check, the errored-suite 0.000 line disappears again and no test fails. Add a render check next to the NOT_GRADED case: build twoFinalStatus.ERRORrows withweighted_score=None, thenassert "**Average weighted score**: 0.000" in _render_suite_markdown(rollup). This covers both sides of the new rule: NOT_GRADED hides the average, ERROR shows 0.000. - [Axis 3] The new parametrized test covers only an unscored TIMEOUT; a TIMEOUT that has a score (counts at its own score, not 0.0) is never tested (
tests/test_suite_rollup.py:150) — The new REPORT_SCHEMA.md row says a row with no score, including "an unscoredTIMEOUT", counts as0.0. That wording implies a scored TIMEOUT keeps its ownweighted_score. Both new tests (pytest.param([*_GRADED_ROWS, ("TIMEOUT", None)], 1 / 3, id="unscored-timeout-counts-as-zero")here, and(FinalStatus.TIMEOUT, None)in tests/test_ungraded_reporting.py:158) use only the unscored form. No test covers the status-category path for a TIMEOUT with a partial score. Addpytest.param([*_GRADED_ROWS, ("TIMEOUT", 0.5)], 0.5, id="scored-timeout-keeps-its-score"). The expected value is (1.0+0.0+0.5)/3 = 0.5. It also shows thatrows_measuredcounts the scored TIMEOUT as evidence. - [Axis 8] REPORT_SCHEMA null condition for suite.json average_weighted_score contradicts the all-ERROR (0.0) case (
docs/REPORT_SCHEMA.md:333) — The new row says: "nullwhen no row produced a verdict, aspass_rateis." and "the same rule as run.md's Avg Reliability Score andVariantAggregate.average_score". The gate in reports/markdown.py:963-966 issum(scores) / len(scores) if scores and not nothing_was_measured(not_graded=rows_not_graded, measured=rows_measured) else None, andnothing_was_measuredisnot_graded > 0 and measured == 0. So a suite of only ERROR rows, where no row produced a verdict, returns 0.0, not null. The PR's own test pins this:pytest.param([("ERROR", None), ("ERROR", None)], 0.0, id="all-error-suite-scores-zero"). Also, for [NOT_GRADED, ERROR] suite.json gives null butVariantAggregate.average_score(_mean_graded_score) gives 0.0, which I checked by running it. So the 'same rule' claim covers the zeroing rule only, not the null gate. A downstream consumer (the coder-eval-uipath nightly) that reads this contract would expect null for an all-crash suite. Change the wording to the real condition, e.g. "nullwhen the suite has no rows, or when it has NOT_GRADED rows and no row has a score (the sharednothing_was_measuredgate). A suite of only errored rows scores 0.0." Also say thatVariantAggregate.average_scoredoes not have the null gate yet (the known gap in .claude/notes/orchestration.md). Out of scope for this axis but relevant: the staleSuiteRollup.average_weighted_scorefield description in models/results.py:971 ("Mean weighted_score across rows that produced one.") belongs to the What's Missing pass.
What's Missing
Downstream consumers:
- 🟡 The Pydantic field description for
SuiteRollup.average_weighted_score(src/coder_eval/models/results.py:970-972) was not updated. It still says "Mean weighted_score across rows that produced one.", which is the old rule this PR removes. The project treats the schema model as the single source of truth for field definitions (CLAUDE.md DRY / SSOT). After this PR, the model contradicts both the new REPORT_SCHEMA.md row and markdown.py:962-967: ERROR/BUILD_FAILED/unscored TIMEOUT rows count as 0.0, NOT_GRADED rows leave the mean, and the value is None whennothing_was_measuredholds. Rewrite the description to the real rule, in the same style as thepass_ratedescription above it. (trigger: src/coder_eval/reports/markdown.py) (restates: Axis 8: REPORT_SCHEMA null condition for suite.json average_weighted_score contradicts the all-ERROR (0.0) case)
Parallel paths:
- 🔵 suite.json now has the
nothing_was_measurednull gate, but its siblingVariantAggregate.average_score(_mean_graded_scorein orchestration/experiment.py:807) still does not. For one set of rows, e.g. [NOT_GRADED, ERROR], suite.json publishes null and experiment.json publishes 0.0. The new parity test claims "one mean for one set of rows", but its five-row fixture has measured rows, so the two surfaces agree there and the divergence never shows. Add a case with no measured rows that pins the current divergence (or xfail it with a pointer to the known gap in .claude/notes/orchestration.md), so the follow-up experiment.json PR has a test to flip. (trigger: tests/test_ungraded_reporting.py) (restates: Axis 8: REPORT_SCHEMA null condition for suite.json average_weighted_score contradicts the all-ERROR (0.0) case)
Tests:
- 🔵
test_omits_avg_score_when_nonewas moved from an ERROR row to a NOT_GRADED row, and nothing replaced the ERROR render case. No test checks that suite.md now shows**Average weighted score**: 0.000for an all-errored suite. The new 0.0 value is tested only at the rollup level, not at the markdown.py:1112 render guard, so a change from theis not Nonecheck to a truthiness check would pass every test. (trigger: tests/test_suite_rollup.py) (restates: Axis 3: suite.md now shows a 0.000 average for an all-errored suite, but no render test checks it)
Nightly pipeline:
- 🔵 The PR body says the score goes down and that
passed/CI exit codes do not change. It does not cover two effects on consumers of the suite.json report shape (the coder-eval-uipath nightly and any dashboards or trend queries over suite.json): (a) for an all-ERROR suite the value changes type, from null to 0.0, so a consumer that read null as 'the suite crashed' now sees a real-looking zero; (b) every suite with errored rows has a one-time step drop in its historical trend at the merge date. State both in the PR description and the release notes, so readers of the nightly trend do not take the drop for an agent regression. (trigger: docs/REPORT_SCHEMA.md)
Harness & Lint Improvements
Static checks (lint / type):
- [ce-lint] CE072 (next free number after CE071; check CE069/CE070 are not taken on another branch first). First do the fix: add
measured_mean(scores: list[float], *, not_graded: int, measured: int) -> float | Nonetosrc/coder_eval/result_metrics.py, next tomeasured_scores. It holds the visibility gate and the mean together:None if not scores or nothing_was_measured(...) else sum(scores) / len(scores). Then the rule: insrc/coder_eval/outsideresult_metrics.py, a function may not takesum(v) / len(v)(orstatistics.mean(v)/fmean(v)) wherevis a name bound frommeasured_scores(...)or from the_measured_scores(...)wrapper. It may also not callnothing_was_measured(more than once. Both are found with an AST walk of one function body: record the names assigned from those calls, then flag aBinOp(Div)whose operands aresum(<name>)andlen(<name>), and count thenothing_was_measuredcalls. Put it attests/lint/rules/ce072_headline_mean_via_helper.pyand add it totests/lint/runner.py. Blind spots for the docstring: a list that goes through a rename or a helper, and a mean computed by hand in a loop. Prevents: Finding 1 (A1/A5 medium). It would flag the gate-and-mean written out again at reports/markdown.py:962-967 (suite.json) and the existing copy at markdown.py:474/481 (run.md). It would also flag the secondnothing_was_measuredcall at markdown.py:1084 in_compute_suite_rollup. This rule follows the CLAUDE.md order 'delete before you guard': the helper is the fix, and the rule only stops someone from writing the expression inline again. Onceorchestration/experiment.py:_mean_graded_scoreis changed to callmeasured_mean, the same rule also closes the suite.json-vs-VariantAggregate.average_scorenull-gate split described in Finding 6. Today experiment.py:807 takes its own mean without the gate. - [ce-lint] CE073: extend the idea of CE058 ('unknown must stay None, not 0.0') from timing fields to score and rate fields in the rendering layer. In
src/coder_eval/reports/andsrc/coder_eval/result_metrics.py, the rule forbids a truthiness test on a name or attribute whose last segment matches^(pass_rate|average_[a-z_]*score|[a-z_]*_score|[a-z_]*_rate)$. The forbidden forms areif x:,if not x:,x or <default>, andx if x else .... The code must useis None/is not Noneinstead. This is an AST check onIf.test,IfExp.test,BoolOp(Or)andUnaryOp(Not). For the file layout, copy CE058: compile one_SCORE_NAMEregex and allow# noqa: CE073where a falsy check is intended. Do not use pyright for this.reportOptionalOperanddoes not flag truthiness onfloat | None, and no ruff rule does either. Prevents: Finding 4 (A3 low). The new suite.md line 'Average weighted score: 0.000' for an all-ERROR suite depends on the guardif rollup.average_weighted_score is not None:at markdown.py:1112. If someone changes it to a truthiness check, the line is hidden again and no test fails. The rule makes that change failmake lint, whether or not a render test exists. It also protects the guards beside it onpass_rate,average_scoreandratein the reports layer, where 0.0 is a real published value. - [ci-gate] Add a report-schema parity test, the same type as CE030 / CE033. For each run-record model that docs/REPORT_SCHEMA.md documents in a
| Key | Type | Meaning |table (start withSuiteRollup,CriterionStats,FailedRowSummary,CriterionAggregate), the set of serialized field names must equal the set of keys in the table. The better option is to generate each 'Meaning' cell from the PydanticField(description=...)(CLAUDE.md 'DRY: field descriptions defined once'), with amake report-schematarget and a drift check, likemake plugin-reference/ CE033. Then the null condition is written in one place, next to the field. Add it as a@pytest.mark.lintclass in tests/test_custom_lint.py, because it reasons over Markdown. Prevents: Finding 6 (A1/A7/A8 low). The REPORT_SCHEMA.md:333 row saysnullwhen no row produced a verdict, but code publishes 0.0 for an all-ERROR suite. The model description at models/results.py:971 says 'Mean weighted_score across rows that produced one.', which is also stale. So the field has two prose sources and both have drifted from the code. Generating the row from the model leaves one source to keep correct, and the stale description at models/results.py:971 then shows in the doc diff. A static check cannot tell whether the prose matches the gate in markdown.py:963-966. The cross-surface matrix test in the harness bucket covers that part.
Harness improvements (not statically reachable):
- Replace the two copied parity tests (tests/test_ungraded_reporting.py:121-131 and 154-164) with one 'one mean, every surface' matrix test. Declare the row mixes once at module level, then parametrize over surfaces × row mixes. The surfaces are run.md Avg Reliability Score, suite.json
average_weighted_score,VariantAggregate.average_scorevia_mean_graded_score, and the rendered suite.md line. The row mixes must include: graded-only; +ERROR(None); +unscored TIMEOUT; +scored TIMEOUT(0.5) (expected (1.0+0.0+0.5)/3 = 0.5, androws_measuredcounts it); all-ERROR (expected 0.0, and suite.md shows**Average weighted score**: 0.000); all-NOT_GRADED (expected None, line hidden); [NOT_GRADED, ERROR]. For every mix, the test asserts that every surface publishes the same value, or it records the known divergence as an explicitxfailwith a pointer to the gap noted in .claude/notes/orchestration.md. Why not static: The defect is a difference in computed values between surfaces that take different input shapes (task_results dicts, EvaluationResult rows, VariantResult). Only running each surface on the same rows can show that they publish different numbers. An AST rule cannot evaluate the gate. Prevents: Finding 3 (copied fixture), Finding 4 (no render test for all-ERROR 0.000), Finding 5 (scored TIMEOUT never tested), and Finding 6 (the [NOT_GRADED, ERROR] split: suite.json gives None and VariantAggregate gives 0.0, which the doc wrongly calls 'the same rule'). The xfail row puts the known gap on record, so it cannot drift again without notice. - Do not add a lint rule for a comprehension target that shadows a local in the enclosing function (
scoresat markdown.py:962 versusfor ctype, scores in sorted(by_type.items())at 978-986). Fix it with a rename (suite_scores/type_scores). Record the decision in .claude/harness-candidates.md so later reviews do not propose the rule again. Why not static: The pattern can be detected statically, but ruff does not implement pylint's redefined-outer-name for comprehension scopes. A custom rule would flag many legal, readable uses in the codebase. The CLAUDE.md bar is that 'a rule is right when the pattern is genuinely unavoidable', and this pattern only affects readability, with no behaviour change. So the check gives too little value for its noise, and this is recorded as a decision. Prevents: Finding 2 (A1 low) — a review note only, with no gate. - Add a section to the review checklist and .claude/notes/orchestration.md § Aggregation: when a PR changes a headline-mean or rate rule, it must list every surface that publishes that number. The surfaces are run.md, suite.json/suite.md, experiment.json
VariantAggregate, the evalboard, and reports/junit. For each surface, the PR must say whether that surface changed or keeps a known gap. It must also state the cross-repo effect on the coder-eval-uipath nightly. Examples: suite.jsonaverage_weighted_scorenow goes from null to 0.0 for an all-crash suite, and a consumer that reads the doc as 'null on no verdict' sees a different value. Why not static: The list of surfaces includes consumers outside this repo, and the question 'does this change the nightly's reading of the field' needs judgment. No in-repo check can see those consumers. Prevents: Finding 6 (contract wording) and the per-surface drift class behind Finding 1. The bug this PR fixes is itself an example of that drift.
Top 5 Priority Actions
- Change the average_weighted_score row at docs/REPORT_SCHEMA.md:333 to state the real null condition: null only when the suite has no rows, or when it has NOT_GRADED rows and no scored row (nothing_was_measured); a suite of only ERROR rows scores 0.0; VariantAggregate.average_score does not yet apply the null gate.
- Add one measured_mean(scores, *, not_graded, measured) -> float | None helper to result_metrics.py next to measured_scores, call it from both run.md (src/coder_eval/reports/markdown.py:474-481) and suite.json (src/coder_eval/reports/markdown.py:962-967), and in _compute_suite_rollup compute nothing_was_measured once so that pass_rate (markdown.py:1084) and the average use the same value.
- Add a render test next to tests/test_suite_rollup.py:487 that builds two FinalStatus.ERROR rows with weighted_score=None and asserts "Average weighted score: 0.000" is in _render_suite_markdown(rollup), so that a truthiness change to the guard at markdown.py:1112 fails a test.
- Add pytest.param([*_GRADED_ROWS, ("TIMEOUT", 0.5)], 0.5, id="scored-timeout-keeps-its-score") at tests/test_suite_rollup.py:150 to prove that a scored TIMEOUT keeps its own score and counts toward rows_measured.
- Rename the suite-level local
scoresat src/coder_eval/reports/markdown.py:962 (for example to suite_scores) so it does not have the same name as the per-criterion comprehension variable at lines 978-986, and merge the run.md and suite.json parity tests in tests/test_ungraded_reporting.py:121-164 into one test with one shared row fixture.
Stats: 0 🔴 · 0 🟠 · 1 🟡 · 5 🔵 across 8 axes reviewed.
The suite.json average and run.md's score each wrote out the same gate and mean inline. Both now call result_metrics.measured_mean, which takes measured_scores' mean and returns None when nothing_was_measured holds. _compute_suite_rollup is back to its main complexity (radon E 38). - REPORT_SCHEMA.md and SuiteRollup's field description state the real null condition: an all-ERROR suite scores 0.0, and VariantAggregate.average_score does not apply the gate yet. - Tests: a scored TIMEOUT keeps its score; suite.md renders 0.000 for an all-errored suite; the run.md and suite.json parity tests share one helper; the variant-mean gap is pinned by a strict xfail.
|
Thanks. Pushed 03e67d0:
Not here: CE072/CE073, the report-schema parity check and the review-checklist section all sound useful, but they're bigger than this fix. |
suite.json's
average_weighted_scoreonly averaged rows that had aweighted_score, so ERROR, BUILD_FAILED and unscored TIMEOUT rows left the mean. A suite with one SUCCESS (1.0), one FAILURE (0.0) and one ERROR reported 0.500, where run.md and experiment.json report 0.333 for the same rows. This is the suite.json gap left out of #257 and recorded in.claude/notes/orchestration.md._compute_suite_rollupnow usesresult_metrics.measured_scores: NOT_GRADED rows leave the mean, any other unscored row counts as 0.0.Nonewhennothing_was_measuredholds, the same gate as run.md's score andpass_rate. Anexecutesuite with a crashed row shows no score instead of 0.000.Heads up: suite.json's
average_weighted_scoreand suite.md's "Average weighted score" go down for suites with errored rows.passedand the CI exit code don't read it, so gates don't change.Tests: a parametrized case in
test_suite_rollup.py(ERROR, BUILD_FAILED and unscored TIMEOUT count as 0.0, NOT_GRADED leaves, all-ERROR gives 0.0, all-NOT_GRADED and execute-with-a-crash giveNone), and a parity test that suite.json andVariantAggregate.average_scoregive 0.25 for the same five rows.test_mixed_outcomesnow expects 0.467 instead of 0.7.test_omits_avg_score_when_noneuses a NOT_GRADED row, since an all-ERROR suite now scores 0.0. 6 of these fail on main.make verifypasses.This and the experiment.json PR both edit the same paragraph in the orchestration notes, so whichever lands second needs a small merge there. I'll take care of it.
For nightly readers of suite.json: a suite of only errored rows now reports
average_weighted_score0.0 instead ofnull, and any suite with errored rows steps down once at the merge date. Neither is an agent regression.