Repository navigation
Conversation
…eplicates aggregate_results counted VariantAggregate.tasks_measured from the folded VariantResult score, and _fold_replicates turns an errored replicate's missing score into 0.0. So one crash in an `execute` run counted as a verdict: experiment.json reported pass_rate 0.0 and average_score 0.0 where run.json reported None for the same rows. tasks_measured now counts tasks with at least one replicate that carries a verdict, and average_score is None when nothing_was_measured holds, as pass_rate is.
uipreliga
left a comment
There was a problem hiding this comment.
Review: coder_eval — pr:266
Scope: pr:266 · branch fix/experiment-measured-from-replicates · c62d28c · 2026-11-21T07:24Z · workflow variant
Change class: complex — changes how experiment.json counts tasks_measured and when average_score is null, i.e. persisted metric/classification semantics
PR #266 is sound: the nothing-measured gate on VariantAggregate works, security and error handling have no findings, and no finding changes a task's own score or final_status, but the gate reads a separate key-joined measured_keys set and not the per-task fold, so experiment.json and experiment.md can still name a winner and show 0.0 rates for a variant that has no verdict; fix that and add a two-variant test before you merge.
Summary
| Axis | Score | 🔴 | 🟠 | 🟡 | 🔵 | Top Issue |
|---|---|---|---|---|---|---|
| 1. Code Quality & Style | 9.3 / 10 | 0 | 0 | 1 | 2 | aggregate_results grows to CC 42 (F) from 38 (E); the PR adds more inline per-variant logic instead of extracting it |
| 2. Type Safety | 9.9 / 10 | 0 | 0 | 0 | 1 | The VariantAggregate field descriptions (average_score, tasks_measured) describe a gate that the model does not apply; only the producer applies it |
| 3. Test Health | 9.3 / 10 | 0 | 0 | 1 | 2 | The new per-(task, variant) verdict key is tested only with a single variant; a regression that keys on task_id alone passes the full suite |
| 4. Security | 10 / 10 | 0 | 0 | 0 | 0 | — |
| 5. Architecture & Design | 9.5 / 10 | 0 | 0 | 1 | 0 | Nothing-measured gate is applied at VariantAggregate but not at the per-task fold, so experiment.json (and experiment.md) contradict themselves for a crashed variant |
| 6. Error Handling & Resilience | 10 / 10 | 0 | 0 | 0 | 0 | — |
| 7. API Surface & Maintainability | 9.7 / 10 | 0 | 0 | 0 | 3 | The notes claim the experiment.json/run.json crash divergence is fixed, but a crashed replicate plus an ungraded replicate still folds to ERROR and reports 0.0 |
| 8. Evaluation Harness Quality | 10 / 10 | 0 | 0 | 0 | 0 | — |
Overall Score: 9.7 / 10 · Weakest Axis: Code Quality & Style at 9.3 / 10
Totals: 🔴 0 · 🟠 0 · 🟡 3 · 🔵 8 across 8 axes.
Blockers
None.
Non-blocking, but please consider before merge
- [Axis 1] aggregate_results grows to CC 42 (F) from 38 (E); the PR adds more inline per-variant logic instead of extracting it (
src/coder_eval/orchestration/experiment.py:855) — radon at PR head reportsF 855:0 aggregate_results - F (42). On main it was E (38). The PR adds three new branches inline: themeasured_keys = { key for key, reps in task_variant_reps.items() if any(r.result.weighted_score is not None for r in reps) }pass (line 894),tasks_measured = sum(1 for v in vr_list if (v.task_id, vid) in measured_keys)(line 945), and theaverage_score=(None if nothing_was_measured(...) else _mean_graded_score(vr_list))conditional (lines 957-961). The module already records why this function should shrink:_fold_replicatessays it was "Extracted fromaggregate_results, which was already the largest function in the module" (line 828). The PR goes the other way. Extract the per-variant block (lines 925-965) into a helper, e.g._aggregate_variant(vid, vr_list, reps_by_key) -> VariantAggregate, and count tasks_measured inside it straight from the replicates (sum(1 for (t, v), reps in reps_by_key.items() if v == vid and any(r.result.weighted_score is not None for r in reps))). That also removes the separatemeasured_keysset and the tuple lookup. orchestration/experiment.py is not a hot module, so this is filed at Medium. - [Axis 3] The new per-(task, variant) verdict key is tested only with a single variant; a regression that keys on task_id alone passes the full suite (
tests/test_ungraded_reporting.py:177) — The new aggregate_results logic keys verdict evidence on the (task, variant) pair:measured_keys = { key for key, reps in task_variant_reps.items() if any(r.result.weighted_score is not None for r in reps) }(src/coder_eval/orchestration/experiment.py:894-895) andtasks_measured = sum(1 for v in vr_list if (v.task_id, vid) in measured_keys)(:945). Every new test callsaggregate_results("e", "", ["v"], rows, 1.0)(line 177 and 188), and_task_rowhardcodesvariant_id="v", so the variant part of the key is never tested. I made a mutation in /tmp/pr-266-worktree: I changed the set tokey[0] for key ...and the lookup toif v.task_id in measured_keys. With that mutation the routed subset still gave 366 passed, 3 skipped. The full non-live suite failed only the same 10 test_judge_litellm.py tests (a missing litellm module) that also fail without the mutation. So nothing in the repo catches it. With the mutation, in an A/B where variant A scores t0 and variant B only crashes or runs ungraded on t0, B counts as measured. Then B's pass_rate and average_score become 0.0 where they should be None. That is the exact bug this PR fixes, now leaking between variants. Fix: add a parametrized case with two variants. Let_task_rowtake avariant_idargument. Example rows: t0 OK 1.0 in varianta; t0 ERR plus t1 NG in variantb. Callaggregate_results(..., ["a", "b"], ...)and assertvariant_aggregates["b"].tasks_measured == 0,.pass_rate is Noneand.average_score is None, and thatastays measured. - [Axis 5] Nothing-measured gate is applied at VariantAggregate but not at the per-task fold, so experiment.json (and experiment.md) contradict themselves for a crashed variant (
src/coder_eval/orchestration/experiment.py:894) — The PR states that an errored replicate's 0.0 from_fold_replicatesis not a verdict. It applies that rule to only one consumer of the fold. It does not carry the evidence on the foldedVariantResult. Instead it adds a separate set that is joined back by key:measured_keys = { key for key, reps in task_variant_reps.items() if any(r.result.weighted_score is not None for r in reps) }(L894) andtasks_measured = sum(1 for v in vr_list if (v.task_id, vid) in measured_keys)(L945). The other consumer of the same fold in the same function still reads the folded 0.0 as a score:scored = [(v, v.weighted_score) for v in variants if v.weighted_score is not None](L904) sets best_variant, is_tie and score_spread. reports/experiment.py:288 and reports/html.py:1469 count win rates from those values. I ran this on the PR head: execute run, variants A and B, t0 = A ERROR / B NOT_GRADED, t1 = both NOT_GRADED. Result: both aggregates give tasks_measured=0, pass_rate=None, average_score=None, but t0 has best_variant='A', is_tie=False. So experiment.md shows Average Score and Pass Rate as '—' for A, and also 'A: 1/2 tasks (50%)' under Win Rates. Before the PR the two halves agreed, because both treated 0.0 as a score. The PR introduces this divergence. The edited notes paragraph (.claude/notes/orchestration.md:938-947) now says 'an execute run with a crash has no pass_rate or average_score' just before 'Only graded variants can win or set a spread', and the code no longer makes the second statement true. Fix: put the evidence on the fold. GiveVariantResulta measured flag (or measured-replicate count) set in_fold_replicates, or make_fold_replicatesreturn weighted_score=None when no replicate has a verdict. Then the aggregate, the winner/spread selection, and the renderers all read one field, and the key-joined side set goes away. This also removes the +4 CC this PR adds toaggregate_results(radon F(42), was E(38)).
Nits
- [Axis 1] The gated measured-mean rule is now written at two consumer sites, but the notes say the rule lives once (
src/coder_eval/orchestration/experiment.py:957) — The newaverage_score=(None if nothing_was_measured(not_graded=tasks_not_graded, measured=tasks_measured) else _mean_graded_score(vr_list))repeats the composition already in reports/markdown.py:481:if scores and not nothing_was_measured(not_graded=summary.tasks_not_graded, measured=summary.tasks_measured):. The edited notes paragraph still says "The rule lives once, inresult_metrics.measured_scores". Now the mean rule and the gate are put together separately at each call site. VariantAggregate's own pass_rate also applies this gate inside the model, but average_score is gated only by its one constructor. Put the gated mean into one helper in result_metrics.py, e.g.measured_mean(rows, *, not_graded, measured) -> float | None, and call it from both sites. Alternatively, enforceaverage_score is Noneundernothing_was_measuredin a VariantAggregate validator, so the model cannot hold the inconsistent state. - [Axis 1] The notes paragraph says run.json has no average_score, but run.json does not store an average score (
.claude/notes/orchestration.md:940) — Lines 940-941 say "so anexecuterun with a crash has nopass_rateoraverage_scorein experiment.json, as in run.json." run.json has no average-score field. docs/REPORT_SCHEMA.md:74 says "run.md's Avg Reliability Score is not stored inrun.json". Write it as "...in experiment.json, as run.json has no pass_rate and run.md shows no Avg Reliability Score", so the parity claim names the surface that holds each value. - [Axis 2] The VariantAggregate field descriptions (average_score, tasks_measured) describe a gate that the model does not apply; only the producer applies it (
src/coder_eval/models/experiment.py:235) — The PR changes whenaverage_scoreis None. The aggregator now writesNone if nothing_was_measured(not_graded=tasks_not_graded, measured=tasks_measured) else _mean_graded_score(vr_list)(orchestration/experiment.py:957-961). So 9 NOT_GRADED rows plus 1 ERROR row (1 graded row, 0 measured) now gives None. But the contract on the persisted field still reads# None when nothing in this variant was graded: a 0.0 is indistinguishable from/# "measured and scored zero"./average_score: float | None(lines 235-237). The field has noField(description=...), and the inline comment at orchestration/experiment.py:956 (# GRADED rows only, and None when there are none.) has the same staleness. The PR updated thetasks_measureddescription to say it 'Gates pass_rate and average_score', but the gated field's own contract was not updated. The model also does not enforce the gate.pass_rateis a computed_field that callsnothing_was_measureditself, butaverage_scoreis a plain stored field. A VariantAggregate built anywhere else, or loaded from experiment.json, can carryaverage_score=0.0withtasks_measured=0, tasks_not_graded>0. The model accepts this even thoughpass_rateon the same object is None. Fix: change line 237 toaverage_score: float | None = Field(description="Mean over measured rows (errored = 0.0, ungraded leaves). None when no row is measured or nothing_was_measured holds.")and delete the stale comments. Optionally, also add a check to_check_task_count_invariant(or a sibling after-validator) that rejects a non-Noneaverage_scorewhennothing_was_measured(...)holds. Then the model holds the invariant thatpass_ratealready holds. - [Axis 3] The run.json/experiment.json parity test checks tasks_measured and pass_rate but not the newly gated average_score (
tests/test_ungraded_reporting.py:190) — The PR claims that average_score is now None when nothing_was_measured holds, so that it agrees with pass_rate and with run.md's Avg Reliability Score. But the cross-surface test asserts onlyassert (aggregate.tasks_measured, aggregate.pass_rate) == (summary.tasks_measured, summary.pass_rate) == (0, None). The parametrized test checks average_score on the aggregate alone, so no test compares the two surfaces on the execute-plus-crash rows. Addassert aggregate.average_score is Noneto this test. Also add an assertion thatReportGenerator.generate_markdown(summary)has noAvg Reliability Scoreline (markdown.py:481 hides it under the same nothing_was_measured gate). This follows the existingtest_run_md_and_the_variant_average_publish_one_mean_for_one_set_of_rows. Then the agreement claim holds on all three surfaces. - [Axis 3] The documented known edge (a crashed replicate plus ungraded replicates only, which still reports 0.0) has no test (
tests/test_ungraded_reporting.py:152) — Casea-crashed-replicate-is-not-a-verdict([_task_row("t0", _ERR), _task_row("t0", _NG, replicate_index=1), _task_row("t1", _NG)]) gets None only because the separate task t1 adds to tasks_not_graded. The PR text and the orchestration notes accept an edge case: a variant made ONLY of tasks whose replicates are a crash plus ungraded runs._pick_worst_statusfolds each such task to ERROR, so tasks_not_graded stays 0 and pass_rate and average_score show 0.0. No test pins this. A later fix, or a silent change in that behavior, will not show in the suite. Add a case with only[_task_row("t0", _ERR), _task_row("t0", _NG, replicate_index=1)]. Mark itpytest.mark.xfail(strict=True, reason=...)with the target of None, or assert the current 0.0 with a reason that points to the notes. Then the gap is recorded in a test, not only in prose. - [Axis 7] The notes claim the experiment.json/run.json crash divergence is fixed, but a crashed replicate plus an ungraded replicate still folds to ERROR and reports 0.0 (
.claude/notes/orchestration.md:938) — Lines 938-941 now say: "VariantAggregate.tasks_measuredis counted from the replicates ... so anexecuterun with a crash has nopass_rateoraverage_scorein experiment.json, as in run.json. One known gap remains." The only gap listed is suite.json. The PR description admits a second gap, and I reproduced it at PR head. Underexecutewith repeats, give each task one ERROR replicate and one NOT_GRADED replicate (t0: ERR+NG, t1: ERR+NG).build_run_summarythen reports tasks_measured=0, tasks_not_graded=2, pass_rate=None.aggregate_resultsreports tasks_measured=0, tasks_not_graded=0, pass_rate=0.0, average_score=0.0. The cause:tasks_measuredis now read from the replicates (experiment.py:893measured_keys), but the other input to the gate,tasks_not_graded = sum(1 for v in vr_list if v.final_status.category == "ungraded")(experiment.py:942), is still read from the fold._pick_worst_statusfolds ERR+NG to ERROR. Sonothing_was_measuredsees not_graded=0. Either (a) add this case to the notes' known-gap list, so the cross-repo consumer (eval-runner reads experiment.json) knows that the two files can still disagree, or (b) close the gap the same way asmeasured_keys: compute 'has an ungraded replicate' fromtask_variant_repsand pass that count to thenothing_was_measuredgate. If you choose (b), add the case to the parity test in tests/test_ungraded_reporting.py. - [Axis 7] The comments on average_score's None rule in aggregate_results are stale after the gate changed (
src/coder_eval/orchestration/experiment.py:956) — The PR adds a second None condition toaverage_score:None if nothing_was_measured(not_graded=tasks_not_graded, measured=tasks_measured) else _mean_graded_score(vr_list). This condition also nulls a variant that has ERROR-bucket rows, which are graded by category. But the inline comment directly above it (line 956) still says# GRADED rows only, and None when there are none.. In the touched model file, src/coder_eval/models/experiment.py:235, the comment# None when nothing in this variant was graded: a 0.0 is indistinguishable from ...was also not updated. Change both to 'None when no row carries a verdict (nothing_was_measured) or every row is ungraded'. Also, the model's field description at models/experiment.py:261 says tasks_measured "Gates pass_rate and average_score", but the model computes only thepass_rategate. Theaverage_scoregate is applied only by the producer,aggregate_results. Name that producer in the description, or delete the comment and let the description carry the contract. Thepass_ratedocstring warns against exactly this kind of 'claim rather than fact' mirror. - [Axis 7] The variant.json field list mentions pass_rate but does not list the serialized pass_rate / tasks_graded fields (
docs/REPORT_SCHEMA.md:299) — The new text says tasks_measured "gatespass_rateandaverage_scoreasRunSummary.tasks_measuredgates the run's rate". But thevariant.json — VariantAggregatefield list (lines 292-300) still does not listpass_rateortasks_graded. Both are@computed_fields onVariantAggregate, soagg.model_dump_json(reports/experiment.py:598) serializes them into variant.json and experiment.json. The run.json table documents both (lines 65 and 67). This PR is the edit that makes the doc point atpass_rate, so addpass_rate(float | None, None when nothing_was_measured or tasks_graded==0) andtasks_graded(tasks_run - tasks_not_graded) to this list. Then a downstream consumer can find the gated field that the sentence names.
What's Missing
Nightly pipeline:
- 🟡 The PR changes the value of a serialized experiment.json field on the graded
runpath, which is the nightly path, and does not say so. Before,tasks_measuredcounted folded rows whose score was not None, and the fold scores an errored task 0.0, so an errored task counted. Now an errored task does not count. Example: a graded nightly with 10 tasks, 2 of them ERROR, publishedtasks_measured=10and now publishes 8.pass_rateandaverage_scoredo not change, becausenothing_was_measuredneedsnot_graded > 0. The description says only that graded all-ERROR runs "still report 0.0". The notes also say a change to a serialized field that downstream consumers read must be its own change. So the PR description should state the value change for eval-runner and other experiment.json readers. (trigger: src/coder_eval/orchestration/experiment.py)
Downstream consumers:
- 🟡 The new rule "a crashed replicate is not a verdict" reaches only the per-variant aggregate. Other consumers of the same fold still read the 0.0:
task_summariesbest_variant / is_tie / score_spread, the Win Rates in reports/experiment.py:282-292 and reports/html.py:1463-1472, and each per-taskVariantResult.weighted_scorecell. For an execute plus crash run, experiment.json and experiment.md show '—' for the variant's Average Score and Pass Rate, but give the same variant a win and a 0.000 score cell. (trigger: src/coder_eval/orchestration/experiment.py) (restates: Axis 5: Nothing-measured gate is applied at VariantAggregate but not at the per-task fold) - 🔵 Two places in docs/REPORT_SCHEMA.md still describe the old rule. The
average_scoresentence at lines 294-295 ("an errored row counts as 0.0 ... null when nothing was measured") is now wrong for execute runs: an execute run with a crash is null even though its errored row is "measured" as 0.0 by that rule. The task_summaries text at lines 313-316 says a per-taskweighted_scoreerrored replicate counts as 0.0, the "same rule as VariantAggregate.average_score above". After this PR the two rules are no longer the same. Update both so experiment.json consumers see the new gate. (trigger: docs/REPORT_SCHEMA.md) (restates: Axis 2: The VariantAggregate field descriptions (average_score, tasks_measured) describe a gate that the model does not apply) - 🔵 In .claude/notes/orchestration.md, the paragraph after the edited one is now stale. Lines 954-957 say "
tasks_measuredcounts rows with a score" and "average_scoreisNonerather than0.000when nothing was graded". Both now contradict the replicate-based count and thenothing_was_measuredgate, which the PR added a few lines above. Update the paragraph, or merge it into the edited one. (trigger: .claude/notes/orchestration.md) (restates: Axis 7: The comments on average_score's None rule in aggregate_results are stale after the gate changed)
Parallel paths:
- 🔵 The PR moved only one of the two inputs of the
nothing_was_measuredgate from the fold to the replicates.tasks_measurednow comes fromtask_variant_reps.tasks_not_gradedstill counts foldedfinal_status == ungraded, and_pick_worst_statusturns ERR+NG into ERROR. So a crash plus ungraded replicates still diverges from run.json, wherebuild_run_summarycounts raw replicate rows. (trigger: src/coder_eval/orchestration/experiment.py) (restates: Axis 7: The notes claim the experiment.json/run.json crash divergence is fixed) - 🔵 The PR edited the
VariantAggregate.tasks_measureddescription and did not update the twin field. In src/coder_eval/models/results.py:1178-1187,RunSummary.tasks_measuredstill has the comment "Symmetric with VariantAggregate.tasks_measured so the three published rates read one input" and the description "Rows carrying a criteria verdict". Now the variant field counts TASKS that have at least one verdict-carrying replicate, and the run field counts replicate ROWS. With repeats the two numbers differ. Example: 1 task with 3 scored replicates gives run.json 3 and experiment.json 1. Only the gate is the same. docs/REPORT_SCHEMA.md:49 (run.jsontasks_measured) has the same wording. Either update both sides to state the unit, or remove the symmetry claim. (trigger: src/coder_eval/models/experiment.py)
Tests:
- 🟡 No test has more than one variant.
_task_rowhardcodesvariant_id="v", so the variant half of the(task_id, variant_id)key inmeasured_keyshas no test. A mutation to a task-only key passes the full suite. (trigger: tests/test_ungraded_reporting.py) (restates: Axis 3: The new per-(task, variant) verdict key is tested only with a single variant) - 🔵 No test checks how the new state renders: a variant with
tasks_error > 0andaverage_score=None. No test asserts that experiment.md's**Average Score**(reports/experiment.py:459,format_score(None)) and the html_score_pill(None)cells (reports/html.py:1656/1707) show '—' for the execute plus crash rows. No test checks that the Win Rates section agrees with them. The cross-surface parity test checks onlytasks_measuredandpass_rateon the models. (trigger: tests/test_ungraded_reporting.py) (restates: Axis 3: The run.json/experiment.json parity test checks tasks_measured and pass_rate but not the newly gated average_score) - 🔵 No test covers the crash plus ungraded-only edge case that the PR leaves open. A variant whose only tasks fold ERR+NG to ERROR still reports 0.0. Record it in a test, with xfail(strict) or a pinned current value, so that a fix or a regression shows in the suite. (trigger: tests/test_ungraded_reporting.py) (restates: Axis 3: The documented known edge (a crashed replicate plus ungraded replicates only, which still reports 0.0) has no test)
Harness & Lint Improvements
Static checks (lint / type):
- [ce-lint] CE072 (next free number: CE069/CE070 are taken on feat/harness-central-enforcement, CE071 is in use) is a cyclomatic-complexity ratchet. Make it a whole-tree
@pytest.mark.lintclassTestCE072ComplexityRatchetin tests/test_custom_lint.py, with the logic in a new tests/lint/complexity_ratchet.py. radon is already a runtime dependency. The rule runsradon.complexity.cc_visitover src/coder_eval/ and compares each function's CC with a checked-in baseline, tests/lint/complexity_baseline.json, keyed bymodule:qualname. It fails when (a) a function's CC is higher than its baseline, or (b) a function not in the baseline is above C (CC > 20). A lower value tightens the baseline through amake complexity-baselineregeneration target, so the cap never sits above the current value. Ruff C901 does not fit here: 51 functions are already rank D or worse, so a globalmax-complexityneeds 51 noqa lines and still lets an F function grow. Blind spot: complexity moved into a new private helper that stays at C or below passes, which is the intended fix. Prevents: Finding 1, the only one this would have failed: aggregate_results grows from E(38) to F(42) in src/coder_eval/orchestration/experiment.py:855 because the PR puts the measured_keys pass, the tasks_measured sum and the average_score conditional inline. The rule would also have pushed the PR to extract the per-variant block. Finding 8's notes give that extraction as the way to remove the +4 CC. - [ce-lint] CE073 is an AST rule in tests/lint/rules/ce073_measured_gate_in_models.py, wired in tests/lint/runner.py. It forbids any call to
nothing_was_measured(outsidesrc/coder_eval/models/. A producer or renderer must read a gated property (pass_rate, a gatedaverage_score) and must not build the gate itself. To comply, VariantAggregate.average_score becomes a gated value inside the model: either a validator that rejects a non-None stored value when nothing_was_measured holds, or a stored raw mean behind a gated computed_field, as pass_rate already does. The existing calls in reports/markdown.py (lines 481 and 1078) move behind a RunSummary/CriterionSummary property in the same change. Do not add an allowlist (delete before you guard). Blind spot: code that copies the gate's logic inline, e.g.not_graded and not measured, without calling the helper passes. Prevents: Finding 2 (the gated-mean rule written at two consumer sites, orchestration/experiment.py:957 and reports/markdown.py:481). Finding 4 (VariantAggregate.average_score gated only by its producer, so a loaded or hand-built aggregate can hold average_score=0.0 with pass_rate=None). Part of finding 10 (the field description says tasks_measured 'Gates pass_rate and average_score' when the model applies only one of the two gates). With the gate in the model, the description becomes true. - [ce-lint] Extend CE030 (tests/lint/doc_schema_parity.py) to the run-record contract. Register (VariantAggregate, 'docs/REPORT_SCHEMA.md') and (RunSummary, 'docs/REPORT_SCHEMA.md') in DOCUMENTED_MODELS. Walk
model_computed_fieldsas well asmodel_fields, because a@computed_fieldis serialized into variant.json/experiment.json/run.json and is part of the cross-repo contract that eval-runner reads. Match per section: each model maps to a heading anchor, e.g. '### variant.json — VariantAggregate', and a field passes only if it shows as inline code inside that section. Otherwisepass_rate, which the run.json table documents, satisfies VariantAggregate as well, which is the existing shared-name blind spot. Update the CLAUDE.md CE030 tracked-model list in the same change, because a test keeps that list in sync. Prevents: Finding 11. The variant.json field list in docs/REPORT_SCHEMA.md (lines 292-300) does not list the serialized computed fieldspass_rateandtasks_graded, but the PR's new sentence refers to pass_rate.
Harness improvements (not statically reachable):
- Add a cross-surface parity property test, e.g. tests/test_measured_parity.py with hypothesis or a seeded exhaustive grid. Generate random TaskResult sets with 1-3 variants, 1-3 tasks, 1-3 replicates, and a status for each replicate from SUCCESS/FAILURE/ERROR/TIMEOUT/NOT_GRADED, with or without a weighted_score. For each variant, assert that build_run_summary over that variant's rows and aggregate_results' VariantAggregate agree on (tasks_measured, tasks_not_graded, pass_rate, average_score is None). Also assert that a task where no variant has a measured replicate gets is_tie=True and does not count toward any variant's win rate. Put any (input, divergence) pair that the notes accept as a known gap in a named
pytest.mark.xfail(strict=True)case, so a silent fix or a silent regression fails the suite. Why not static: Two producers must run over the same rows and their outputs must be compared. The defects are semantic (which fold output counts as a verdict, which key the evidence joins on), not a syntax shape an AST rule can match. Prevents: Finding 5: the per-(task, variant) key is tested with one variant only. A key on task_id alone passes the suite but fails this test as soon as there are two variants. Finding 6: average_score is not compared across surfaces. Finding 7 and its merged A6 finding: the ERR+NG replicate mix, where run.json gives None and experiment.json gives 0.0, is unlisted. Finding 8: best_variant/is_tie/win rate come from the folded 0.0 while the aggregate says nothing was measured. Finding 9: a documented edge has no test. - Run scoped mutation testing on scoring and aggregation code in CI, or as a
make mutate-scoringtarget that the review workflow calls. Run mutmut or cosmic-ray on orchestration/experiment.py (aggregate_results, _fold_replicates), models/experiment.py (VariantAggregate gates) and models/results.py (nothing_was_measured, RunSummary rates), with a surviving-mutant budget of 0 for those functions. The reviewer found finding 5 by hand with one mutation (key[0]andv.task_id in measured_keys). A mutation run makes that check automatic. Why not static: Mutation testing runs the test suite against changed code at runtime. It measures whether the assertions are strong enough, and no linter can see that. Prevents: Finding 5, plus findings 6 and 9 when a mutant that drops the average_score gate or changes the ERR+NG fold survives. - Make the notes' known-gap claims verifiable. Each 'One known gap remains' or 'known edge' bullet in .claude/notes/*.md must cite a test node id (
tests/...::test_name[case]). A lint check (in test_custom_lint.py, next to the docs-budget checks) confirms that each cited node exists and is marked xfail(strict=True) or asserts the documented value. Only the presence and existence of the citation can be checked mechanically. Whether the gap list is complete still needs the parity test above. Why not static: A linter can check that a cited test exists. It cannot check that the prose lists every real gap, or that it says the right thing about which file holds which value, e.g. 'as in run.json' for average_score. That needs the runtime parity test, or a human reading the text. Prevents: Finding 9: the documented crash-plus-ungraded edge has no test. Finding 7 and its merged A6 finding: the notes claim the experiment.json/run.json divergence is fixed while the ERR+NG mix still differs. Finding 3: the parity claim names run.json for average_score, but run.json does not store one. Findings 3 and 10 (stale comments on average_score's None rule) are prose drift that static analysis cannot reach. Only the CE073 model-level gate reduces how many comments must restate the rule.
Top 5 Priority Actions
- Put the verdict evidence on the fold in src/coder_eval/orchestration/experiment.py:894: give
VariantResulta measured flag set in_fold_replicates(or return weighted_score=None when no replicate has a verdict), so that best_variant / is_tie / score_spread (L904) and the Win Rates in reports/experiment.py:288 and reports/html.py:1469 agree with the gated pass_rate / average_score; this also deletes themeasured_keysside set and the +4 CC (F 42) this PR adds toaggregate_results. - Count
tasks_not_gradedfrom the replicates and not from the fold in src/coder_eval/orchestration/experiment.py:942, because a task with ERROR+NOT_GRADED replicates folds to ERROR and gives pass_rate=0.0 / average_score=0.0 in experiment.json while run.json gives None; if you defer the fix, add this case to the known-gap list in .claude/notes/orchestration.md:938 and pin it with a strict xfail at tests/test_ungraded_reporting.py:152. - Add a two-variant case to tests/test_ungraded_reporting.py:177 (let
_task_rowtake avariant_id; a/t0 SUCCESS 1.0, b/t0 ERROR, b/t1 NOT_GRADED) and assert that variant b has tasks_measured==0, pass_rate None and average_score None; at present, a mutation that keys on task_id alone passes the full suite. - Make the average_score gate a model invariant in src/coder_eval/models/experiment.py:235: add a
Field(description=...)and an after-validator that rejects a non-None average_score whennothing_was_measuredholds (or add onemeasured_meanhelper in result_metrics.py and use it in experiment.py:957 and reports/markdown.py:481), then delete the stale None-rule comments at experiment.py:956 and models/experiment.py:235. - Close the doc and test drift: assert
aggregate.average_score is Noneand no 'Avg Reliability Score' line in the parity test at tests/test_ungraded_reporting.py:190, addpass_rateandtasks_gradedto the variant.json field list in docs/REPORT_SCHEMA.md:299, and correct the wording in .claude/notes/orchestration.md:940, because run.json has no average_score field.
Stats: 0 🔴 · 0 🟠 · 3 🟡 · 8 🔵 across 8 axes reviewed.
…t variant wins no task The first commit gated VariantAggregate on a side set joined back by (task, variant) key, while the task winner and spread still read the fold's 0.0. An execute run with a crash then showed "-" for the variant's score and pass rate but still gave it a task win. - _fold_replicates sets VariantResult.measured. A variant where nothing_was_measured holds loses its per-task scores and per_replicate_scores, so it wins no task and sets no spread. Graded runs keep the errored 0.0. - result_metrics.measured_mean holds the gated mean, read by VariantAggregate.average_score and run.md's Avg Reliability Score. - aggregate_results splits into _summarize_task and _aggregate_variant (radon C 13, was E 38 on main). - Field contracts, REPORT_SCHEMA.md and the notes name the gate and the unit of each tasks_measured. The crash-plus-ungraded fold edge is listed as a known gap and pinned by a strict xfail.
|
Thanks, both 🟡s were right. Pushed dedddf3:
Not here: the CC ratchet, CE073, the CE030 extension, the parity property test and mutation testing all sound useful, but they're bigger than this fix. #265 now calls the same |
aggregate_resultscountedVariantAggregate.tasks_measuredfrom the foldedVariantResult.weighted_score, and_fold_replicatesturns an errored replicate's missing score into 0.0. So a crash looked like a verdict. The 10-taskexecuterun with one crash that thepass_ratedocstring describes still gavepass_rate0.0 andaverage_score0.0 in experiment.json ("Pass Rate: 0.0% (0/1)" in experiment.md), while run.json gaveNonefor the same rows. This is the experiment.json gap from the #257 review, recorded in.claude/notes/orchestration.md.tasks_measurednow counts tasks where at least one replicate carries a verdict, read from the replicates before the fold.average_scoreisNonewhennothing_was_measuredholds, so it agrees withpass_rateand with run.md.No change for graded runs: an all-ERROR run still reports 0.0 for both. One edge stays: a task whose replicates are a crash plus ungraded runs folds to ERROR (
_pick_worst_statuspicks the worst), so a variant made only of such tasks has no ungraded task and still shows 0.0. That's the fold's status rule, so I left it alone.Tests: the existing parity test builds
tasks_measured=0by hand, so it never reachedaggregate_results. The new ones go through it: a parametrized test (execute with a crash, a crashed replicate beside an ungraded one, one scored replicate measuring its task, all-ERROR still 0.0) and a test that run.json and experiment.json agree on(tasks_measured, pass_rate)for the execute-with-a-crash run. 4 of these fail on main.make verifypasses.This and the suite.json PR both edit the same paragraph in the orchestration notes, so whichever lands second needs a small merge there. I'll take care of it.
Heads up for nightlies: on graded runs, experiment.json's
tasks_measureddrops by the number of errored tasks (10 tasks with 2 ERROR: 10 → 8);pass_rateandaverage_scoredon't change.VariantResultgainsmeasured, and a variant with no verdict now hasnullper-task scores and emptyper_replicate_scores.