Skip to content

fix(orchestration): count experiment.json's tasks_measured from the replicates - #266

Open
JYunth wants to merge 2 commits into
UiPath:mainfrom
JYunth:fix/experiment-measured-from-replicates
Open

JYunth wants to merge 2 commits into
UiPath:mainfrom
JYunth:fix/experiment-measured-from-replicates

Conversation

@JYunth

@JYunth JYunth commented Nov 21, 2026 •

Copy link
Copy Markdown
Contributor

aggregate_results counted VariantAggregate.tasks_measured from the folded VariantResult.weighted_score, and _fold_replicates turns an errored replicate's missing score into 0.0. So a crash looked like a verdict. The 10-task execute run with one crash that the pass_rate docstring describes still gave pass_rate 0.0 and average_score 0.0 in experiment.json ("Pass Rate: 0.0% (0/1)" in experiment.md), while run.json gave None for the same rows. This is the experiment.json gap from the #257 review, recorded in .claude/notes/orchestration.md.

  • tasks_measured now counts tasks where at least one replicate carries a verdict, read from the replicates before the fold.
  • average_score is None when nothing_was_measured holds, so it agrees with pass_rate and with run.md.
  • The field description, REPORT_SCHEMA.md's variant.json entry and the notes are updated.

No change for graded runs: an all-ERROR run still reports 0.0 for both. One edge stays: a task whose replicates are a crash plus ungraded runs folds to ERROR (_pick_worst_status picks the worst), so a variant made only of such tasks has no ungraded task and still shows 0.0. That's the fold's status rule, so I left it alone.

Tests: the existing parity test builds tasks_measured=0 by hand, so it never reached aggregate_results. The new ones go through it: a parametrized test (execute with a crash, a crashed replicate beside an ungraded one, one scored replicate measuring its task, all-ERROR still 0.0) and a test that run.json and experiment.json agree on (tasks_measured, pass_rate) for the execute-with-a-crash run. 4 of these fail on main. make verify passes.

This and the suite.json PR both edit the same paragraph in the orchestration notes, so whichever lands second needs a small merge there. I'll take care of it.

Heads up for nightlies: on graded runs, experiment.json's tasks_measured drops by the number of errored tasks (10 tasks with 2 ERROR: 10 → 8); pass_rate and average_score don't change. VariantResult gains measured, and a variant with no verdict now has null per-task scores and empty per_replicate_scores.

…eplicates

aggregate_results counted VariantAggregate.tasks_measured from the folded
VariantResult score, and _fold_replicates turns an errored replicate's
missing score into 0.0. So one crash in an `execute` run counted as a
verdict: experiment.json reported pass_rate 0.0 and average_score 0.0
where run.json reported None for the same rows.

tasks_measured now counts tasks with at least one replicate that carries
a verdict, and average_score is None when nothing_was_measured holds,
as pass_rate is.

@uipreliga uipreliga left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Review: coder_eval — pr:266

Scope: pr:266 · branch fix/experiment-measured-from-replicates · c62d28c · 2026-11-21T07:24Z · workflow variant

Change class: complex — changes how experiment.json counts tasks_measured and when average_score is null, i.e. persisted metric/classification semantics

PR #266 is sound: the nothing-measured gate on VariantAggregate works, security and error handling have no findings, and no finding changes a task's own score or final_status, but the gate reads a separate key-joined measured_keys set and not the per-task fold, so experiment.json and experiment.md can still name a winner and show 0.0 rates for a variant that has no verdict; fix that and add a two-variant test before you merge.

Summary

Axis Score 🔴 🟠 🟡 🔵 Top Issue
1. Code Quality & Style 9.3 / 10 0 0 1 2 aggregate_results grows to CC 42 (F) from 38 (E); the PR adds more inline per-variant logic instead of extracting it
2. Type Safety 9.9 / 10 0 0 0 1 The VariantAggregate field descriptions (average_score, tasks_measured) describe a gate that the model does not apply; only the producer applies it
3. Test Health 9.3 / 10 0 0 1 2 The new per-(task, variant) verdict key is tested only with a single variant; a regression that keys on task_id alone passes the full suite
4. Security 10 / 10 0 0 0 0 —
5. Architecture & Design 9.5 / 10 0 0 1 0 Nothing-measured gate is applied at VariantAggregate but not at the per-task fold, so experiment.json (and experiment.md) contradict themselves for a crashed variant
6. Error Handling & Resilience 10 / 10 0 0 0 0 —
7. API Surface & Maintainability 9.7 / 10 0 0 0 3 The notes claim the experiment.json/run.json crash divergence is fixed, but a crashed replicate plus an ungraded replicate still folds to ERROR and reports 0.0
8. Evaluation Harness Quality 10 / 10 0 0 0 0 —

Overall Score: 9.7 / 10 · Weakest Axis: Code Quality & Style at 9.3 / 10
Totals: 🔴 0 · 🟠 0 · 🟡 3 · 🔵 8 across 8 axes.

Blockers

None.

Non-blocking, but please consider before merge

  1. [Axis 1] aggregate_results grows to CC 42 (F) from 38 (E); the PR adds more inline per-variant logic instead of extracting it (src/coder_eval/orchestration/experiment.py:855) — radon at PR head reports F 855:0 aggregate_results - F (42). On main it was E (38). The PR adds three new branches inline: the measured_keys = { key for key, reps in task_variant_reps.items() if any(r.result.weighted_score is not None for r in reps) } pass (line 894), tasks_measured = sum(1 for v in vr_list if (v.task_id, vid) in measured_keys) (line 945), and the average_score=(None if nothing_was_measured(...) else _mean_graded_score(vr_list)) conditional (lines 957-961). The module already records why this function should shrink: _fold_replicates says it was "Extracted from aggregate_results, which was already the largest function in the module" (line 828). The PR goes the other way. Extract the per-variant block (lines 925-965) into a helper, e.g. _aggregate_variant(vid, vr_list, reps_by_key) -> VariantAggregate, and count tasks_measured inside it straight from the replicates (sum(1 for (t, v), reps in reps_by_key.items() if v == vid and any(r.result.weighted_score is not None for r in reps))). That also removes the separate measured_keys set and the tuple lookup. orchestration/experiment.py is not a hot module, so this is filed at Medium.
  2. [Axis 3] The new per-(task, variant) verdict key is tested only with a single variant; a regression that keys on task_id alone passes the full suite (tests/test_ungraded_reporting.py:177) — The new aggregate_results logic keys verdict evidence on the (task, variant) pair: measured_keys = { key for key, reps in task_variant_reps.items() if any(r.result.weighted_score is not None for r in reps) } (src/coder_eval/orchestration/experiment.py:894-895) and tasks_measured = sum(1 for v in vr_list if (v.task_id, vid) in measured_keys) (:945). Every new test calls aggregate_results("e", "", ["v"], rows, 1.0) (line 177 and 188), and _task_row hardcodes variant_id="v", so the variant part of the key is never tested. I made a mutation in /tmp/pr-266-worktree: I changed the set to key[0] for key ... and the lookup to if v.task_id in measured_keys. With that mutation the routed subset still gave 366 passed, 3 skipped. The full non-live suite failed only the same 10 test_judge_litellm.py tests (a missing litellm module) that also fail without the mutation. So nothing in the repo catches it. With the mutation, in an A/B where variant A scores t0 and variant B only crashes or runs ungraded on t0, B counts as measured. Then B's pass_rate and average_score become 0.0 where they should be None. That is the exact bug this PR fixes, now leaking between variants. Fix: add a parametrized case with two variants. Let _task_row take a variant_id argument. Example rows: t0 OK 1.0 in variant a; t0 ERR plus t1 NG in variant b. Call aggregate_results(..., ["a", "b"], ...) and assert variant_aggregates["b"].tasks_measured == 0, .pass_rate is None and .average_score is None, and that a stays measured.
  3. [Axis 5] Nothing-measured gate is applied at VariantAggregate but not at the per-task fold, so experiment.json (and experiment.md) contradict themselves for a crashed variant (src/coder_eval/orchestration/experiment.py:894) — The PR states that an errored replicate's 0.0 from _fold_replicates is not a verdict. It applies that rule to only one consumer of the fold. It does not carry the evidence on the folded VariantResult. Instead it adds a separate set that is joined back by key: measured_keys = { key for key, reps in task_variant_reps.items() if any(r.result.weighted_score is not None for r in reps) } (L894) and tasks_measured = sum(1 for v in vr_list if (v.task_id, vid) in measured_keys) (L945). The other consumer of the same fold in the same function still reads the folded 0.0 as a score: scored = [(v, v.weighted_score) for v in variants if v.weighted_score is not None] (L904) sets best_variant, is_tie and score_spread. reports/experiment.py:288 and reports/html.py:1469 count win rates from those values. I ran this on the PR head: execute run, variants A and B, t0 = A ERROR / B NOT_GRADED, t1 = both NOT_GRADED. Result: both aggregates give tasks_measured=0, pass_rate=None, average_score=None, but t0 has best_variant='A', is_tie=False. So experiment.md shows Average Score and Pass Rate as '—' for A, and also 'A: 1/2 tasks (50%)' under Win Rates. Before the PR the two halves agreed, because both treated 0.0 as a score. The PR introduces this divergence. The edited notes paragraph (.claude/notes/orchestration.md:938-947) now says 'an execute run with a crash has no pass_rate or average_score' just before 'Only graded variants can win or set a spread', and the code no longer makes the second statement true. Fix: put the evidence on the fold. Give VariantResult a measured flag (or measured-replicate count) set in _fold_replicates, or make _fold_replicates return weighted_score=None when no replicate has a verdict. Then the aggregate, the winner/spread selection, and the renderers all read one field, and the key-joined side set goes away. This also removes the +4 CC this PR adds to aggregate_results (radon F(42), was E(38)).

Nits

  1. [Axis 1] The gated measured-mean rule is now written at two consumer sites, but the notes say the rule lives once (src/coder_eval/orchestration/experiment.py:957) — The new average_score=(None if nothing_was_measured(not_graded=tasks_not_graded, measured=tasks_measured) else _mean_graded_score(vr_list)) repeats the composition already in reports/markdown.py:481: if scores and not nothing_was_measured(not_graded=summary.tasks_not_graded, measured=summary.tasks_measured):. The edited notes paragraph still says "The rule lives once, in result_metrics.measured_scores". Now the mean rule and the gate are put together separately at each call site. VariantAggregate's own pass_rate also applies this gate inside the model, but average_score is gated only by its one constructor. Put the gated mean into one helper in result_metrics.py, e.g. measured_mean(rows, *, not_graded, measured) -> float | None, and call it from both sites. Alternatively, enforce average_score is None under nothing_was_measured in a VariantAggregate validator, so the model cannot hold the inconsistent state.
  2. [Axis 1] The notes paragraph says run.json has no average_score, but run.json does not store an average score (.claude/notes/orchestration.md:940) — Lines 940-941 say "so an execute run with a crash has no pass_rate or average_score in experiment.json, as in run.json." run.json has no average-score field. docs/REPORT_SCHEMA.md:74 says "run.md's Avg Reliability Score is not stored in run.json". Write it as "...in experiment.json, as run.json has no pass_rate and run.md shows no Avg Reliability Score", so the parity claim names the surface that holds each value.
  3. [Axis 2] The VariantAggregate field descriptions (average_score, tasks_measured) describe a gate that the model does not apply; only the producer applies it (src/coder_eval/models/experiment.py:235) — The PR changes when average_score is None. The aggregator now writes None if nothing_was_measured(not_graded=tasks_not_graded, measured=tasks_measured) else _mean_graded_score(vr_list) (orchestration/experiment.py:957-961). So 9 NOT_GRADED rows plus 1 ERROR row (1 graded row, 0 measured) now gives None. But the contract on the persisted field still reads # None when nothing in this variant was graded: a 0.0 is indistinguishable from / # "measured and scored zero". / average_score: float | None (lines 235-237). The field has no Field(description=...), and the inline comment at orchestration/experiment.py:956 (# GRADED rows only, and None when there are none.) has the same staleness. The PR updated the tasks_measured description to say it 'Gates pass_rate and average_score', but the gated field's own contract was not updated. The model also does not enforce the gate. pass_rate is a computed_field that calls nothing_was_measured itself, but average_score is a plain stored field. A VariantAggregate built anywhere else, or loaded from experiment.json, can carry average_score=0.0 with tasks_measured=0, tasks_not_graded>0. The model accepts this even though pass_rate on the same object is None. Fix: change line 237 to average_score: float | None = Field(description="Mean over measured rows (errored = 0.0, ungraded leaves). None when no row is measured or nothing_was_measured holds.") and delete the stale comments. Optionally, also add a check to _check_task_count_invariant (or a sibling after-validator) that rejects a non-None average_score when nothing_was_measured(...) holds. Then the model holds the invariant that pass_rate already holds.
  4. [Axis 3] The run.json/experiment.json parity test checks tasks_measured and pass_rate but not the newly gated average_score (tests/test_ungraded_reporting.py:190) — The PR claims that average_score is now None when nothing_was_measured holds, so that it agrees with pass_rate and with run.md's Avg Reliability Score. But the cross-surface test asserts only assert (aggregate.tasks_measured, aggregate.pass_rate) == (summary.tasks_measured, summary.pass_rate) == (0, None). The parametrized test checks average_score on the aggregate alone, so no test compares the two surfaces on the execute-plus-crash rows. Add assert aggregate.average_score is None to this test. Also add an assertion that ReportGenerator.generate_markdown(summary) has no Avg Reliability Score line (markdown.py:481 hides it under the same nothing_was_measured gate). This follows the existing test_run_md_and_the_variant_average_publish_one_mean_for_one_set_of_rows. Then the agreement claim holds on all three surfaces.
  5. [Axis 3] The documented known edge (a crashed replicate plus ungraded replicates only, which still reports 0.0) has no test (tests/test_ungraded_reporting.py:152) — Case a-crashed-replicate-is-not-a-verdict ([_task_row("t0", _ERR), _task_row("t0", _NG, replicate_index=1), _task_row("t1", _NG)]) gets None only because the separate task t1 adds to tasks_not_graded. The PR text and the orchestration notes accept an edge case: a variant made ONLY of tasks whose replicates are a crash plus ungraded runs. _pick_worst_status folds each such task to ERROR, so tasks_not_graded stays 0 and pass_rate and average_score show 0.0. No test pins this. A later fix, or a silent change in that behavior, will not show in the suite. Add a case with only [_task_row("t0", _ERR), _task_row("t0", _NG, replicate_index=1)]. Mark it pytest.mark.xfail(strict=True, reason=...) with the target of None, or assert the current 0.0 with a reason that points to the notes. Then the gap is recorded in a test, not only in prose.
  6. [Axis 7] The notes claim the experiment.json/run.json crash divergence is fixed, but a crashed replicate plus an ungraded replicate still folds to ERROR and reports 0.0 (.claude/notes/orchestration.md:938) — Lines 938-941 now say: "VariantAggregate.tasks_measured is counted from the replicates ... so an execute run with a crash has no pass_rate or average_score in experiment.json, as in run.json. One known gap remains." The only gap listed is suite.json. The PR description admits a second gap, and I reproduced it at PR head. Under execute with repeats, give each task one ERROR replicate and one NOT_GRADED replicate (t0: ERR+NG, t1: ERR+NG). build_run_summary then reports tasks_measured=0, tasks_not_graded=2, pass_rate=None. aggregate_results reports tasks_measured=0, tasks_not_graded=0, pass_rate=0.0, average_score=0.0. The cause: tasks_measured is now read from the replicates (experiment.py:893 measured_keys), but the other input to the gate, tasks_not_graded = sum(1 for v in vr_list if v.final_status.category == "ungraded") (experiment.py:942), is still read from the fold. _pick_worst_status folds ERR+NG to ERROR. So nothing_was_measured sees not_graded=0. Either (a) add this case to the notes' known-gap list, so the cross-repo consumer (eval-runner reads experiment.json) knows that the two files can still disagree, or (b) close the gap the same way as measured_keys: compute 'has an ungraded replicate' from task_variant_reps and pass that count to the nothing_was_measured gate. If you choose (b), add the case to the parity test in tests/test_ungraded_reporting.py.
  7. [Axis 7] The comments on average_score's None rule in aggregate_results are stale after the gate changed (src/coder_eval/orchestration/experiment.py:956) — The PR adds a second None condition to average_score: None if nothing_was_measured(not_graded=tasks_not_graded, measured=tasks_measured) else _mean_graded_score(vr_list). This condition also nulls a variant that has ERROR-bucket rows, which are graded by category. But the inline comment directly above it (line 956) still says # GRADED rows only, and None when there are none.. In the touched model file, src/coder_eval/models/experiment.py:235, the comment # None when nothing in this variant was graded: a 0.0 is indistinguishable from ... was also not updated. Change both to 'None when no row carries a verdict (nothing_was_measured) or every row is ungraded'. Also, the model's field description at models/experiment.py:261 says tasks_measured "Gates pass_rate and average_score", but the model computes only the pass_rate gate. The average_score gate is applied only by the producer, aggregate_results. Name that producer in the description, or delete the comment and let the description carry the contract. The pass_rate docstring warns against exactly this kind of 'claim rather than fact' mirror.
  8. [Axis 7] The variant.json field list mentions pass_rate but does not list the serialized pass_rate / tasks_graded fields (docs/REPORT_SCHEMA.md:299) — The new text says tasks_measured "gates pass_rate and average_score as RunSummary.tasks_measured gates the run's rate". But the variant.json — VariantAggregate field list (lines 292-300) still does not list pass_rate or tasks_graded. Both are @computed_fields on VariantAggregate, so agg.model_dump_json (reports/experiment.py:598) serializes them into variant.json and experiment.json. The run.json table documents both (lines 65 and 67). This PR is the edit that makes the doc point at pass_rate, so add pass_rate (float | None, None when nothing_was_measured or tasks_graded==0) and tasks_graded (tasks_run - tasks_not_graded) to this list. Then a downstream consumer can find the gated field that the sentence names.

What's Missing

Nightly pipeline:

  • 🟡 The PR changes the value of a serialized experiment.json field on the graded run path, which is the nightly path, and does not say so. Before, tasks_measured counted folded rows whose score was not None, and the fold scores an errored task 0.0, so an errored task counted. Now an errored task does not count. Example: a graded nightly with 10 tasks, 2 of them ERROR, published tasks_measured=10 and now publishes 8. pass_rate and average_score do not change, because nothing_was_measured needs not_graded > 0. The description says only that graded all-ERROR runs "still report 0.0". The notes also say a change to a serialized field that downstream consumers read must be its own change. So the PR description should state the value change for eval-runner and other experiment.json readers. (trigger: src/coder_eval/orchestration/experiment.py)

Downstream consumers:

  • 🟡 The new rule "a crashed replicate is not a verdict" reaches only the per-variant aggregate. Other consumers of the same fold still read the 0.0: task_summaries best_variant / is_tie / score_spread, the Win Rates in reports/experiment.py:282-292 and reports/html.py:1463-1472, and each per-task VariantResult.weighted_score cell. For an execute plus crash run, experiment.json and experiment.md show '—' for the variant's Average Score and Pass Rate, but give the same variant a win and a 0.000 score cell. (trigger: src/coder_eval/orchestration/experiment.py) (restates: Axis 5: Nothing-measured gate is applied at VariantAggregate but not at the per-task fold)
  • 🔵 Two places in docs/REPORT_SCHEMA.md still describe the old rule. The average_score sentence at lines 294-295 ("an errored row counts as 0.0 ... null when nothing was measured") is now wrong for execute runs: an execute run with a crash is null even though its errored row is "measured" as 0.0 by that rule. The task_summaries text at lines 313-316 says a per-task weighted_score errored replicate counts as 0.0, the "same rule as VariantAggregate.average_score above". After this PR the two rules are no longer the same. Update both so experiment.json consumers see the new gate. (trigger: docs/REPORT_SCHEMA.md) (restates: Axis 2: The VariantAggregate field descriptions (average_score, tasks_measured) describe a gate that the model does not apply)
  • 🔵 In .claude/notes/orchestration.md, the paragraph after the edited one is now stale. Lines 954-957 say "tasks_measured counts rows with a score" and "average_score is None rather than 0.000 when nothing was graded". Both now contradict the replicate-based count and the nothing_was_measured gate, which the PR added a few lines above. Update the paragraph, or merge it into the edited one. (trigger: .claude/notes/orchestration.md) (restates: Axis 7: The comments on average_score's None rule in aggregate_results are stale after the gate changed)

Parallel paths:

  • 🔵 The PR moved only one of the two inputs of the nothing_was_measured gate from the fold to the replicates. tasks_measured now comes from task_variant_reps. tasks_not_graded still counts folded final_status == ungraded, and _pick_worst_status turns ERR+NG into ERROR. So a crash plus ungraded replicates still diverges from run.json, where build_run_summary counts raw replicate rows. (trigger: src/coder_eval/orchestration/experiment.py) (restates: Axis 7: The notes claim the experiment.json/run.json crash divergence is fixed)
  • 🔵 The PR edited the VariantAggregate.tasks_measured description and did not update the twin field. In src/coder_eval/models/results.py:1178-1187, RunSummary.tasks_measured still has the comment "Symmetric with VariantAggregate.tasks_measured so the three published rates read one input" and the description "Rows carrying a criteria verdict". Now the variant field counts TASKS that have at least one verdict-carrying replicate, and the run field counts replicate ROWS. With repeats the two numbers differ. Example: 1 task with 3 scored replicates gives run.json 3 and experiment.json 1. Only the gate is the same. docs/REPORT_SCHEMA.md:49 (run.json tasks_measured) has the same wording. Either update both sides to state the unit, or remove the symmetry claim. (trigger: src/coder_eval/models/experiment.py)

Tests:

  • 🟡 No test has more than one variant. _task_row hardcodes variant_id="v", so the variant half of the (task_id, variant_id) key in measured_keys has no test. A mutation to a task-only key passes the full suite. (trigger: tests/test_ungraded_reporting.py) (restates: Axis 3: The new per-(task, variant) verdict key is tested only with a single variant)
  • 🔵 No test checks how the new state renders: a variant with tasks_error > 0 and average_score=None. No test asserts that experiment.md's **Average Score** (reports/experiment.py:459, format_score(None)) and the html _score_pill(None) cells (reports/html.py:1656/1707) show '—' for the execute plus crash rows. No test checks that the Win Rates section agrees with them. The cross-surface parity test checks only tasks_measured and pass_rate on the models. (trigger: tests/test_ungraded_reporting.py) (restates: Axis 3: The run.json/experiment.json parity test checks tasks_measured and pass_rate but not the newly gated average_score)
  • 🔵 No test covers the crash plus ungraded-only edge case that the PR leaves open. A variant whose only tasks fold ERR+NG to ERROR still reports 0.0. Record it in a test, with xfail(strict) or a pinned current value, so that a fix or a regression shows in the suite. (trigger: tests/test_ungraded_reporting.py) (restates: Axis 3: The documented known edge (a crashed replicate plus ungraded replicates only, which still reports 0.0) has no test)

Harness & Lint Improvements

Static checks (lint / type):

  • [ce-lint] CE072 (next free number: CE069/CE070 are taken on feat/harness-central-enforcement, CE071 is in use) is a cyclomatic-complexity ratchet. Make it a whole-tree @pytest.mark.lint class TestCE072ComplexityRatchet in tests/test_custom_lint.py, with the logic in a new tests/lint/complexity_ratchet.py. radon is already a runtime dependency. The rule runs radon.complexity.cc_visit over src/coder_eval/ and compares each function's CC with a checked-in baseline, tests/lint/complexity_baseline.json, keyed by module:qualname. It fails when (a) a function's CC is higher than its baseline, or (b) a function not in the baseline is above C (CC > 20). A lower value tightens the baseline through a make complexity-baseline regeneration target, so the cap never sits above the current value. Ruff C901 does not fit here: 51 functions are already rank D or worse, so a global max-complexity needs 51 noqa lines and still lets an F function grow. Blind spot: complexity moved into a new private helper that stays at C or below passes, which is the intended fix. Prevents: Finding 1, the only one this would have failed: aggregate_results grows from E(38) to F(42) in src/coder_eval/orchestration/experiment.py:855 because the PR puts the measured_keys pass, the tasks_measured sum and the average_score conditional inline. The rule would also have pushed the PR to extract the per-variant block. Finding 8's notes give that extraction as the way to remove the +4 CC.
  • [ce-lint] CE073 is an AST rule in tests/lint/rules/ce073_measured_gate_in_models.py, wired in tests/lint/runner.py. It forbids any call to nothing_was_measured( outside src/coder_eval/models/. A producer or renderer must read a gated property (pass_rate, a gated average_score) and must not build the gate itself. To comply, VariantAggregate.average_score becomes a gated value inside the model: either a validator that rejects a non-None stored value when nothing_was_measured holds, or a stored raw mean behind a gated computed_field, as pass_rate already does. The existing calls in reports/markdown.py (lines 481 and 1078) move behind a RunSummary/CriterionSummary property in the same change. Do not add an allowlist (delete before you guard). Blind spot: code that copies the gate's logic inline, e.g. not_graded and not measured, without calling the helper passes. Prevents: Finding 2 (the gated-mean rule written at two consumer sites, orchestration/experiment.py:957 and reports/markdown.py:481). Finding 4 (VariantAggregate.average_score gated only by its producer, so a loaded or hand-built aggregate can hold average_score=0.0 with pass_rate=None). Part of finding 10 (the field description says tasks_measured 'Gates pass_rate and average_score' when the model applies only one of the two gates). With the gate in the model, the description becomes true.
  • [ce-lint] Extend CE030 (tests/lint/doc_schema_parity.py) to the run-record contract. Register (VariantAggregate, 'docs/REPORT_SCHEMA.md') and (RunSummary, 'docs/REPORT_SCHEMA.md') in DOCUMENTED_MODELS. Walk model_computed_fields as well as model_fields, because a @computed_field is serialized into variant.json/experiment.json/run.json and is part of the cross-repo contract that eval-runner reads. Match per section: each model maps to a heading anchor, e.g. '### variant.json — VariantAggregate', and a field passes only if it shows as inline code inside that section. Otherwise pass_rate, which the run.json table documents, satisfies VariantAggregate as well, which is the existing shared-name blind spot. Update the CLAUDE.md CE030 tracked-model list in the same change, because a test keeps that list in sync. Prevents: Finding 11. The variant.json field list in docs/REPORT_SCHEMA.md (lines 292-300) does not list the serialized computed fields pass_rate and tasks_graded, but the PR's new sentence refers to pass_rate.

Harness improvements (not statically reachable):

  • Add a cross-surface parity property test, e.g. tests/test_measured_parity.py with hypothesis or a seeded exhaustive grid. Generate random TaskResult sets with 1-3 variants, 1-3 tasks, 1-3 replicates, and a status for each replicate from SUCCESS/FAILURE/ERROR/TIMEOUT/NOT_GRADED, with or without a weighted_score. For each variant, assert that build_run_summary over that variant's rows and aggregate_results' VariantAggregate agree on (tasks_measured, tasks_not_graded, pass_rate, average_score is None). Also assert that a task where no variant has a measured replicate gets is_tie=True and does not count toward any variant's win rate. Put any (input, divergence) pair that the notes accept as a known gap in a named pytest.mark.xfail(strict=True) case, so a silent fix or a silent regression fails the suite. Why not static: Two producers must run over the same rows and their outputs must be compared. The defects are semantic (which fold output counts as a verdict, which key the evidence joins on), not a syntax shape an AST rule can match. Prevents: Finding 5: the per-(task, variant) key is tested with one variant only. A key on task_id alone passes the suite but fails this test as soon as there are two variants. Finding 6: average_score is not compared across surfaces. Finding 7 and its merged A6 finding: the ERR+NG replicate mix, where run.json gives None and experiment.json gives 0.0, is unlisted. Finding 8: best_variant/is_tie/win rate come from the folded 0.0 while the aggregate says nothing was measured. Finding 9: a documented edge has no test.
  • Run scoped mutation testing on scoring and aggregation code in CI, or as a make mutate-scoring target that the review workflow calls. Run mutmut or cosmic-ray on orchestration/experiment.py (aggregate_results, _fold_replicates), models/experiment.py (VariantAggregate gates) and models/results.py (nothing_was_measured, RunSummary rates), with a surviving-mutant budget of 0 for those functions. The reviewer found finding 5 by hand with one mutation (key[0] and v.task_id in measured_keys). A mutation run makes that check automatic. Why not static: Mutation testing runs the test suite against changed code at runtime. It measures whether the assertions are strong enough, and no linter can see that. Prevents: Finding 5, plus findings 6 and 9 when a mutant that drops the average_score gate or changes the ERR+NG fold survives.
  • Make the notes' known-gap claims verifiable. Each 'One known gap remains' or 'known edge' bullet in .claude/notes/*.md must cite a test node id (tests/...::test_name[case]). A lint check (in test_custom_lint.py, next to the docs-budget checks) confirms that each cited node exists and is marked xfail(strict=True) or asserts the documented value. Only the presence and existence of the citation can be checked mechanically. Whether the gap list is complete still needs the parity test above. Why not static: A linter can check that a cited test exists. It cannot check that the prose lists every real gap, or that it says the right thing about which file holds which value, e.g. 'as in run.json' for average_score. That needs the runtime parity test, or a human reading the text. Prevents: Finding 9: the documented crash-plus-ungraded edge has no test. Finding 7 and its merged A6 finding: the notes claim the experiment.json/run.json divergence is fixed while the ERR+NG mix still differs. Finding 3: the parity claim names run.json for average_score, but run.json does not store one. Findings 3 and 10 (stale comments on average_score's None rule) are prose drift that static analysis cannot reach. Only the CE073 model-level gate reduces how many comments must restate the rule.

Top 5 Priority Actions

  1. Put the verdict evidence on the fold in src/coder_eval/orchestration/experiment.py:894: give VariantResult a measured flag set in _fold_replicates (or return weighted_score=None when no replicate has a verdict), so that best_variant / is_tie / score_spread (L904) and the Win Rates in reports/experiment.py:288 and reports/html.py:1469 agree with the gated pass_rate / average_score; this also deletes the measured_keys side set and the +4 CC (F 42) this PR adds to aggregate_results.
  2. Count tasks_not_graded from the replicates and not from the fold in src/coder_eval/orchestration/experiment.py:942, because a task with ERROR+NOT_GRADED replicates folds to ERROR and gives pass_rate=0.0 / average_score=0.0 in experiment.json while run.json gives None; if you defer the fix, add this case to the known-gap list in .claude/notes/orchestration.md:938 and pin it with a strict xfail at tests/test_ungraded_reporting.py:152.
  3. Add a two-variant case to tests/test_ungraded_reporting.py:177 (let _task_row take a variant_id; a/t0 SUCCESS 1.0, b/t0 ERROR, b/t1 NOT_GRADED) and assert that variant b has tasks_measured==0, pass_rate None and average_score None; at present, a mutation that keys on task_id alone passes the full suite.
  4. Make the average_score gate a model invariant in src/coder_eval/models/experiment.py:235: add a Field(description=...) and an after-validator that rejects a non-None average_score when nothing_was_measured holds (or add one measured_mean helper in result_metrics.py and use it in experiment.py:957 and reports/markdown.py:481), then delete the stale None-rule comments at experiment.py:956 and models/experiment.py:235.
  5. Close the doc and test drift: assert aggregate.average_score is None and no 'Avg Reliability Score' line in the parity test at tests/test_ungraded_reporting.py:190, add pass_rate and tasks_graded to the variant.json field list in docs/REPORT_SCHEMA.md:299, and correct the wording in .claude/notes/orchestration.md:940, because run.json has no average_score field.

Stats: 0 🔴 · 0 🟠 · 3 🟡 · 8 🔵 across 8 axes reviewed.

…t variant wins no task

The first commit gated VariantAggregate on a side set joined back by
(task, variant) key, while the task winner and spread still read the
fold's 0.0. An execute run with a crash then showed "-" for the
variant's score and pass rate but still gave it a task win.

- _fold_replicates sets VariantResult.measured. A variant where
  nothing_was_measured holds loses its per-task scores and
  per_replicate_scores, so it wins no task and sets no spread. Graded
  runs keep the errored 0.0.
- result_metrics.measured_mean holds the gated mean, read by
  VariantAggregate.average_score and run.md's Avg Reliability Score.
- aggregate_results splits into _summarize_task and _aggregate_variant
  (radon C 13, was E 38 on main).
- Field contracts, REPORT_SCHEMA.md and the notes name the gate and the
  unit of each tasks_measured. The crash-plus-ungraded fold edge is
  listed as a known gap and pinned by a strict xfail.
@JYunth

JYunth commented Nov 21, 2026

Copy link
Copy Markdown
Contributor Author

Thanks, both 🟡s were right. Pushed dedddf3:

  • The evidence is on the fold now: _fold_replicates sets VariantResult.measured, and measured_keys is gone. A variant where nothing_was_measured holds also drops its per-task scores and per_replicate_scores, so your case (t0 = A ERROR / B NOT_GRADED, t1 both NOT_GRADED) now gives t0 a tie and "A: 0/2" under Win Rates. Graded runs are unchanged: an errored task still scores 0.0 and can lose, and a test pins that.
  • Two-variant test with your rows, plus the winner case above.
  • aggregate_results is split into _summarize_task and _aggregate_variant; radon has it at C (13).
  • The gated mean lives once in result_metrics.measured_mean, read by VariantAggregate.average_score and run.md. average_score has a Field(description=...) naming the gate and where it's applied, and the stale comments are gone. I didn't add a validator; happy to if you want the model to enforce it.
  • The parity test also checks average_score and that run.md has no Avg Reliability Score line.
  • ERROR plus NOT_GRADED replicates only: I took option (a). It's in the notes' known gaps and pinned with a strict xfail. Closing it means changing _pick_worst_status or pass_rate's gate input, so I'd rather do it separately.
  • Docs: variant.json lists pass_rate, tasks_graded and tasks_measured; run.json's tasks_measured says it counts rows and experiment.json's counts tasks; the task_summaries text covers measured and the winner rule; the notes wording and the stale paragraph are fixed.
  • Added the nightly change to the description: on graded runs tasks_measured drops by the number of errored tasks, and pass_rate and average_score don't change.

Not here: the CC ratchet, CE073, the CE030 extension, the parity property test and mutation testing all sound useful, but they're bigger than this fix. #265 now calls the same measured_mean, so whichever lands second just drops #265's xfail for the variant-mean gap.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants