Skip to content

fix(reports): count unscored rows as 0.0 in suite.json's average_weighted_score - #265

Open
JYunth wants to merge 2 commits into
UiPath:mainfrom
JYunth:fix/suite-json-avg-score-counts-errors
Open

JYunth wants to merge 2 commits into
UiPath:mainfrom
JYunth:fix/suite-json-avg-score-counts-errors

Conversation

@JYunth

@JYunth JYunth commented Nov 21, 2026 •

Copy link
Copy Markdown
Contributor

suite.json's average_weighted_score only averaged rows that had a weighted_score, so ERROR, BUILD_FAILED and unscored TIMEOUT rows left the mean. A suite with one SUCCESS (1.0), one FAILURE (0.0) and one ERROR reported 0.500, where run.md and experiment.json report 0.333 for the same rows. This is the suite.json gap left out of #257 and recorded in .claude/notes/orchestration.md.

  • _compute_suite_rollup now uses result_metrics.measured_scores: NOT_GRADED rows leave the mean, any other unscored row counts as 0.0.
  • The score is None when nothing_was_measured holds, the same gate as run.md's score and pass_rate. An execute suite with a crashed row shows no score instead of 0.000.
  • REPORT_SCHEMA.md now defines the field, and the notes drop this gap.

Heads up: suite.json's average_weighted_score and suite.md's "Average weighted score" go down for suites with errored rows. passed and the CI exit code don't read it, so gates don't change.

Tests: a parametrized case in test_suite_rollup.py (ERROR, BUILD_FAILED and unscored TIMEOUT count as 0.0, NOT_GRADED leaves, all-ERROR gives 0.0, all-NOT_GRADED and execute-with-a-crash give None), and a parity test that suite.json and VariantAggregate.average_score give 0.25 for the same five rows. test_mixed_outcomes now expects 0.467 instead of 0.7. test_omits_avg_score_when_none uses a NOT_GRADED row, since an all-ERROR suite now scores 0.0. 6 of these fail on main. make verify passes.

This and the experiment.json PR both edit the same paragraph in the orchestration notes, so whichever lands second needs a small merge there. I'll take care of it.

For nightly readers of suite.json: a suite of only errored rows now reports average_weighted_score 0.0 instead of null, and any suite with errored rows steps down once at the merge date. Neither is an agent regression.

…hted_score

SuiteRollup.average_weighted_score averaged only the rows with a score,
so an ERROR, BUILD_FAILED or unscored TIMEOUT row left the mean and a
suite with errors scored higher than a clean one. It now uses
measured_scores, the rule run.md's Avg Reliability Score and
VariantAggregate.average_score already use: ungraded rows leave the
mean, any other unscored row counts as 0.0. It is None when
nothing_was_measured holds, as pass_rate is.

@uipreliga uipreliga left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Review: coder_eval — pr:265 (5 files) axis:1,2,3,4,5,6,7,8

Scope: pr:265 (5 files) axis:1,2,3,4,5,6,7,8 · branch fix/suite-json-avg-score-counts-errors (PR #265 head) · c7397c7 · 2026-11-21T08:36Z · workflow variant

Change class: complex — changes the formula of a serialized suite.json metric (average_weighted_score) consumed downstream; correctness needs reasoning over status classification

PR #265 correctly makes suite.json's average_weighted_score count unscored ERROR and TIMEOUT rows as 0.0, using the shared measured_scores filter, and it has no type, security, error-handling or architecture defects; the real risks are a REPORT_SCHEMA.md row that tells downstream consumers to expect null for an all-ERROR suite that now publishes 0.0, and a gate-plus-mean expression copied into a second report surface, which is the same per-surface drift that caused this bug; bottom line: merge after you correct the doc row, and add the shared helper and the two missing tests as follow-ups or in this PR.

Summary

Axis Score 🔴 🟠 🟡 🔵 Top Issue
1. Code Quality & Style 9.3 / 10 0 0 1 2 Gated suite headline-mean composition duplicated inline in markdown.py (second copy of run.md gate-plus-mean; nothing_was_measured called twice)
2. Type Safety 10 / 10 0 0 0 0 —
3. Test Health 9.8 / 10 0 0 0 2 suite.md now shows a 0.000 average for an all-errored suite, but no render test checks it; the only render test for this behaviour was moved to NOT_GRADED
4. Security 10 / 10 0 0 0 0 —
5. Architecture & Design 10 / 10 0 0 0 0 —
6. Error Handling & Resilience 10 / 10 0 0 0 0 —
7. API Surface & Maintainability 10 / 10 0 0 0 0 —
8. Evaluation Harness Quality 9.9 / 10 0 0 0 1 REPORT_SCHEMA null condition for suite.json average_weighted_score contradicts the all-ERROR (0.0) case

Overall Score: 9.9 / 10 · Weakest Axis: Code Quality & Style at 9.3 / 10
Totals: 🔴 0 · 🟠 0 · 🟡 1 · 🔵 5 across 8 axes.

Blockers

None.

Non-blocking, but please consider before merge

  1. [Axis 1] Gated suite headline-mean composition duplicated inline in markdown.py (second copy of run.md gate-plus-mean; nothing_was_measured called twice) (src/coder_eval/reports/markdown.py:962-967) — The PR makes suite.json use the same rule as run.md by pasting run.md's expression a second time. run.md, markdown.py:474/481: scores = measured_scores(...) / if scores and not nothing_was_measured(not_graded=summary.tasks_not_graded, measured=summary.tasks_measured): then sum(scores) / len(scores). suite.json, markdown.py:962-967: scores = measured_scores((r.result.final_status, r.result.weighted_score) for r in rows) / average_weighted_score = (sum(scores) / len(scores) if scores and not nothing_was_measured(not_graded=rows_not_graded, measured=rows_measured) else None). The notes now say the rule "lives once" (orchestration.md:936). That is true of the filter, but the visibility gate and the mean are written out at each surface. This is the same per-surface drift that caused the bug being fixed. The same function also calls the predicate again at markdown.py:1084 (if nothing_was_measured(not_graded=rows_not_graded, measured=rows_measured)). The routed radon result shows _compute_suite_rollup going from CC 38 to CC 40 (grade E). Fix: add one measured_mean(rows, *, not_graded, measured) -> float | None to result_metrics.py next to measured_scores, and call it from both report surfaces. In _compute_suite_rollup, compute unmeasured = nothing_was_measured(...) once and use it for both pass_rate and average_weighted_score. Duplication across 2 sites is the Axis 1 Medium anchor.

Nits

  1. [Axis 1] The new local scores has the same name as the comprehension variable a few lines below (src/coder_eval/reports/markdown.py:962) — The PR renames the old local scored to scores (line 962: scores = measured_scores(...)). The per-criterion block at lines 978-986 then reads rows_evaluated=len(scores) and average_score=sum(scores) / len(scores) if scores else 0.0 inside for ctype, scores in sorted(by_type.items()). Comprehension scoping keeps the two names separate, so the behaviour is correct. But a reader sees len(scores) and can take it to mean the suite-level list declared 16 lines above. Rename one of them, e.g. keep suite_scores for the rollup mean, or type_scores in the comprehension.
  2. [Axis 1] The new suite.json parity test copies the previous test's row fixture word for word (tests/test_ungraded_reporting.py:154-164) — test_suite_json_and_the_variant_average_publish_one_mean_for_one_set_of_rows re-declares rows: list[tuple[FinalStatus, float | None]] = [(FinalStatus.SUCCESS, 1.0), (FinalStatus.FAILURE, 0.0), (FinalStatus.ERROR, None), (FinalStatus.TIMEOUT, None), (FinalStatus.NOT_GRADED, None)] and the variant_rows = [VariantResult(...) for i, (status, score) in enumerate(rows)] comprehension. Both are already at lines 121-131 in test_run_md_and_the_variant_average_publish_one_mean_for_one_set_of_rows. Both tests assert the same parity (== _mean_graded_score(variant_rows) == 0.25). Move the fixture to a module-level constant, or merge the two into one test that checks run.md, suite.json and the variant mean against one set of rows. That merged test states the 'one mean, every surface' contract once.
  3. [Axis 3] suite.md now shows a 0.000 average for an all-errored suite, but no render test checks it; the only render test for this behaviour was moved to NOT_GRADED (tests/test_suite_rollup.py:487) — Before this PR, test_omits_avg_score_when_none built a single FinalStatus.ERROR row (with error_message="boom") and checked that suite.md left out the average. The PR changes that row to final_status=FinalStatus.NOT_GRADED, which is right, because an errored suite now scores 0.0. But no test replaced the old one for the new suite.md output. The parametrized case id="all-error-suite-scores-zero" (line 152) checks only the rollup value, and the one test that checks the rendered line (line 388, assert "Average weighted score**:" in md) uses a 2/3 SUCCESS suite with a non-zero mean. If someone changes the render guard at markdown.py:1112 (if rollup.average_weighted_score is not None:) to a truthiness check, the errored-suite 0.000 line disappears again and no test fails. Add a render check next to the NOT_GRADED case: build two FinalStatus.ERROR rows with weighted_score=None, then assert "**Average weighted score**: 0.000" in _render_suite_markdown(rollup). This covers both sides of the new rule: NOT_GRADED hides the average, ERROR shows 0.000.
  4. [Axis 3] The new parametrized test covers only an unscored TIMEOUT; a TIMEOUT that has a score (counts at its own score, not 0.0) is never tested (tests/test_suite_rollup.py:150) — The new REPORT_SCHEMA.md row says a row with no score, including "an unscored TIMEOUT", counts as 0.0. That wording implies a scored TIMEOUT keeps its own weighted_score. Both new tests (pytest.param([*_GRADED_ROWS, ("TIMEOUT", None)], 1 / 3, id="unscored-timeout-counts-as-zero") here, and (FinalStatus.TIMEOUT, None) in tests/test_ungraded_reporting.py:158) use only the unscored form. No test covers the status-category path for a TIMEOUT with a partial score. Add pytest.param([*_GRADED_ROWS, ("TIMEOUT", 0.5)], 0.5, id="scored-timeout-keeps-its-score"). The expected value is (1.0+0.0+0.5)/3 = 0.5. It also shows that rows_measured counts the scored TIMEOUT as evidence.
  5. [Axis 8] REPORT_SCHEMA null condition for suite.json average_weighted_score contradicts the all-ERROR (0.0) case (docs/REPORT_SCHEMA.md:333) — The new row says: "null when no row produced a verdict, as pass_rate is." and "the same rule as run.md's Avg Reliability Score and VariantAggregate.average_score". The gate in reports/markdown.py:963-966 is sum(scores) / len(scores) if scores and not nothing_was_measured(not_graded=rows_not_graded, measured=rows_measured) else None, and nothing_was_measured is not_graded > 0 and measured == 0. So a suite of only ERROR rows, where no row produced a verdict, returns 0.0, not null. The PR's own test pins this: pytest.param([("ERROR", None), ("ERROR", None)], 0.0, id="all-error-suite-scores-zero"). Also, for [NOT_GRADED, ERROR] suite.json gives null but VariantAggregate.average_score (_mean_graded_score) gives 0.0, which I checked by running it. So the 'same rule' claim covers the zeroing rule only, not the null gate. A downstream consumer (the coder-eval-uipath nightly) that reads this contract would expect null for an all-crash suite. Change the wording to the real condition, e.g. "null when the suite has no rows, or when it has NOT_GRADED rows and no row has a score (the shared nothing_was_measured gate). A suite of only errored rows scores 0.0." Also say that VariantAggregate.average_score does not have the null gate yet (the known gap in .claude/notes/orchestration.md). Out of scope for this axis but relevant: the stale SuiteRollup.average_weighted_score field description in models/results.py:971 ("Mean weighted_score across rows that produced one.") belongs to the What's Missing pass.

What's Missing

Downstream consumers:

  • 🟡 The Pydantic field description for SuiteRollup.average_weighted_score (src/coder_eval/models/results.py:970-972) was not updated. It still says "Mean weighted_score across rows that produced one.", which is the old rule this PR removes. The project treats the schema model as the single source of truth for field definitions (CLAUDE.md DRY / SSOT). After this PR, the model contradicts both the new REPORT_SCHEMA.md row and markdown.py:962-967: ERROR/BUILD_FAILED/unscored TIMEOUT rows count as 0.0, NOT_GRADED rows leave the mean, and the value is None when nothing_was_measured holds. Rewrite the description to the real rule, in the same style as the pass_rate description above it. (trigger: src/coder_eval/reports/markdown.py) (restates: Axis 8: REPORT_SCHEMA null condition for suite.json average_weighted_score contradicts the all-ERROR (0.0) case)

Parallel paths:

  • 🔵 suite.json now has the nothing_was_measured null gate, but its sibling VariantAggregate.average_score (_mean_graded_score in orchestration/experiment.py:807) still does not. For one set of rows, e.g. [NOT_GRADED, ERROR], suite.json publishes null and experiment.json publishes 0.0. The new parity test claims "one mean for one set of rows", but its five-row fixture has measured rows, so the two surfaces agree there and the divergence never shows. Add a case with no measured rows that pins the current divergence (or xfail it with a pointer to the known gap in .claude/notes/orchestration.md), so the follow-up experiment.json PR has a test to flip. (trigger: tests/test_ungraded_reporting.py) (restates: Axis 8: REPORT_SCHEMA null condition for suite.json average_weighted_score contradicts the all-ERROR (0.0) case)

Tests:

  • 🔵 test_omits_avg_score_when_none was moved from an ERROR row to a NOT_GRADED row, and nothing replaced the ERROR render case. No test checks that suite.md now shows **Average weighted score**: 0.000 for an all-errored suite. The new 0.0 value is tested only at the rollup level, not at the markdown.py:1112 render guard, so a change from the is not None check to a truthiness check would pass every test. (trigger: tests/test_suite_rollup.py) (restates: Axis 3: suite.md now shows a 0.000 average for an all-errored suite, but no render test checks it)

Nightly pipeline:

  • 🔵 The PR body says the score goes down and that passed/CI exit codes do not change. It does not cover two effects on consumers of the suite.json report shape (the coder-eval-uipath nightly and any dashboards or trend queries over suite.json): (a) for an all-ERROR suite the value changes type, from null to 0.0, so a consumer that read null as 'the suite crashed' now sees a real-looking zero; (b) every suite with errored rows has a one-time step drop in its historical trend at the merge date. State both in the PR description and the release notes, so readers of the nightly trend do not take the drop for an agent regression. (trigger: docs/REPORT_SCHEMA.md)

Harness & Lint Improvements

Static checks (lint / type):

  • [ce-lint] CE072 (next free number after CE071; check CE069/CE070 are not taken on another branch first). First do the fix: add measured_mean(scores: list[float], *, not_graded: int, measured: int) -> float | None to src/coder_eval/result_metrics.py, next to measured_scores. It holds the visibility gate and the mean together: None if not scores or nothing_was_measured(...) else sum(scores) / len(scores). Then the rule: in src/coder_eval/ outside result_metrics.py, a function may not take sum(v) / len(v) (or statistics.mean(v) / fmean(v)) where v is a name bound from measured_scores(...) or from the _measured_scores(...) wrapper. It may also not call nothing_was_measured( more than once. Both are found with an AST walk of one function body: record the names assigned from those calls, then flag a BinOp(Div) whose operands are sum(<name>) and len(<name>), and count the nothing_was_measured calls. Put it at tests/lint/rules/ce072_headline_mean_via_helper.py and add it to tests/lint/runner.py. Blind spots for the docstring: a list that goes through a rename or a helper, and a mean computed by hand in a loop. Prevents: Finding 1 (A1/A5 medium). It would flag the gate-and-mean written out again at reports/markdown.py:962-967 (suite.json) and the existing copy at markdown.py:474/481 (run.md). It would also flag the second nothing_was_measured call at markdown.py:1084 in _compute_suite_rollup. This rule follows the CLAUDE.md order 'delete before you guard': the helper is the fix, and the rule only stops someone from writing the expression inline again. Once orchestration/experiment.py:_mean_graded_score is changed to call measured_mean, the same rule also closes the suite.json-vs-VariantAggregate.average_score null-gate split described in Finding 6. Today experiment.py:807 takes its own mean without the gate.
  • [ce-lint] CE073: extend the idea of CE058 ('unknown must stay None, not 0.0') from timing fields to score and rate fields in the rendering layer. In src/coder_eval/reports/ and src/coder_eval/result_metrics.py, the rule forbids a truthiness test on a name or attribute whose last segment matches ^(pass_rate|average_[a-z_]*score|[a-z_]*_score|[a-z_]*_rate)$. The forbidden forms are if x:, if not x:, x or <default>, and x if x else .... The code must use is None / is not None instead. This is an AST check on If.test, IfExp.test, BoolOp(Or) and UnaryOp(Not). For the file layout, copy CE058: compile one _SCORE_NAME regex and allow # noqa: CE073 where a falsy check is intended. Do not use pyright for this. reportOptionalOperand does not flag truthiness on float | None, and no ruff rule does either. Prevents: Finding 4 (A3 low). The new suite.md line 'Average weighted score: 0.000' for an all-ERROR suite depends on the guard if rollup.average_weighted_score is not None: at markdown.py:1112. If someone changes it to a truthiness check, the line is hidden again and no test fails. The rule makes that change fail make lint, whether or not a render test exists. It also protects the guards beside it on pass_rate, average_score and rate in the reports layer, where 0.0 is a real published value.
  • [ci-gate] Add a report-schema parity test, the same type as CE030 / CE033. For each run-record model that docs/REPORT_SCHEMA.md documents in a | Key | Type | Meaning | table (start with SuiteRollup, CriterionStats, FailedRowSummary, CriterionAggregate), the set of serialized field names must equal the set of keys in the table. The better option is to generate each 'Meaning' cell from the Pydantic Field(description=...) (CLAUDE.md 'DRY: field descriptions defined once'), with a make report-schema target and a drift check, like make plugin-reference / CE033. Then the null condition is written in one place, next to the field. Add it as a @pytest.mark.lint class in tests/test_custom_lint.py, because it reasons over Markdown. Prevents: Finding 6 (A1/A7/A8 low). The REPORT_SCHEMA.md:333 row says null when no row produced a verdict, but code publishes 0.0 for an all-ERROR suite. The model description at models/results.py:971 says 'Mean weighted_score across rows that produced one.', which is also stale. So the field has two prose sources and both have drifted from the code. Generating the row from the model leaves one source to keep correct, and the stale description at models/results.py:971 then shows in the doc diff. A static check cannot tell whether the prose matches the gate in markdown.py:963-966. The cross-surface matrix test in the harness bucket covers that part.

Harness improvements (not statically reachable):

  • Replace the two copied parity tests (tests/test_ungraded_reporting.py:121-131 and 154-164) with one 'one mean, every surface' matrix test. Declare the row mixes once at module level, then parametrize over surfaces × row mixes. The surfaces are run.md Avg Reliability Score, suite.json average_weighted_score, VariantAggregate.average_score via _mean_graded_score, and the rendered suite.md line. The row mixes must include: graded-only; +ERROR(None); +unscored TIMEOUT; +scored TIMEOUT(0.5) (expected (1.0+0.0+0.5)/3 = 0.5, and rows_measured counts it); all-ERROR (expected 0.0, and suite.md shows **Average weighted score**: 0.000); all-NOT_GRADED (expected None, line hidden); [NOT_GRADED, ERROR]. For every mix, the test asserts that every surface publishes the same value, or it records the known divergence as an explicit xfail with a pointer to the gap noted in .claude/notes/orchestration.md. Why not static: The defect is a difference in computed values between surfaces that take different input shapes (task_results dicts, EvaluationResult rows, VariantResult). Only running each surface on the same rows can show that they publish different numbers. An AST rule cannot evaluate the gate. Prevents: Finding 3 (copied fixture), Finding 4 (no render test for all-ERROR 0.000), Finding 5 (scored TIMEOUT never tested), and Finding 6 (the [NOT_GRADED, ERROR] split: suite.json gives None and VariantAggregate gives 0.0, which the doc wrongly calls 'the same rule'). The xfail row puts the known gap on record, so it cannot drift again without notice.
  • Do not add a lint rule for a comprehension target that shadows a local in the enclosing function (scores at markdown.py:962 versus for ctype, scores in sorted(by_type.items()) at 978-986). Fix it with a rename (suite_scores / type_scores). Record the decision in .claude/harness-candidates.md so later reviews do not propose the rule again. Why not static: The pattern can be detected statically, but ruff does not implement pylint's redefined-outer-name for comprehension scopes. A custom rule would flag many legal, readable uses in the codebase. The CLAUDE.md bar is that 'a rule is right when the pattern is genuinely unavoidable', and this pattern only affects readability, with no behaviour change. So the check gives too little value for its noise, and this is recorded as a decision. Prevents: Finding 2 (A1 low) — a review note only, with no gate.
  • Add a section to the review checklist and .claude/notes/orchestration.md § Aggregation: when a PR changes a headline-mean or rate rule, it must list every surface that publishes that number. The surfaces are run.md, suite.json/suite.md, experiment.json VariantAggregate, the evalboard, and reports/junit. For each surface, the PR must say whether that surface changed or keeps a known gap. It must also state the cross-repo effect on the coder-eval-uipath nightly. Examples: suite.json average_weighted_score now goes from null to 0.0 for an all-crash suite, and a consumer that reads the doc as 'null on no verdict' sees a different value. Why not static: The list of surfaces includes consumers outside this repo, and the question 'does this change the nightly's reading of the field' needs judgment. No in-repo check can see those consumers. Prevents: Finding 6 (contract wording) and the per-surface drift class behind Finding 1. The bug this PR fixes is itself an example of that drift.

Top 5 Priority Actions

  1. Change the average_weighted_score row at docs/REPORT_SCHEMA.md:333 to state the real null condition: null only when the suite has no rows, or when it has NOT_GRADED rows and no scored row (nothing_was_measured); a suite of only ERROR rows scores 0.0; VariantAggregate.average_score does not yet apply the null gate.
  2. Add one measured_mean(scores, *, not_graded, measured) -> float | None helper to result_metrics.py next to measured_scores, call it from both run.md (src/coder_eval/reports/markdown.py:474-481) and suite.json (src/coder_eval/reports/markdown.py:962-967), and in _compute_suite_rollup compute nothing_was_measured once so that pass_rate (markdown.py:1084) and the average use the same value.
  3. Add a render test next to tests/test_suite_rollup.py:487 that builds two FinalStatus.ERROR rows with weighted_score=None and asserts "Average weighted score: 0.000" is in _render_suite_markdown(rollup), so that a truthiness change to the guard at markdown.py:1112 fails a test.
  4. Add pytest.param([*_GRADED_ROWS, ("TIMEOUT", 0.5)], 0.5, id="scored-timeout-keeps-its-score") at tests/test_suite_rollup.py:150 to prove that a scored TIMEOUT keeps its own score and counts toward rows_measured.
  5. Rename the suite-level local scores at src/coder_eval/reports/markdown.py:962 (for example to suite_scores) so it does not have the same name as the per-criterion comprehension variable at lines 978-986, and merge the run.md and suite.json parity tests in tests/test_ungraded_reporting.py:121-164 into one test with one shared row fixture.

Stats: 0 🔴 · 0 🟠 · 1 🟡 · 5 🔵 across 8 axes reviewed.

The suite.json average and run.md's score each wrote out the same gate
and mean inline. Both now call result_metrics.measured_mean, which takes
measured_scores' mean and returns None when nothing_was_measured holds.
_compute_suite_rollup is back to its main complexity (radon E 38).

- REPORT_SCHEMA.md and SuiteRollup's field description state the real
  null condition: an all-ERROR suite scores 0.0, and
  VariantAggregate.average_score does not apply the gate yet.
- Tests: a scored TIMEOUT keeps its score; suite.md renders 0.000 for an
  all-errored suite; the run.md and suite.json parity tests share one
  helper; the variant-mean gap is pinned by a strict xfail.
@JYunth

JYunth commented Nov 21, 2026

Copy link
Copy Markdown
Contributor Author

Thanks. Pushed 03e67d0:

  • The gated mean lives in one helper now, result_metrics.measured_mean, and both run.md and suite.json call it. _compute_suite_rollup is back to CC 38 and calls nothing_was_measured once. The shadowed scores is gone with it.
  • REPORT_SCHEMA.md and the SuiteRollup field description state the real rule: null when there are no counted rows or nothing_was_measured holds, 0.0 for a suite of only errored rows, a scored TIMEOUT keeps its score, and VariantAggregate.average_score doesn't apply the gate yet.
  • Tests: the scored-TIMEOUT case, a render test that suite.md shows 0.000 for an all-errored suite, and the two parity tests are now one helper that reads run.md, suite.json and the variant mean from the same rows. The [NOT_GRADED, ERROR] split is a strict xfail; fix(orchestration): count experiment.json's tasks_measured from the replicates #266 closes it, so whichever of the two lands second drops the marker.
  • Added the nightly effects to the description: an all-errored suite now reports 0.0 instead of null, and suites with errored rows step down once at the merge.

Not here: CE072/CE073, the report-schema parity check and the review-checklist section all sound useful, but they're bigger than this fix.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants