Skip to content

Tests: compare the token allocation variants round by round, and run them alone - #144

Merged
electricessence merged 2 commits into
masterfrom
test/token-alloc-paired
Nov 16, 2026
Merged

electricessence merged 2 commits into
masterfrom
test/token-alloc-paired

Conversation

@electricessence

@electricessence electricessence commented Nov 15, 2026 •

Copy link
Copy Markdown
Collaborator

What this changes

TokenTerminalTests.Allocation_* compared four token variants by independent "least of 7 rounds" figures against a fixed slack of 16 bytes per call. On the Debug net11.0 leg of CI the process-wide allocation level moves between rounds, so the test failed in about 1 run in 3 on identical code, with the failing shape changing from run to run.

  • Paired differences. Each round runs the four variants back to back, rotating which goes first, and compares each variant with command-only of the same round. The test passes when the median of 41 paired differences is at most Slack = 16 bytes per call.
  • Where the numbers come from. A CI diagnostic run (a probe that printed every round's figures) showed the paired difference has no bias (mean about -2 B) and a per-round standard deviation of 14.5 B on CI. The median of 41 rounds then has a standard deviation of about 2.8 B, so a 16 B slack is about 5.6 standard deviations away from a false failure. The slack is two thirds of 24 B, the smallest object a call can add on a 64-bit runtime, so one extra object per call sits about 2.8 standard deviations over the limit. The derivation is in the comment above Slack.
  • What it detects. One extra object per call (24 B on x64). It does not claim to see anything smaller: the earlier 9 B claim from the first version of this change is withdrawn.
  • Run alone. TokenTerminalTests joins the existing "Heap measurements" non-parallel collection, so no other test class allocates at the same time.
  • Still catches a real cost. With an extra allocation injected on the call-token path (in a temporary copy of the test, not committed), the test fails for every one of the 25 shapes at 16, 24, 32 and 48 bytes per call (5 of 5 runs each), and an injection of 8 bytes per call (a 24 B object every third call) passes all 25 (5 of 5 runs).
  • Stable locally. 10 consecutive full-suite runs in Debug on net11.0 and 10 on net10.0 (10,200 tests each), no failure.
  • Time. The Allocation_ tests take about 5 s instead of about 2 s locally (+2.8 s).

Only Tests/ changes: no change to Source/, docs or build props.

Checks

  • Release build of both target frameworks with 0 warnings; the format check passes.
  • The full test gate and a Visual Studio MSBuild of the solution are green.
  • Leak scan clean.
  • Still to do before merge: the CI Debug legs green on three runs of the final head.

🤖 Generated with Claude Code

electricessence and others added 2 commits November 15, 2026 11:12
…them alone

The allocation tests that check a call token costs what the command token costs
failed about one CI run in three, on the Debug leg only. The code under test is
identical for both variants, so it was the measure: each variant was reduced to
its own least of seven rounds, and the cost of a call moves with the state of the
process (JIT tiering and recompiles) by tens of bytes, for all variants together,
so each variant's least came from a different round and the gap between them was
the jitter of a round (up to about 10 bytes locally, 14 to 23 on CI), more than
the 16 byte slack.

The four variants still take turns call by call, but now the rounds are kept
(15 instead of 7), the first call of a step also rotates with the round, and each
variant is compared with the command-token-only variant of the same round. The
median of the 15 paired differences must be within 8 bytes per call. Measured
over 16 runs of the whole suite, the medians were between -1 and +2.3 bytes while
single rounds ranged over -18 to +21. An allocation planted on the call-token path
is detected from 9 bytes per call (all 25 tests failed in 5 of 5 runs), where
the old test passed at 16 bytes and failed only from 20. The test class also
joins the existing collection that runs alone, so no other test class loads the
JIT and the GC while it measures (single rounds then ranged over -5 to +6 bytes
instead of -18 to +21); the whole suite took the same time with it (median 63 s against 66 s over 8
runs each). Waiting for two successive warm-up rounds to agree was tried and did
not reduce the jitter (medians up to 2.2 bytes against 1.8), so it is not in.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
…tests

A diagnostic on the CI Debug leg showed that the per-call level there is
about 590 bytes (392 locally) and the same cold and settled, so tiering is not
the cause: the excess is async state machine boxes that depend on how the work
was scheduled, the same for all variants. The paired differences against the
command-token-only variant have no bias (mean about -2 bytes) but a standard
deviation of 14.5 bytes per round, so the median of 15 rounds (standard
deviation 4.7 bytes, slack 8) could still fail by chance, about 4.5 percent per
comparison.

The test now keeps 41 rounds, which brings the standard deviation of the median
to 1.2533 * 14.5 / sqrt(41) = 2.84 bytes, and the slack is 16 bytes, two thirds
of the 24 bytes of the smallest object a call can add on x64: a false failure
is 5.6 standard deviations away, and one extra object per call is 2.8 standard
deviations above the slack. The test detects one extra object per call (24 B on
x64). The earlier claim of detecting 9 bytes per call is withdrawn: it held on
a quiet machine only, and the slack was too tight for CI. The derivation is in
the comment of the slack. The comments no longer name JIT tiering as the cause.
The pairing, the rotation and the collection that runs the class alone are
unchanged.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
@electricessence
electricessence merged commit 09754da into master Nov 16, 2026
17 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant