Repository navigation
Tests: compare the token allocation variants round by round, and run them alone - #144
Merged
Merged
Conversation
…them alone The allocation tests that check a call token costs what the command token costs failed about one CI run in three, on the Debug leg only. The code under test is identical for both variants, so it was the measure: each variant was reduced to its own least of seven rounds, and the cost of a call moves with the state of the process (JIT tiering and recompiles) by tens of bytes, for all variants together, so each variant's least came from a different round and the gap between them was the jitter of a round (up to about 10 bytes locally, 14 to 23 on CI), more than the 16 byte slack. The four variants still take turns call by call, but now the rounds are kept (15 instead of 7), the first call of a step also rotates with the round, and each variant is compared with the command-token-only variant of the same round. The median of the 15 paired differences must be within 8 bytes per call. Measured over 16 runs of the whole suite, the medians were between -1 and +2.3 bytes while single rounds ranged over -18 to +21. An allocation planted on the call-token path is detected from 9 bytes per call (all 25 tests failed in 5 of 5 runs), where the old test passed at 16 bytes and failed only from 20. The test class also joins the existing collection that runs alone, so no other test class loads the JIT and the GC while it measures (single rounds then ranged over -5 to +6 bytes instead of -18 to +21); the whole suite took the same time with it (median 63 s against 66 s over 8 runs each). Waiting for two successive warm-up rounds to agree was tried and did not reduce the jitter (medians up to 2.2 bytes against 1.8), so it is not in. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
…tests A diagnostic on the CI Debug leg showed that the per-call level there is about 590 bytes (392 locally) and the same cold and settled, so tiering is not the cause: the excess is async state machine boxes that depend on how the work was scheduled, the same for all variants. The paired differences against the command-token-only variant have no bias (mean about -2 bytes) but a standard deviation of 14.5 bytes per round, so the median of 15 rounds (standard deviation 4.7 bytes, slack 8) could still fail by chance, about 4.5 percent per comparison. The test now keeps 41 rounds, which brings the standard deviation of the median to 1.2533 * 14.5 / sqrt(41) = 2.84 bytes, and the slack is 16 bytes, two thirds of the 24 bytes of the smallest object a call can add on x64: a false failure is 5.6 standard deviations away, and one extra object per call is 2.8 standard deviations above the slack. The test detects one extra object per call (24 B on x64). The earlier claim of detecting 9 bytes per call is withdrawn: it held on a quiet machine only, and the slack was too tight for CI. The derivation is in the comment of the slack. The comments no longer name JIT tiering as the cause. The pairing, the rotation and the collection that runs the class alone are unchanged. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What this changes
TokenTerminalTests.Allocation_*compared four token variants by independent "least of 7 rounds" figures against a fixed slack of 16 bytes per call. On the Debug net11.0 leg of CI the process-wide allocation level moves between rounds, so the test failed in about 1 run in 3 on identical code, with the failing shape changing from run to run.Slack= 16 bytes per call.Slack.TokenTerminalTestsjoins the existing "Heap measurements" non-parallel collection, so no other test class allocates at the same time.Allocation_tests take about 5 s instead of about 2 s locally (+2.8 s).Only
Tests/changes: no change toSource/, docs or build props.Checks
🤖 Generated with Claude Code