Repository navigation
Conversation
A hash aggregate reserves memory for its group values only after it interns a batch. When the batch makes the group hash table grow, hashbrown allocates the new table before it frees the old one, so the real heap briefly holds both. The memory pool never sees that peak. This adds two `GroupValues` methods that bound the bytes above `size()` that interning may hold at its peak. The bound comes from the current capacity, not from history: if the table can fit the new groups it returns 0, otherwise each doubling counts old plus new storage. - `max_intern_growth(num_rows)` bounds interning `num_rows` rows, all assumed to be new groups. - `max_intern_growth_for(cols)` bounds interning exactly `cols`. The default calls `max_intern_growth` with the row count. `GroupValuesPrimitive` returns 0 when the batch fits the free capacity, and otherwise counts only the keys that are not stored yet. So a full table that sees only existing keys does not reserve a growth that does not happen. - `growth.rs` holds the `hash_table_growth` and `vec_growth` helpers. Its tests compare the bound with the measured old + new allocation sizes. - `GroupValuesRows` and `GroupValuesColumn` bound their hash table. Their value buffers grow by data width, which is not known before interning. Nothing calls these methods yet. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The Partial, PartialReduce, Final and Single hash aggregate streams aggregated a batch and then resized their reservation. If the batch grew the group hash table, the old and new tables were both allocated before the pool could refuse. The spill or early emit then ran after the memory was already used. With many partitions growing at once, the process went well past the pool limit. Each stream now evaluates a batch once and reserves `memory_size() + max_batch_growth(&evaluated)` before it aggregates the evaluated batch. The growth comes from `max_intern_growth_for`, so it is non-zero only when the batch can grow the table. If the pool refuses and the table holds groups, the stream spills (Final, Single) or emits early (Partial, PartialReduce) first. After the batch, it shrinks the reservation to the actual size as before. Final and Single also reserve the spill sort headroom for the groups that the batch can add. If the pool refuses and spilling is disabled, the stream goes on, and the check after the batch decides. `reservation_covers_group_table_growth` counts the heap per thread and checks that the pool peak covers it while the table doubles. Without this change, the Single case fails: heap peak 2004319 bytes against a reserved peak of 1572976 bytes. The call sequence snapshots gain the pre-batch `try_grow` and `shrink` calls, and `spills_twice_then_replays` needs 40 KB instead of 39 KB to keep its two spills. Accumulator state is not covered: `GroupsAccumulator` does not report its capacity. The clustered streams and the legacy `GroupedHashAggregateStream` are unchanged. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…reserve-before-group-growth # Conflicts: # datafusion/physical-plan/src/aggregates/hash_stream.rs
The doc linked to `Self::max_batch_growth`, which the fix commit removed, so `cargo doc -D warnings` failed. The doc now states the contract on its own. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Which issue does this PR close?
Rationale for this change
A high-cardinality
GROUP BYcan use far more memory than its memory limit, and under a hard memory cap the process gets OOM-killed.The hash aggregate streams reserve memory for their group table only after they aggregate a batch. When a batch makes the group hash table grow, hashbrown allocates the new table before it frees the old one, so the heap briefly holds both. The pool never sees that peak. The stream spills afterwards, but the memory is already allocated. With several partitions growing at once, the process goes well past the pool limit.
MRE in stock
datafusion-cli(Linux, 8 partitions,--memory-limit 2g,FairSpillPool):sudo systemd-run --scope -p MemoryMax=1200M -p MemorySwapMax=0 --uid=$(id -u) \ /usr/bin/time -v datafusion-cli --memory-limit 2g --mem-pool-type fair -q -f q1.sqlMemoryMaxmainWith a counting allocator, the heap peak of this query on
mainis about 1.9× the pool's tracked peak (1016 MB against 530 MB). With this PR, it is about 0.95×.What changes are included in this PR?
The PR has two commits, and each one builds and passes tests on its own.
38a5c47:GroupValues::max_intern_growthandmax_intern_growth_forreturn an upper bound on the bytes abovesize()that interning may hold at its peak. Both the old and the new storage count while a buffer grows. The bound comes from the current capacity, so it is 0 when the batch fits.GroupValuesPrimitivealso counts only the keys that are not stored yet. So a full table that sees only existing keys reserves nothing. Nothing calls these methods yet.191bc17: the Partial, PartialReduce, Final and Single streams evaluate a batch, reservememory_size() + growth, and then aggregate it. If the pool refuses, they spill or emit early first. After the batch, they shrink the reservation to the actual size, as before. If spilling is disabled, a refused pre-check does not fail the query. The check after the batch decides.Review tip: start with
group_values/growth.rs, which has the model and its ASCII diagram, thenAggregateHashTable::max_evaluated_growth, then the four stream call sites.Not covered:
GroupsAccumulatordoes not report its capacity.GroupValuesRowsandGroupValuesColumn. Only their hash tables are bounded.GroupedHashAggregateStream.What is the testing strategy for this PR?
growth.rsunit tests compare the bound with the measured old + new allocation sizes while a table and aVecgrow. The bound is exact where hashbrown uses SSE2 groups, and at most 16 bytes high elsewhere.intern_growth_for_existing_keys_is_zerofills a table to capacity. Existing keys give a bound of 0, and one new key gives exactly one doubling.reservation_covers_group_table_growthcounts heap per thread and checks that the pool peak covers the doubling. Onmainit fails: heap peak 2004319 bytes against a reserved peak of 1572976 bytes.aggregate_memory_call_sequencesnapshots gain the pre-batchtry_growandshrinkcalls.spills_twice_then_replaysneeds 40 KB instead of 39 KB to keep its two spills.Performance: no criterion benches yet. In the worst case, where every batch sits within one batch of a table doubling (1 partition, 29.4 M groups), wall time is 4.1 s against 3.7 to 3.8 s on
main. On the MRE above, wall time is unchanged. I will run the aggregate benches before this leaves draft.Are there any user-facing changes?
Queries with a high-cardinality
GROUP BYstay closer to the memory limit. They may spill slightly earlier.GroupValuesis a public trait. It gainsmax_intern_growthandmax_intern_growth_for, both with default implementations, so this is not a breaking change.🤖 Generated with Claude Code