Skip to content

fix: reserve incoming aggregate state before merging - #27129

Open
yashdotrv wants to merge 5 commits into
apache:mainfrom
yashdotrv:fix/aggregate-state-reservation
Open

yashdotrv wants to merge 5 commits into
apache:mainfrom
yashdotrv:fix/aggregate-state-reservation

Conversation

@yashdotrv

Copy link
Copy Markdown
Contributor

Which issue does this PR close?

Addresses #26705.

Rationale for this change

A partial aggregate can emit a state batch containing all its input values and then release its memory reservation. The final aggregate merges that batch before reserving memory for it. For aggregates such as median and array_agg, this leaves input-sized state unaccounted for by the memory pool.

What changes are included in this PR?

Final aggregates now reserve incoming state batches before merging them. The hash and clustered final paths can spill their current state and retry when the pool cannot admit a batch. I also updated the affected memory reservation and spill tests.

PR #26728 adds related memory accounting tests, but its cases for this issue are currently ignored.

What is the testing strategy for this PR?

I added reservation tests for the ungrouped, hash grouped, and clustered grouped final aggregate paths. The extended workspace test suite, formatting check, and Clippy pass.

I also ran the issue’s 30-million-row, four-partition median reproducer with a 300 MB memory limit. It now returns ResourcesExhausted when the incoming state cannot be reserved. Peak RSS can still exceed the configured limit; reducing temporary memory use during buffer growth is outside this PR.

Are there any user-facing changes?

Under memory pressure, a final aggregate may now spill or return ResourcesExhausted instead of continuing with unreserved state. There are no public API changes.

@github-actions github-actions Bot added physical-plan Changes to the physical-plan crate sqllogictest SQL Logic Tests (.slt) and removed sqllogictest SQL Logic Tests (.slt) labels Nov 5, 2026
@yashdotrv
yashdotrv force-pushed the fix/aggregate-state-reservation branch from 1c93e8c to 0539bfb Compare November 5, 2026 19:20
@yashdotrv
yashdotrv marked this pull request as ready for review November 5, 2026 19:21
@jayzhan211
jayzhan211 self-requested a review November 7, 2026 02:06

@jayzhan211 jayzhan211 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@yashdotrv thanks for picking this up. Please correct the description, since most of the state is still unreserved and buffer growth is not the main cause. With the ignored #26705 cases in #26728 turned on, unreserved memory grows 121-124 MB for median (122-143 MB on main) and 500-505 MB for array_agg (531-589 MB), against a 16 MB bound. At the peak 19.1 MB of 152 MB is reserved for median, about one of the four partial states; the others appear to be queued in CoalescePartitionsExec.

A follow-up is fine because that queue is outside these streams, so #26705 should stay open. The description should also say that array_agg over 1000 groups at 30-60 MB goes from passing to ResourcesExhausted, while peak RSS at 100 MB stays near 610 MB.

@yashdotrv

Copy link
Copy Markdown
Contributor Author

@jayzhan211 ty for your review, I'll work on the suggestions and get back.

@yashdotrv
yashdotrv force-pushed the fix/aggregate-state-reservation branch from 0539bfb to a7850b7 Compare November 11, 2026 11:10
@yashdotrv

Copy link
Copy Markdown
Contributor Author

@jayzhan211 Thanks again for the review and detailed pointers! I have updated the branch with the requested changes:

  1. Per-Row Charging Threshold: Replaced the fixed 64 KiB threshold with MAX_IN_FLIGHT_STATE_BYTES_PER_ROW = 1024 and MIN_CHARGED_STATE_BATCH_BYTES = 64 * 1024. Normal bounded batches remain exempt, avoiding unnecessary spills/refusals on queries with many aggregates.
  2. Soft Limit Early Break: Kept the early break for soft group limit before try_resize in FinalHashAggregateStream.
  3. Spill Logic Cleanup: Extracted the duplicate spill-and-resize logic in FinalHashAggregateStream into a shared spill_table helper.
  4. Ungrouped Aggregate Guard: Guarded the calculation of incoming state batch memory in AggregateStream strictly to AggregateInputMode::Partial.
  5. Call Sequence Tests Restored: Restored aggregate_memory_call_sequence.rs to match main without edits; all 8 snapshot tests pass.
  6. Test Coverage:
    • Updated final_aggregate_reserves_incoming_state_batch and final_hash_reserves_incoming_state_batch to use 64-row sliced batches retaining multi-megabyte underlying buffers.
    • Added clustered_final_refuses_large_state_batch.
    • Added final_hash_spills_to_admit_large_state_batch.
    • Added clustered_final_spills_to_admit_large_state_batch to test the spill retry path under GroupClusteringMode::Partial(vec![0]).
  7. PR Description: Clarified that Partial aggregate state reaches the final aggregate without a memory reservation (median, array_agg) #26705 remains open for the queued partial states in CoalescePartitionsExec.

@jayzhan211 jayzhan211 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@yashdotrv thanks for the quick turnaround; every point from the first round is addressed. One new problem remains: the charge counts the whole buffer behind a zero-copy slice, so a grouped query without a hash repartition goes from passing to ResourcesExhausted. Details and a fix are inline on incoming_state_batch_reservation_bytes. The fix changes how the new tests should build their large state batch, so I've also asked for one more test there.

yashdotrv and others added 5 commits November 22, 2026 00:25
- Charge partial-state batches only when bytes per row exceed 1 KiB and total bytes exceed 64 KiB
- Restore early break on soft group limit before table memory resize in FinalHashAggregateStream
- Extract shared spill_table helper in FinalHashAggregateStream
- Guard incoming batch memory reservation calculation to Partial input mode in AggregateStream
- Replace synthetic tests with large-buffer retaining batches to properly test state batch admission
- Add test for ClusteredFinalAggregateStream spill-and-retry on partial clustering
- Add test for FinalHashAggregateStream spill-and-retry on large state batch
- Restore aggregate_memory_call_sequence.rs to match main
… aggregate

- Measure only bytes referenced by slice rows using SpilledRowSizes when buffer exceeds in_flight_bytes
- Add unit tests for bounded state batches, shared buffer slices, and Single mode raw input
- Update test cases to use realistic state payloads
@yashdotrv
yashdotrv force-pushed the fix/aggregate-state-reservation branch from b1f2c29 to b8d1065 Compare November 21, 2026 18:59
@yashdotrv

Copy link
Copy Markdown
Contributor Author

@yashdotrv thanks for the quick turnaround; every point from the first round is addressed. One new problem remains: the charge counts the whole buffer behind a zero-copy slice, so a grouped query without a hash repartition goes from passing to ResourcesExhausted. Details and a fix are inline on incoming_state_batch_reservation_bytes. The fix changes how the new tests should build their large state batch, so I've also asked for one more test there.

@jayzhan211 Thanks for catching the zero-copy buffer slicing issue and for the detailed suggestions!

I have updated the PR and rebased against the latest main:

  1. Zero-Copy Slice Buffer Accounting:

    • In incoming_state_batch_reservation_bytes, added a fast check buffer_bytes <= in_flight_bytes.
    • When buffer_bytes > in_flight_bytes, we now measure the memory actually referenced by the slice's rows using SpilledRowSizes::new(batch). Slices of larger batches (e.g., when repartition_aggregations = false) will not double-charge the shared parent buffer that the producer already tracks.
  2. Realistic Test Payloads:

    • Updated existing unit tests to use payloads that genuinely exceed the per-row threshold (e.g. ListArray with 131k elements for array_agg, and large Utf8 strings).
  3. New Test Coverage:

    • final_hash_does_not_charge_buffers_shared_by_slices: verifies that 8,192-row slices from a 1M-group batch (16 MB buffer) are not over-charged and run cleanly.
    • final_aggregate_does_not_charge_bounded_state_batch: pins the per-row rule by asserting an 8,192-row batch of two Int64 states (128 KB) in a 96 KB pool is not charged.
    • single_aggregate_does_not_charge_raw_input: verifies that ungrouped Single mode over raw input (64 rows of 16 KB strings = 1 MB in a 512 KB pool) does not charge raw input batches.

@jayzhan211 jayzhan211 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@yashdotrv thanks for adopting every point from the last round. The 500k-group query without a hash repartition now passes at 40–80 MB again, and the 30M-row median at 300 MB is still refused. Two tests no longer reach the paths they are named for; the fixes are inline.

Optional: the Rationale opens with the mechanism, so a reviewer can't see the symptom from its first paragraph, and "What changes" doesn't mention the 1 KiB/row rule, which is the part that decides which batches are charged. Something like:

SELECT median(v) FROM generate_series(1, 30000000) with 4 target partitions and a 300 MB memory limit runs past the limit (656–939 MB peak RSS in #26705): the partial aggregate releases its reservation when it emits its state, and the final aggregate reserves memory only after merging it. This PR reserves each incoming state batch before merging it, so that query now returns ResourcesExhausted.

A batch is charged only when its rows reference more than 1 KiB each on average (and 64 KiB in total), so ordinary batch_size-bounded state batches are not charged. Slices of a larger partial output are charged only for the bytes their rows reference.

)?);
// Fits the large batch, but not next to the table of 4000 groups.
let runtime = RuntimeEnvBuilder::new()
.with_memory_pool(Arc::new(GreedyMemoryPool::new(2600 * 1024)))

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@yashdotrv thanks for the update; the slice fix and the new tests look good. With the new row sizing, this test no longer reaches the spill-and-retry path: the table holds 432 KB and the batch charge is 2.05 MB, so both fit in 2600 KB, and removing the retry passes every test. At 2400 KB the batch no longer fits beside the table, so the retry runs, and the test fails without it:

Suggested change
.with_memory_pool(Arc::new(GreedyMemoryPool::new(2600 * 1024)))
.with_memory_pool(Arc::new(GreedyMemoryPool::new(2400 * 1024)))


// Fits the large batch, but not next to the table of 4000 groups.
let runtime = RuntimeEnvBuilder::new()
.with_memory_pool(Arc::new(GreedyMemoryPool::new(2600 * 1024)))

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Same as in hash_stream.rs: the table (310 KB) and the batch charge (2.05 MB) both fit in 2600 KB, so the retry never runs and removing it passes every test. 2300 KB makes this test reach the retry and fail without it:

Suggested change
.with_memory_pool(Arc::new(GreedyMemoryPool::new(2600 * 1024)))
.with_memory_pool(Arc::new(GreedyMemoryPool::new(2300 * 1024)))

}

#[tokio::test]
async fn final_aggregate_reserves_incoming_state_batch() -> Result<()> {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This test still passes if the up-front charge is removed, because array_agg keeps the batch and is charged after merging anyway. Could you add a test with max, which keeps only one value of the batch? Then only the new charge can refuse the batch: the test below passes now and fails with the charge removed.

Test for the ungrouped path (uses imports already in the module)
/// `max` keeps one value of the batch, so only the up-front reservation
/// can refuse a state batch that holds 1 MB.
#[tokio::test]
async fn final_aggregate_reserves_state_batch_it_does_not_retain() -> Result<()> {
    let schema = Arc::new(Schema::new(vec![Field::new("v", DataType::Utf8, false)]));
    let aggregate = Arc::new(
        AggregateExprBuilder::new(max_udaf(), vec![col("v", &schema)?])
            .schema(Arc::clone(&schema))
            .alias("max")
            .build()?,
    );
    let empty = TestMemoryExec::try_new_exec(&[vec![]], Arc::clone(&schema), None)?;
    let state = AggregateExec::try_new(
        AggregateMode::Partial,
        PhysicalGroupBy::default(),
        vec![Arc::clone(&aggregate)],
        vec![None],
        empty,
        Arc::clone(&schema),
    )?
    .schema();
    // 64 rows of 16 KB string states (1 MB).
    let value = "a".repeat(16 * 1024);
    let values: ArrayRef = Arc::new(StringArray::from_iter_values(
        (0..64).map(|_| value.as_str()),
    ));
    let batch = RecordBatch::try_new(Arc::clone(&state), vec![values])?;
    let input =
        TestMemoryExec::try_new_exec(&[vec![batch]], Arc::clone(&state), None)?;
    let plan: Arc<dyn ExecutionPlan> = Arc::new(AggregateExec::try_new(
        AggregateMode::Final,
        PhysicalGroupBy::default(),
        vec![aggregate],
        vec![None],
        input,
        state,
    )?);
    let runtime = RuntimeEnvBuilder::new()
        .with_memory_pool(Arc::new(GreedyMemoryPool::new(512 * 1024)))
        .build_arc()?;
    let context = Arc::new(TaskContext::default().with_runtime(runtime));
    let err = collect(plan, context).await.unwrap_err();
    assert!(matches!(
        err.find_root(),
        DataFusionError::ResourcesExhausted(_)
    ));
    Ok(())
}

.max(MIN_CHARGED_STATE_BATCH_BYTES);
let buffer_bytes =
datafusion_common::utils::memory::get_record_batch_memory_size(batch);
if buffer_bytes <= in_flight_bytes {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Optional: this early return also decides the outcome for shared Utf8View values, not only the cost. 8192 views of one 2 KB value hold 130 KB, but their rows count 16 MB, so removing it makes such a batch fail in a 4 MB pool. No test covers that today; the one below passes now and fails without the early return.

Test for `aggregate_stream.rs`
/// Views of one shared value count it once per row, so only the buffers
/// the batch holds decide whether it is charged.
#[tokio::test]
async fn final_aggregate_does_not_charge_shared_view_bytes() -> Result<()> {
    let schema = Arc::new(Schema::new(vec![Field::new(
        "v",
        DataType::Utf8View,
        false,
    )]));
    let aggregate = Arc::new(
        AggregateExprBuilder::new(max_udaf(), vec![col("v", &schema)?])
            .schema(Arc::clone(&schema))
            .alias("max")
            .build()?,
    );
    let empty = TestMemoryExec::try_new_exec(&[vec![]], Arc::clone(&schema), None)?;
    let state = AggregateExec::try_new(
        AggregateMode::Partial,
        PhysicalGroupBy::default(),
        vec![Arc::clone(&aggregate)],
        vec![None],
        empty,
        Arc::clone(&schema),
    )?
    .schema();
    // 8192 views of one 2 KB value: 130 KB of buffers, 16 MB counted per row.
    let value = "a".repeat(2 * 1024);
    let mut builder =
        arrow::array::StringViewBuilder::new().with_deduplicate_strings();
    for _ in 0..8192 {
        builder.append_value(&value);
    }
    let values: ArrayRef = Arc::new(builder.finish());
    let batch = RecordBatch::try_new(Arc::clone(&state), vec![values])?;
    let input =
        TestMemoryExec::try_new_exec(&[vec![batch]], Arc::clone(&state), None)?;
    let plan: Arc<dyn ExecutionPlan> = Arc::new(AggregateExec::try_new(
        AggregateMode::Final,
        PhysicalGroupBy::default(),
        vec![aggregate],
        vec![None],
        input,
        state,
    )?);
    let runtime = RuntimeEnvBuilder::new()
        .with_memory_pool(Arc::new(GreedyMemoryPool::new(4 * 1024 * 1024)))
        .build_arc()?;
    let context = Arc::new(TaskContext::default().with_runtime(runtime));
    collect(plan, context).await?;
    Ok(())
}

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

physical-plan Changes to the physical-plan crate

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants