Repository navigation
Conversation
1c93e8c to
0539bfb
Compare
jayzhan211
left a comment
There was a problem hiding this comment.
@yashdotrv thanks for picking this up. Please correct the description, since most of the state is still unreserved and buffer growth is not the main cause. With the ignored #26705 cases in #26728 turned on, unreserved memory grows 121-124 MB for median (122-143 MB on main) and 500-505 MB for array_agg (531-589 MB), against a 16 MB bound. At the peak 19.1 MB of 152 MB is reserved for median, about one of the four partial states; the others appear to be queued in CoalescePartitionsExec.
A follow-up is fine because that queue is outside these streams, so #26705 should stay open. The description should also say that array_agg over 1000 groups at 30-60 MB goes from passing to ResourcesExhausted, while peak RSS at 100 MB stays near 610 MB.
|
@jayzhan211 ty for your review, I'll work on the suggestions and get back. |
0539bfb to
a7850b7
Compare
|
@jayzhan211 Thanks again for the review and detailed pointers! I have updated the branch with the requested changes:
|
jayzhan211
left a comment
There was a problem hiding this comment.
@yashdotrv thanks for the quick turnaround; every point from the first round is addressed. One new problem remains: the charge counts the whole buffer behind a zero-copy slice, so a grouped query without a hash repartition goes from passing to ResourcesExhausted. Details and a fix are inline on incoming_state_batch_reservation_bytes. The fix changes how the new tests should build their large state batch, so I've also asked for one more test there.
- Charge partial-state batches only when bytes per row exceed 1 KiB and total bytes exceed 64 KiB - Restore early break on soft group limit before table memory resize in FinalHashAggregateStream - Extract shared spill_table helper in FinalHashAggregateStream - Guard incoming batch memory reservation calculation to Partial input mode in AggregateStream - Replace synthetic tests with large-buffer retaining batches to properly test state batch admission - Add test for ClusteredFinalAggregateStream spill-and-retry on partial clustering - Add test for FinalHashAggregateStream spill-and-retry on large state batch - Restore aggregate_memory_call_sequence.rs to match main
… aggregate - Measure only bytes referenced by slice rows using SpilledRowSizes when buffer exceeds in_flight_bytes - Add unit tests for bounded state batches, shared buffer slices, and Single mode raw input - Update test cases to use realistic state payloads
b1f2c29 to
b8d1065
Compare
@jayzhan211 Thanks for catching the zero-copy buffer slicing issue and for the detailed suggestions! I have updated the PR and rebased against the latest
|
jayzhan211
left a comment
There was a problem hiding this comment.
@yashdotrv thanks for adopting every point from the last round. The 500k-group query without a hash repartition now passes at 40–80 MB again, and the 30M-row median at 300 MB is still refused. Two tests no longer reach the paths they are named for; the fixes are inline.
Optional: the Rationale opens with the mechanism, so a reviewer can't see the symptom from its first paragraph, and "What changes" doesn't mention the 1 KiB/row rule, which is the part that decides which batches are charged. Something like:
SELECT median(v) FROM generate_series(1, 30000000)with 4 target partitions and a 300 MB memory limit runs past the limit (656–939 MB peak RSS in #26705): the partial aggregate releases its reservation when it emits its state, and the final aggregate reserves memory only after merging it. This PR reserves each incoming state batch before merging it, so that query now returnsResourcesExhausted.A batch is charged only when its rows reference more than 1 KiB each on average (and 64 KiB in total), so ordinary
batch_size-bounded state batches are not charged. Slices of a larger partial output are charged only for the bytes their rows reference.
| )?); | ||
| // Fits the large batch, but not next to the table of 4000 groups. | ||
| let runtime = RuntimeEnvBuilder::new() | ||
| .with_memory_pool(Arc::new(GreedyMemoryPool::new(2600 * 1024))) |
There was a problem hiding this comment.
@yashdotrv thanks for the update; the slice fix and the new tests look good. With the new row sizing, this test no longer reaches the spill-and-retry path: the table holds 432 KB and the batch charge is 2.05 MB, so both fit in 2600 KB, and removing the retry passes every test. At 2400 KB the batch no longer fits beside the table, so the retry runs, and the test fails without it:
| .with_memory_pool(Arc::new(GreedyMemoryPool::new(2600 * 1024))) | |
| .with_memory_pool(Arc::new(GreedyMemoryPool::new(2400 * 1024))) |
|
|
||
| // Fits the large batch, but not next to the table of 4000 groups. | ||
| let runtime = RuntimeEnvBuilder::new() | ||
| .with_memory_pool(Arc::new(GreedyMemoryPool::new(2600 * 1024))) |
There was a problem hiding this comment.
Same as in hash_stream.rs: the table (310 KB) and the batch charge (2.05 MB) both fit in 2600 KB, so the retry never runs and removing it passes every test. 2300 KB makes this test reach the retry and fail without it:
| .with_memory_pool(Arc::new(GreedyMemoryPool::new(2600 * 1024))) | |
| .with_memory_pool(Arc::new(GreedyMemoryPool::new(2300 * 1024))) |
| } | ||
|
|
||
| #[tokio::test] | ||
| async fn final_aggregate_reserves_incoming_state_batch() -> Result<()> { |
There was a problem hiding this comment.
This test still passes if the up-front charge is removed, because array_agg keeps the batch and is charged after merging anyway. Could you add a test with max, which keeps only one value of the batch? Then only the new charge can refuse the batch: the test below passes now and fails with the charge removed.
Test for the ungrouped path (uses imports already in the module)
/// `max` keeps one value of the batch, so only the up-front reservation
/// can refuse a state batch that holds 1 MB.
#[tokio::test]
async fn final_aggregate_reserves_state_batch_it_does_not_retain() -> Result<()> {
let schema = Arc::new(Schema::new(vec![Field::new("v", DataType::Utf8, false)]));
let aggregate = Arc::new(
AggregateExprBuilder::new(max_udaf(), vec![col("v", &schema)?])
.schema(Arc::clone(&schema))
.alias("max")
.build()?,
);
let empty = TestMemoryExec::try_new_exec(&[vec![]], Arc::clone(&schema), None)?;
let state = AggregateExec::try_new(
AggregateMode::Partial,
PhysicalGroupBy::default(),
vec![Arc::clone(&aggregate)],
vec![None],
empty,
Arc::clone(&schema),
)?
.schema();
// 64 rows of 16 KB string states (1 MB).
let value = "a".repeat(16 * 1024);
let values: ArrayRef = Arc::new(StringArray::from_iter_values(
(0..64).map(|_| value.as_str()),
));
let batch = RecordBatch::try_new(Arc::clone(&state), vec![values])?;
let input =
TestMemoryExec::try_new_exec(&[vec![batch]], Arc::clone(&state), None)?;
let plan: Arc<dyn ExecutionPlan> = Arc::new(AggregateExec::try_new(
AggregateMode::Final,
PhysicalGroupBy::default(),
vec![aggregate],
vec![None],
input,
state,
)?);
let runtime = RuntimeEnvBuilder::new()
.with_memory_pool(Arc::new(GreedyMemoryPool::new(512 * 1024)))
.build_arc()?;
let context = Arc::new(TaskContext::default().with_runtime(runtime));
let err = collect(plan, context).await.unwrap_err();
assert!(matches!(
err.find_root(),
DataFusionError::ResourcesExhausted(_)
));
Ok(())
}| .max(MIN_CHARGED_STATE_BATCH_BYTES); | ||
| let buffer_bytes = | ||
| datafusion_common::utils::memory::get_record_batch_memory_size(batch); | ||
| if buffer_bytes <= in_flight_bytes { |
There was a problem hiding this comment.
Optional: this early return also decides the outcome for shared Utf8View values, not only the cost. 8192 views of one 2 KB value hold 130 KB, but their rows count 16 MB, so removing it makes such a batch fail in a 4 MB pool. No test covers that today; the one below passes now and fails without the early return.
Test for `aggregate_stream.rs`
/// Views of one shared value count it once per row, so only the buffers
/// the batch holds decide whether it is charged.
#[tokio::test]
async fn final_aggregate_does_not_charge_shared_view_bytes() -> Result<()> {
let schema = Arc::new(Schema::new(vec![Field::new(
"v",
DataType::Utf8View,
false,
)]));
let aggregate = Arc::new(
AggregateExprBuilder::new(max_udaf(), vec![col("v", &schema)?])
.schema(Arc::clone(&schema))
.alias("max")
.build()?,
);
let empty = TestMemoryExec::try_new_exec(&[vec![]], Arc::clone(&schema), None)?;
let state = AggregateExec::try_new(
AggregateMode::Partial,
PhysicalGroupBy::default(),
vec![Arc::clone(&aggregate)],
vec![None],
empty,
Arc::clone(&schema),
)?
.schema();
// 8192 views of one 2 KB value: 130 KB of buffers, 16 MB counted per row.
let value = "a".repeat(2 * 1024);
let mut builder =
arrow::array::StringViewBuilder::new().with_deduplicate_strings();
for _ in 0..8192 {
builder.append_value(&value);
}
let values: ArrayRef = Arc::new(builder.finish());
let batch = RecordBatch::try_new(Arc::clone(&state), vec![values])?;
let input =
TestMemoryExec::try_new_exec(&[vec![batch]], Arc::clone(&state), None)?;
let plan: Arc<dyn ExecutionPlan> = Arc::new(AggregateExec::try_new(
AggregateMode::Final,
PhysicalGroupBy::default(),
vec![aggregate],
vec![None],
input,
state,
)?);
let runtime = RuntimeEnvBuilder::new()
.with_memory_pool(Arc::new(GreedyMemoryPool::new(4 * 1024 * 1024)))
.build_arc()?;
let context = Arc::new(TaskContext::default().with_runtime(runtime));
collect(plan, context).await?;
Ok(())
}
Which issue does this PR close?
Addresses #26705.
Rationale for this change
A partial aggregate can emit a state batch containing all its input values and then release its memory reservation. The final aggregate merges that batch before reserving memory for it. For aggregates such as
medianandarray_agg, this leaves input-sized state unaccounted for by the memory pool.What changes are included in this PR?
Final aggregates now reserve incoming state batches before merging them. The hash and clustered final paths can spill their current state and retry when the pool cannot admit a batch. I also updated the affected memory reservation and spill tests.
PR #26728 adds related memory accounting tests, but its cases for this issue are currently ignored.
What is the testing strategy for this PR?
I added reservation tests for the ungrouped, hash grouped, and clustered grouped final aggregate paths. The extended workspace test suite, formatting check, and Clippy pass.
I also ran the issue’s 30-million-row, four-partition
medianreproducer with a 300 MB memory limit. It now returnsResourcesExhaustedwhen the incoming state cannot be reserved. Peak RSS can still exceed the configured limit; reducing temporary memory use during buffer growth is outside this PR.Are there any user-facing changes?
Under memory pressure, a final aggregate may now spill or return
ResourcesExhaustedinstead of continuing with unreserved state. There are no public API changes.