Skip to content

fix(data): truncate instead of asserting when the batch sampler drops the last batch - #3913

Open
sohailshk wants to merge 1 commit into
modelscope:mainfrom
sohailshk:fix/sampler-drop-last
Open

sohailshk wants to merge 1 commit into
modelscope:mainfrom
sohailshk:fix/sampler-drop-last

Conversation

@sohailshk

Copy link
Copy Markdown
Contributor

Summary

CustomDistributedBatchSampler (the sampler batch_type="example" builds) computes total_size as the largest multiple of batch_size * num_replicas when drop_last=True, which is smaller than the dataset. The "make it evenly divisible" step then runs indices += indices[:padding_size] with a negative padding_size, which appends all but the last few indices instead of truncating. __iter__ then fails its own assert len(indices) == self.total_size for every dataset size that is not already a multiple of batch_size * num_replicas. With fewer examples than one batch it fails the same way, and the later batches[-1] would raise IndexError on an empty list.

This truncates to total_size when the padding is negative and skips the last-batch check when no batch is left. CustomDistributedBufferBatchSampler has the same lines and gets the same change. Behavior with drop_last=False is unchanged.

Type of change

  • Bug fix

Validation

  • python -m compileall funasr examples tests
  • python -m pytest tests/test_batch_sampler_drop_last.py -q: 5 failed (AssertionError at samplers.py:121 and :235), 1 passed on current main (the drop_last=False guard); 6 passed with the fix.
  • Reproduction on main, batch_size=4, shuffle=False, single replica: 10 examples raises AssertionError; 3 examples raises AssertionError; 8 examples works. With the fix, 10 examples give [[0, 1, 2, 3], [4, 5, 6, 7]] and len(sampler) == 2, 3 examples give no batches. drop_last=False still gives [[0, 1, 2, 3], [4, 5, 6, 7], [8, 9, 0, 1]].
  • Through CustomDistributedBatchSampler_fn(..., batch_type="example", drop_last=True) (the path ++dataset_conf.drop_last=true takes): same result as above.
  • Multiple replicas, simulated in one process by patching dist.get_rank/dist.get_world_size (1, 2, 3, 4, 8 replicas × dataset sizes 0-40 × batch sizes 1-5 × shuffle on/off × two epochs): with drop_last=True every rank yields the same number of full batches, equal to len(sampler), no index appears twice across ranks, and fewer than batch_size * num_replicas examples are left out per epoch (a different tail each epoch when shuffling). For all drop_last=False cases, and for drop_last=True wherever main did not raise, the output is identical to main.

User impact

Training or evaluation with dataset_conf.drop_last=true and batch_type=example starts instead of failing at the first epoch whenever the dataset size is not a multiple of the global batch size.

Notes for reviewers

  • No bundled recipe sets drop_last, so this is reachable only through an explicit override.
  • The dynamic (token-based) samplers use a different last-batch rule and are not touched.
  • The new test is added to the existing NumPy matrix workflow with a path trigger for samplers.py.
  • dataset_conf also builds the validation sampler, so with drop_last=true a validation set smaller than one global batch now gives no validation batches instead of an AssertionError (the trainer then reports validation metrics as unavailable; from reading the code).
  • CustomDistributedBufferBatchSampler is not built by any FunASR code path; it gets the same change so the two copies stay identical.

… the last batch

CustomDistributedBatchSampler (and CustomDistributedBufferBatchSampler)
computed a total_size smaller than the dataset for drop_last=True, but the
padding branch then appended indices[:negative] instead of truncating, so
__iter__ failed its own length assertion for any dataset size that is not a
multiple of batch_size * num_replicas. Truncate to total_size, and do not
index batches[-1] when no batch is left.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant