Repository navigation
Conversation
… the last batch CustomDistributedBatchSampler (and CustomDistributedBufferBatchSampler) computed a total_size smaller than the dataset for drop_last=True, but the padding branch then appended indices[:negative] instead of truncating, so __iter__ failed its own length assertion for any dataset size that is not a multiple of batch_size * num_replicas. Truncate to total_size, and do not index batches[-1] when no batch is left.
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
CustomDistributedBatchSampler(the samplerbatch_type="example"builds) computestotal_sizeas the largest multiple ofbatch_size * num_replicaswhendrop_last=True, which is smaller than the dataset. The "make it evenly divisible" step then runsindices += indices[:padding_size]with a negativepadding_size, which appends all but the last few indices instead of truncating.__iter__then fails its ownassert len(indices) == self.total_sizefor every dataset size that is not already a multiple ofbatch_size * num_replicas. With fewer examples than one batch it fails the same way, and the laterbatches[-1]would raiseIndexErroron an empty list.This truncates to
total_sizewhen the padding is negative and skips the last-batch check when no batch is left.CustomDistributedBufferBatchSamplerhas the same lines and gets the same change. Behavior withdrop_last=Falseis unchanged.Type of change
Validation
python -m compileall funasr examples testspython -m pytest tests/test_batch_sampler_drop_last.py -q: 5 failed (AssertionErrorat samplers.py:121 and :235), 1 passed on current main (thedrop_last=Falseguard); 6 passed with the fix.batch_size=4,shuffle=False, single replica: 10 examples raisesAssertionError; 3 examples raisesAssertionError; 8 examples works. With the fix, 10 examples give[[0, 1, 2, 3], [4, 5, 6, 7]]andlen(sampler) == 2, 3 examples give no batches.drop_last=Falsestill gives[[0, 1, 2, 3], [4, 5, 6, 7], [8, 9, 0, 1]].CustomDistributedBatchSampler_fn(..., batch_type="example", drop_last=True)(the path++dataset_conf.drop_last=truetakes): same result as above.dist.get_rank/dist.get_world_size(1, 2, 3, 4, 8 replicas × dataset sizes 0-40 × batch sizes 1-5 × shuffle on/off × two epochs): withdrop_last=Trueevery rank yields the same number of full batches, equal tolen(sampler), no index appears twice across ranks, and fewer thanbatch_size * num_replicasexamples are left out per epoch (a different tail each epoch when shuffling). For alldrop_last=Falsecases, and fordrop_last=Truewherever main did not raise, the output is identical to main.User impact
Training or evaluation with
dataset_conf.drop_last=trueandbatch_type=examplestarts instead of failing at the first epoch whenever the dataset size is not a multiple of the global batch size.Notes for reviewers
drop_last, so this is reachable only through an explicit override.samplers.py.dataset_confalso builds the validation sampler, so withdrop_last=truea validation set smaller than one global batch now gives no validation batches instead of anAssertionError(the trainer then reports validation metrics as unavailable; from reading the code).CustomDistributedBufferBatchSampleris not built by any FunASR code path; it gets the same change so the two copies stay identical.