You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
test(cuda.core): cover re-importing a pinned pool after a release with a pending free - #3531
Adds a regression test for re-importing a pinned pool after a release with an unsynchronized free still pending. The fix already landed in #3511 (the CUDA_ERROR_ALREADY_MAPPED retry in MP_set_access). Only the existing race test catches its removal, and only as a side effect of its timing scenario, so this tests the sequence from the issue directly.
The existing child functions in the module now release their resources through an ExitStack when a test fails.
The test runs in a child process and is parametrized over the import paths named in the issue: a single import, several imports of one raw handle, and concurrent imports of one UUID. It asserts that the re-import grants read/write access, that a buffer imported through it can be filled, and that no CUDAWarning is issued.
…h a pending free
Releasing the last import of a pool while an unsynchronized
cuMemFreeAsync is still pending leaves the driver holding the pool. The
next import returns the same CUmemoryPool, and its first cuMemPoolSetAccess
fails with CUDA_ERROR_ALREADY_MAPPED. The retry added in NVIDIA#3511 handles
this, but no test exercised the sequence.
The new test imports a pool, fills a buffer, closes the buffer without
synchronizing, releases every resource, and imports the pool again. It
asserts that the re-import grants read/write access, that a buffer
imported through it can be filled, and that no CUDAWarning is issued. It
covers a single import, several imports of one raw handle, and concurrent
imports of one UUID.
Without the retry, the three PinnedMemoryResource cases fail with
CUDA_ERROR_ALREADY_MAPPED. The DeviceMemoryResource cases pass either way,
since a device pool import grants no access.
ClosesNVIDIA#3514.
Signed-off-by: Ralf Juengling <rjuengling@nvidia.com>
The child functions in test_ipc_pool_duplicate_import.py closed their
resources and buffers at the end of the body, so a failed assertion or a
failed second import left them open. They now enter each one into an
ExitStack as it is created: closing(resource) for the pool resources,
which are not context managers, and the buffer itself. The stack closes in
reverse order, so buffers still close before the resources they came
from. Closes that are the sequence under test stay explicit, and the stack
repeats them as no-ops.
_import_twice takes the stack, so the first import is released if the
second fails. _import_concurrently closes the imports that succeeded
before it raises. The race test also owns the buffer, resource and stream
that its re-import thread creates, and still synchronizes the stream
before closing it.
Signed-off-by: Ralf Juengling <rjuengling@nvidia.com>
Three helpers remove repetition across the child functions: _enter_import wraps an import in contextlib.closing on an ExitStack, _run_threads starts and joins a list of threads, and the _child_device context manager sets the current device and asserts the owned-pool count returns to its baseline on exit. The child functions use them instead of repeating the setup, the thread start/join loops, and the baseline bookkeeping.
Signed-off-by: Ralf Juengling <rjuengling@nvidia.com>
No actionable comments were generated in the recent review.
Summary
Summary by CodeRabbit
Tests
Expanded coverage for duplicate and repeated imports of shared-memory pools, including imports while a resource release is pending.
Added checks that imported pools remain accessible during pending asynchronous frees across single, raw-handle, and concurrent import paths.
Improved cleanup coverage for imported resources and buffers, including after partial failures during concurrent imports.
The IPC pool tests now use scoped cleanup for imported resources and thread-created results. New child-process tests reimport pools while buffer frees remain pending through single, raw-handle, and concurrent-UUID import paths.
Duplicate-import close tests, the last-release race, and the repeated reentry test use ExitStack cleanup. Concurrent UUID imports close successful imports if a worker fails.
Reimport with pending buffer frees cuda_core/tests/memory_ipc/test_ipc_pool_duplicate_import.py
New child-process tests queue imported buffer frees behind a latch, then reimport through single, raw-handle, and concurrent-UUID paths. They check read-write access and fill a buffer.
Assessment against linked issues
Objective
Addressed
Explanation
Reimport a pool after releasing an imported buffer with a pending free, and verify access through the reimported pool [#3514]
❌
The diff adds tests for this sequence but does not change production code to fix the reported CUDA_ERROR_ALREADY_MAPPED failure.
NVIDIA/cuda-python#3509: Adds import-time read/write access grants for imported pinned pools, which the tests verify.
Change: Other · Severity of issue fixed: Medium
Merge Risk: Low · up to 1db04
This PR adds a regression test for re-importing a pinned memory pool while an asynchronous buffer free is pending, and moves the existing duplicate-import tests to scoped cleanup. There is no production impact. The remaining risk is test-only: if a worker thread fails to start, the test reports a child timeout instead of the start failure. This is rare and does not block merging.
Security architecture risk: Minimal · up to 1db04
The change adds regression coverage for pending GPU memory frees and improves test-resource cleanup. The inspected changes introduce no material security, isolation, or failure-containment risk.
- [ ] Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts
Included review availability: This review used your included allowance. Your plan provides up to 12 included reviews per hour; 9 remain after this review.
301-302: Trivial · Functional Correctness · Heavy lift
suggestion: Keep the pending free outstanding until the re-import grant is checked.
_import_fill_and_release_unsynchronized returns with its buffer frees enqueued but not complete. Its stream.sync() at line 281 runs before the buffers and resources close. _reimport_and_use then checks access without waiting on the stream. Nothing in the test orders the frees before the grant, so the device can finish them first. Issue #3514 shows that a completed free removes the CUDA_ERROR_ALREADY_MAPPED state. In that case the re-import passes without exercising the retry, and the test cannot detect its removal.
Make the pending state hold at the grant check, so each round exercises the path the test names. Confirm on a driver that reproduces #3514 that the test fails when the retry is disabled.
Included review availability: This review used your included allowance. Your plan provides up to 12 included reviews per hour; 11 remain after this review.
The re-import test relied on timing: it closed a buffer and re-imported the pool without ordering the enqueued cuMemFreeAsync before the grant, so a driver that completed the free first would clear the CUDA_ERROR_ALREADY_MAPPED state and pass without exercising the retry (CodeRabbit). It now holds the free pending behind a NanosleepKernel block on a dedicated stream, so every round checks the grant while the pool is still mapped. With the retry disabled all three PinnedMemoryResource paths now fail deterministically. The rounds also drain the stream in a finally block so the imported frees complete before the exporter frees its allocation even when a round fails partway.
Signed-off-by: Ralf Juengling <rjuengling@nvidia.com>
Thanks @coderabbitai. Addressed the determinism nitpick (holding the free pending at the grant check) in 95bb06b: each round now holds the enqueued cuMemFreeAsync behind a NanosleepKernel block on a dedicated stream, so the pool stays mapped while the re-import grants access. Confirmed on the driver that reproduces #3514 (RTX 6000 Ada, 595.99.02) that with the retry disabled all three PinnedMemoryResource paths now fail deterministically, and pass with it restored. Replies to the two inline comments are inline (drain-on-failure addressed; the _run_threads barrier-abort dismissed with rationale).
child_close_one_import synchronized the fill but closed its imported buffer afterwards, through the ExitStack, and never drained again, so the cuMemFreeAsync was still pending when the child exited. child_close_both_imports_with_outstanding_buffer drained on success but not when an assertion failed partway. Both now register stream.sync on the stack before importing, so it runs last and the imported frees complete before the exporter frees its allocation on every path, as _race_reimport_with_last_release already does.
Signed-off-by: Ralf Juengling <rjuengling@nvidia.com>
Included review availability: This review used your included allowance. Your plan provides up to 12 included reviews per hour; 10 remain after this review.
The NanosleepKernel block lasted a fixed 200 ms, so a child descheduled for longer between the launch and the grant check could let the free retire first and pass without exercising the retry (CodeRabbit). The free is now held behind a LatchKernel that is released only after the grant has been checked, and the test asserts that the latch did not time out. The latch is released and the stream drained through the ExitStack on every path. With a deterministic hold one round suffices, so the round loop is gone. LatchKernel needs the CUDA headers and sm70; the parent test skips when either is missing, because a skip raised in the child would surface as a child failure. The previous NanosleepKernel also needed sm70 but had no guard.
Signed-off-by: Ralf Juengling <rjuengling@nvidia.com>
For best results, initiate chat on the files or code changes.
@juenglin suggestion: I’ve noted your reported test results and inline replies. The PR summary also describes a later change from NanosleepKernel to LatchKernel. I have not independently run the tests.
Learnings used
Learnt from: juenglin
Repo: NVIDIA/cuda-python PR: 3511
File: cuda_core/tests/memory_ipc/test_ipc_pool_duplicate_import.py:125-126
Timestamp: 2026-11-20T23:50:16.555Z
Learning: Within repository "NVIDIA/cuda-python", only for files matching one of ["cuda_core/tests/memory_ipc/test_ipc_pool_duplicate_import.py"]: For `_race_reimport_with_last_release` in `cuda_core/tests/memory_ipc/test_ipc_pool_duplicate_import.py`, the author accepts that the timing-based test can pass without overlapping final release. The author reports that the existing delay hook failed reliably against the pre-fix code and considers a second test-only hook in `ReleasingHandleRegistry` too costly for this case. Do not demand that hook solely to eliminate the known false-pass risk without new evidence that changes the trade-off.
You are interacting with an AI system.
This branch has not been deployed
No deployments
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
cuda.coreEverything related to the cuda.core moduletestImprovements or additions to tests
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
closes #3514
Adds a regression test for re-importing a pinned pool after a release with an unsynchronized free still pending. The fix already landed in #3511 (the
CUDA_ERROR_ALREADY_MAPPEDretry inMP_set_access). Only the existing race test catches its removal, and only as a side effect of its timing scenario, so this tests the sequence from the issue directly.The existing child functions in the module now release their resources through an
ExitStackwhen a test fails.The test runs in a child process and is parametrized over the import paths named in the issue: a single import, several imports of one raw handle, and concurrent imports of one UUID. It asserts that the re-import grants read/write access, that a buffer imported through it can be filled, and that no
CUDAWarningis issued.Checklist