Repository navigation
Description
Is this a duplicate?
- I confirmed there appear to be no duplicate issues for this bug and that I agree to the Code of Conduct
Type of Bug
Runtime Error
Component
cuda.core
Describe the bug
A Context returned by public cuda.core APIs can fail to keep the primary context alive. When two threads retain the same primary context and one of them exits, the context registry loses its entry even though the other thread's handle is still live. Later lookups then return a plain, non-retaining Context. Once the remaining retaining handles are released, the driver destroys the primary context while the application still holds a Context for it. Using it then fails with CUDA_ERROR_CONTEXT_IS_DESTROYED.
This is not the shutdown crash from #2743; it happens while the interpreter is running.
There are two root causes, both in cuda_core/cuda/core/_cpp/rt/context.cpp:
get_primary_context()caches handles in athread_localvector. Each thread that misses its own cache callscuDevicePrimaryCtxRetainand creates a separate retaining box, then callscontext_registry.register_handle(ctx, h), replacing the previous thread's entry for the sameCUcontext.- Every context deleter calls
context_registry.unregister_handle(b->resource), which erases the entry for thatCUcontextunconditionally, even when the entry belongs to a different, still-live box.ReleasingHandleRegistry::release()already checks ownership before erasing;HandleRegistry::unregister_handle()does not.
The failing sequence:
- Thread A calls
Device(0).set_current(). It retains the primary context (retain count 1) and registers box A. - Thread B calls
Device(0).set_current(). It retains again (count 2) and registers box B, overwriting A's entry. - Thread B exits. Box B's deleter erases the registry entry and releases its retain (count 1). Box A is still alive but no longer registered.
- Thread A resolves the context of a stream with no context metadata, e.g.
Stream.from_handle(CU_STREAM_LEGACY).context.create_context_handle_ref()misses in the registry and returns a new non-retaining box P. - Thread A exits and releases box A's retain (count 0). Box A's deleter also erases P's registry entry. The driver destroys the primary context while the application still holds the
Contextbacked by P.
Without step 2 and 3, step 4 returns box A itself, which keeps the retain alive, and everything works.
Any path that builds a context reference through create_context_handle_ref() or get_current_context() is exposed: Stream.from_handle(...), the context of streams without context metadata, Device.set_current(ctx) returning the previous context, graph node updates, and Buffer unpickling. The trigger is ordinary thread-pool behavior: multiple threads using cuda.core on the same device, some of which exit.
Impact: the usual result is a CUDA_ERROR_CONTEXT_IS_DESTROYED error from an object the application believes is valid, and loss of all allocations, modules, and streams in that context. In the worst case, a later cuda.core call creates a new primary context; stale handles from the old one (for example a Buffer whose deleter later runs) can then refer to resources in the new context, leading to frees of unrelated memory and silent corruption. Any library that retains the primary context through the CUDA runtime (PyTorch, CuPy, Numba, ...) holds an extra retain and masks the bug.
The same unconditional erase also affects the green-context path: create_context_handle_ref(ctx, h_green_ctx) returns an existing registry entry without the green-context handle, and duplicate boxes for one CUcontext can drop each other's registry entries. If cuda.core ever attaches metadata to context boxes, duplicates will also silently lose that metadata.
How to Reproduce
Run on a machine where nothing else retains the primary context of device 0:
import threading
import time
from cuda.bindings import driver
from cuda.core import Device, Stream
out = []
b_exited = threading.Event()
def thread_a():
Device(0).set_current() # retains the primary context, registers handle A
b_exited.wait()
# Registry miss: returns a Context that holds no retain.
out.append(Stream.from_handle(int(driver.CU_STREAM_LEGACY)).context)
a = threading.Thread(target=thread_a)
a.start()
# Thread B retains again, replaces A's registry entry, and on exit erases it.
b = threading.Thread(target=lambda: Device(0).set_current())
b.start()
b.join()
time.sleep(1) # B's thread_local destructors run after join() returns
b_exited.set()
a.join()
deadline = time.monotonic() + 5 # wait for A's destructors too
while driver.cuDevicePrimaryCtxGetState(0)[2] and time.monotonic() < deadline:
time.sleep(0.05)
print("primary context active:", bool(driver.cuDevicePrimaryCtxGetState(0)[2]))
Device(0).set_current(out[0])
Device(0).create_stream() # CUDA_ERROR_CONTEXT_IS_DESTROYEDOutput:
primary context active: False
Traceback (most recent call last):
...
File "cuda/core/_device.pyx", line 1433, in cuda.core._device.Device.create_stream
File "cuda/core/_stream.pyx", line 162, in cuda.core._stream.Stream._init
...
cuda.core._utils.cuda_utils.CUDAError: CUDA_ERROR_CONTEXT_IS_DESTROYED: This error indicates that the context current to the calling thread has been destroyed using cuCtxDestroy, or is a primary context which has not yet been initialized.
Without thread B (remove b.start() and b.join()), the lookup in thread A returns handle A itself, the primary context stays active, and the script succeeds. With thread B, the failure reproduced in 10 of 10 runs.
Expected behavior
A Context obtained from cuda.core keeps its primary context alive for as long as the application holds it, regardless of which threads retained the context or have exited. One possible fix has two parts:
- Keep one retaining primary-context box per process and device.
get_primary_context()should reuse a live registered box (releasing any extra retain it took) instead of registering a new box per thread; thethread_localcache can remain a fast path in front of it. - Make context deleters erase a registry entry only if it still refers to their own box, as
ReleasingHandleRegistry::release()already does.
The deleter must keep the shutdown guard added in #2744.
Operating System
Ubuntu Linux (kernel 7.0), Python 3.12.3, cuda-core built from main at edf1b31 (1.3.1.dev54), cuda-bindings 13.5.0a3.dev30.
nvidia-smi output
NVIDIA RTX 6000 Ada Generation, driver 595.99.02.
Activity
juenglin commentedon Nov 22, 2026
A second reproducer, where the object left with a destroyed context is an
Eventthatcuda.coreitself created. The default-stream token has no context of its own, soStream.record()resolves the event's context through the context registry and gets the non-retaining reference:Output:
The last line is the event's own deleter failing at exit, because its handle now refers to a destroyed context. Without thread B, the script succeeds. With thread B, it failed in 10 of 10 runs.
Objects whose context comes from
Device.context, such as streams fromDevice.create_stream()and buffers, are not affected, because they hold the device's retaining handle. On a single GPU, saving and restoring the current context withprev = dev.set_current(green_ctx)anddev.set_current(prev)is also safe, becauseDevice.set_current()returns its storedContextwhen it matches the previous context; the cross-device case does go through the registry lookup.juenglin commentedon Nov 22, 2026
Besides the primary-context reproducers in the description and the comment above, the green-context path mentioned in the description has a more severe consequence than lost metadata. When
Device.create_context()gets aCUcontextfor which a plain, non-owning entry is still registered,create_context_handle_ref(ctx, h_green_ctx)returns that entry and dropsh_green_ctx, which is the only owner of the new green context. The green context is destroyed as soon ascreate_context()returns. The caller gets aContextwithis_green == Falsethat compares and hashes equal to the unrelated staleContext, and using it fails withCUDA_ERROR_CONTEXT_IS_DESTROYED.With only
cuda.coreAPIs, the stale plain entry can be created through #3534:Output (the driver reused the address after 1 to 4 tries):
Fixing #3534 removes this cuda.core-only path, but not the bug: the same happens when the plain entry comes from a green context created and destroyed through
cuda.bindingsor another library.create_context_handle_ref()also registers plain boxes unconditionally, so if a plain lookup races withget_primary_context()on another thread, the plain box can displace the retaining one. The fix should therefore also keep plain registrations from displacing a live entry, and let an owning green box replace a plain one.