Skip to content

[BUG]: Primary context can be destroyed while a cuda.core Context still refers to it after a thread exits #3533

Open

Description

@juenglin

Is this a duplicate?

Type of Bug

Runtime Error

Component

cuda.core

Describe the bug

A Context returned by public cuda.core APIs can fail to keep the primary context alive. When two threads retain the same primary context and one of them exits, the context registry loses its entry even though the other thread's handle is still live. Later lookups then return a plain, non-retaining Context. Once the remaining retaining handles are released, the driver destroys the primary context while the application still holds a Context for it. Using it then fails with CUDA_ERROR_CONTEXT_IS_DESTROYED.

This is not the shutdown crash from #2743; it happens while the interpreter is running.

There are two root causes, both in cuda_core/cuda/core/_cpp/rt/context.cpp:

  1. get_primary_context() caches handles in a thread_local vector. Each thread that misses its own cache calls cuDevicePrimaryCtxRetain and creates a separate retaining box, then calls context_registry.register_handle(ctx, h), replacing the previous thread's entry for the same CUcontext.
  2. Every context deleter calls context_registry.unregister_handle(b->resource), which erases the entry for that CUcontext unconditionally, even when the entry belongs to a different, still-live box. ReleasingHandleRegistry::release() already checks ownership before erasing; HandleRegistry::unregister_handle() does not.

The failing sequence:

  1. Thread A calls Device(0).set_current(). It retains the primary context (retain count 1) and registers box A.
  2. Thread B calls Device(0).set_current(). It retains again (count 2) and registers box B, overwriting A's entry.
  3. Thread B exits. Box B's deleter erases the registry entry and releases its retain (count 1). Box A is still alive but no longer registered.
  4. Thread A resolves the context of a stream with no context metadata, e.g. Stream.from_handle(CU_STREAM_LEGACY).context. create_context_handle_ref() misses in the registry and returns a new non-retaining box P.
  5. Thread A exits and releases box A's retain (count 0). Box A's deleter also erases P's registry entry. The driver destroys the primary context while the application still holds the Context backed by P.

Without step 2 and 3, step 4 returns box A itself, which keeps the retain alive, and everything works.

Any path that builds a context reference through create_context_handle_ref() or get_current_context() is exposed: Stream.from_handle(...), the context of streams without context metadata, Device.set_current(ctx) returning the previous context, graph node updates, and Buffer unpickling. The trigger is ordinary thread-pool behavior: multiple threads using cuda.core on the same device, some of which exit.

Impact: the usual result is a CUDA_ERROR_CONTEXT_IS_DESTROYED error from an object the application believes is valid, and loss of all allocations, modules, and streams in that context. In the worst case, a later cuda.core call creates a new primary context; stale handles from the old one (for example a Buffer whose deleter later runs) can then refer to resources in the new context, leading to frees of unrelated memory and silent corruption. Any library that retains the primary context through the CUDA runtime (PyTorch, CuPy, Numba, ...) holds an extra retain and masks the bug.

The same unconditional erase also affects the green-context path: create_context_handle_ref(ctx, h_green_ctx) returns an existing registry entry without the green-context handle, and duplicate boxes for one CUcontext can drop each other's registry entries. If cuda.core ever attaches metadata to context boxes, duplicates will also silently lose that metadata.

How to Reproduce

Run on a machine where nothing else retains the primary context of device 0:

import threading
import time

from cuda.bindings import driver
from cuda.core import Device, Stream

out = []
b_exited = threading.Event()


def thread_a():
    Device(0).set_current()  # retains the primary context, registers handle A
    b_exited.wait()
    # Registry miss: returns a Context that holds no retain.
    out.append(Stream.from_handle(int(driver.CU_STREAM_LEGACY)).context)


a = threading.Thread(target=thread_a)
a.start()
# Thread B retains again, replaces A's registry entry, and on exit erases it.
b = threading.Thread(target=lambda: Device(0).set_current())
b.start()
b.join()
time.sleep(1)  # B's thread_local destructors run after join() returns
b_exited.set()
a.join()
deadline = time.monotonic() + 5  # wait for A's destructors too
while driver.cuDevicePrimaryCtxGetState(0)[2] and time.monotonic() < deadline:
    time.sleep(0.05)

print("primary context active:", bool(driver.cuDevicePrimaryCtxGetState(0)[2]))
Device(0).set_current(out[0])
Device(0).create_stream()  # CUDA_ERROR_CONTEXT_IS_DESTROYED

Output:

primary context active: False
Traceback (most recent call last):
  ...
  File "cuda/core/_device.pyx", line 1433, in cuda.core._device.Device.create_stream
  File "cuda/core/_stream.pyx", line 162, in cuda.core._stream.Stream._init
  ...
cuda.core._utils.cuda_utils.CUDAError: CUDA_ERROR_CONTEXT_IS_DESTROYED: This error indicates that the context current to the calling thread has been destroyed using cuCtxDestroy, or is a primary context which has not yet been initialized.

Without thread B (remove b.start() and b.join()), the lookup in thread A returns handle A itself, the primary context stays active, and the script succeeds. With thread B, the failure reproduced in 10 of 10 runs.

Expected behavior

A Context obtained from cuda.core keeps its primary context alive for as long as the application holds it, regardless of which threads retained the context or have exited. One possible fix has two parts:

  1. Keep one retaining primary-context box per process and device. get_primary_context() should reuse a live registered box (releasing any extra retain it took) instead of registering a new box per thread; the thread_local cache can remain a fast path in front of it.
  2. Make context deleters erase a registry entry only if it still refers to their own box, as ReleasingHandleRegistry::release() already does.

The deleter must keep the shutdown guard added in #2744.

Operating System

Ubuntu Linux (kernel 7.0), Python 3.12.3, cuda-core built from main at edf1b31 (1.3.1.dev54), cuda-bindings 13.5.0a3.dev30.

nvidia-smi output

NVIDIA RTX 6000 Ada Generation, driver 595.99.02.

Activity

  1. added this to the cuda.core 1.4.0 milestone on Nov 22, 2026
  2. added
    bugSomething isn't working
    triageNeeds the team's attention
    cuda.coreEverything related to the cuda.core module
    on Nov 22, 2026
  3. juenglin commented on Nov 22, 2026

    @juenglin
    ContributorAuthor

    A second reproducer, where the object left with a destroyed context is an Event that cuda.core itself created. The default-stream token has no context of its own, so Stream.record() resolves the event's context through the context registry and gets the non-retaining reference:

    import threading
    import time
    
    from cuda.bindings import driver
    from cuda.core import Device
    
    out = []
    b_exited = threading.Event()
    
    
    def thread_a():
        dev = Device(0)
        dev.set_current()  # retains the primary context, registers handle A
        b_exited.wait()
        # The default-stream token has no context, so the event gets a
        # non-retaining Context from the registry.
        out.append(dev.default_stream.record())
    
    
    a = threading.Thread(target=thread_a)
    a.start()
    # Thread B retains again, replaces A's registry entry, and on exit erases it.
    b = threading.Thread(target=lambda: Device(0).set_current())
    b.start()
    b.join()
    time.sleep(1)  # B's thread_local destructors run after join() returns
    b_exited.set()
    a.join()
    deadline = time.monotonic() + 5  # wait for A's destructors too
    while driver.cuDevicePrimaryCtxGetState(0)[2] and time.monotonic() < deadline:
        time.sleep(0.05)
    
    print("primary context active:", bool(driver.cuDevicePrimaryCtxGetState(0)[2]))
    out[0].sync()  # CUDA_ERROR_CONTEXT_IS_DESTROYED

    Output:

    primary context active: False
    Traceback (most recent call last):
      ...
      File "cuda/core/_event.pyx", line 294, in cuda.core._event.Event.sync
      ...
    cuda.core._utils.cuda_utils.CUDAError: CUDA_ERROR_CONTEXT_IS_DESTROYED: This error indicates that the context current to the calling thread has been destroyed using cuCtxDestroy, or is a primary context which has not yet been initialized.
    cuEventDestroy(0x7eabf4da74f0) failed: CUDA_ERROR_CONTEXT_IS_DESTROYED: context is destroyed
    

    The last line is the event's own deleter failing at exit, because its handle now refers to a destroyed context. Without thread B, the script succeeds. With thread B, it failed in 10 of 10 runs.

    Objects whose context comes from Device.context, such as streams from Device.create_stream() and buffers, are not affected, because they hold the device's retaining handle. On a single GPU, saving and restoring the current context with prev = dev.set_current(green_ctx) and dev.set_current(prev) is also safe, because Device.set_current() returns its stored Context when it matches the previous context; the cross-device case does go through the registry lookup.

  4. juenglin commented on Nov 22, 2026

    @juenglin
    ContributorAuthor

    Besides the primary-context reproducers in the description and the comment above, the green-context path mentioned in the description has a more severe consequence than lost metadata. When Device.create_context() gets a CUcontext for which a plain, non-owning entry is still registered, create_context_handle_ref(ctx, h_green_ctx) returns that entry and drops h_green_ctx, which is the only owner of the new green context. The green context is destroyed as soon as create_context() returns. The caller gets a Context with is_green == False that compares and hashes equal to the unrelated stale Context, and using it fails with CUDA_ERROR_CONTEXT_IS_DESTROYED.

    With only cuda.core APIs, the stale plain entry can be created through #3534:

    import threading
    
    from cuda.core import ContextOptions, Device, SMResourceOptions
    
    dev = Device(0)
    dev.set_current()
    primary = dev.context
    groups, _ = dev.resources.sm.split(SMResourceOptions(count=None))
    opts = ContextOptions(resources=[groups[0]])
    
    for _ in range(50):
        a = dev.create_context(opts)
        dev.set_current(a)
        t = threading.Thread(target=a.close)  # #3534: destroys A while it is current here
        t.start()
        t.join()
        stale = dev.set_current(primary)  # plain, non-owning Context for A's destroyed CUcontext
        green = dev.create_context(opts)
        if int(green.handle) == int(stale.handle):
            break
        green.close()
    
    print("is_green:", green.is_green, "equals stale:", green == stale)
    green.create_stream()  # RuntimeError: only supported on green contexts

    Output (the driver reused the address after 1 to 4 tries):

    is_green: False equals stale: True
    RuntimeError: Context.create_stream() is only supported on green contexts. Use Device.create_stream() for primary contexts.
    

    Fixing #3534 removes this cuda.core-only path, but not the bug: the same happens when the plain entry comes from a green context created and destroyed through cuda.bindings or another library.

    create_context_handle_ref() also registers plain boxes unconditionally, so if a plain lookup races with get_primary_context() on another thread, the plain box can displace the retaining one. The fix should therefore also keep plain registrations from displacing a live entry, and let an owning green box replace a plain one.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't workingcuda.coreEverything related to the cuda.core moduletriageNeeds the team's attention

    Type

    No type

    Projects

    No projects

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions