Skip to content

Check node uniqueness in make_unique by reading the refcount - #190

Open
adamv-symbolica wants to merge 1 commit into
Adam-Vandervorst:masterfrom
adamv-symbolica:adam/make-unique-load
Open

adamv-symbolica wants to merge 1 commit into
Adam-Vandervorst:masterfrom
adamv-symbolica:adam/make-unique-load

Conversation

@adamv-symbolica

Copy link
Copy Markdown
Contributor

Stacked on #188: the first three commits are that PR's, and only the last one is new here (src/trie_node.rs, src/zipper_head.rs).

The change

TrieNodeODRc::make_unique took the refcount from 1 to 0 with a compare-exchange and stored 1 back when the node turned out to be unique. That is a locked instruction and a store to the node's first cache line every time a node the trie already owns is taken mutably. It now reads the count and clones when it isn't 1:

if unsafe{ &*ptr }.load(Acquire) != 1 { /* clone */ }

Why reading the count is enough

  • The count is the number of TrieNodeODRcs that point at the node. One is created only by new_in (a fresh node, count 1) or by clone (fetch_add), and destroyed only by drop. The fields are private to the module.
  • Nothing else touches the count: clone, drop, refcount() and make_unique are the only sites. There is no compare-exchange or "increment if non-zero" on a node's count anywhere in src, so there are no weak references and no upgrade. TaggedNodeRef, TaggedNodePtr, SlimNodePtr and TinyRefNode never touch it.
  • So a count of 1 seen through &mut self cannot rise while the borrow lasts: a new reference can only be cloned from an existing one, and the only existing one is mutably borrowed.
  • It cannot be a stale 1 either. Any clone of this reference ended its borrow before the &mut began, which is a happens-before edge, and a load cannot return a value older than a write that precedes it.
  • The load is Acquire, as the compare-exchange was. It pairs with the Release decrement in drop, so whatever a departed owner read from the node happens before the caller's write.
  • Old and new differ only when the count is 1, where the old code wrote 0 and then 1. Nobody else can observe the count in that window.

Arc::make_mut needs its compare-exchange to hold off Weak::upgrade. Arc::is_unique/get_mut read the strong count with load(Acquire), and triomphe's Arc::make_mut, which has no weak references, is written exactly like the new code.

Through a ZipperHead the exclusive access to a TrieNodeODRc comes from the path discipline rather than the borrow checker. That is unchanged, and the compare-exchange did not protect two zippers that reach the same reference either.

Tests

Two new tests take one shared node mutably from several threads at once, which no existing test did:

  • make_unique_from_many_threads: threads that each hold a reference to one node call make_mut together and write. No thread may see another's write, and no two may end up with the same node.
  • parallel_writes_to_shared_subtries_test: write zippers of a ZipperHead on separate threads, at paths that hold grafts of the same map.

Attempts to break it

Each tool was also given a control with the bug put in on purpose, load(Relaxed), to show it would notice.

What ran Result Control
loom A model of clone/drop/refcount/make_unique with the orderings as they are: 8 scenarios for the old and the new variant, searched exhaustively (up to 1,981 interleavings each) all pass rejected, causality violation
ThreadSanitizer (std rebuilt) The threaded tests 30 times, the whole lib suite, and a stress harness in 5 configurations 0 reports 7 data races
Miri, many seeds The 5 threaded tests under 528 seeds with default flags, 528 with a 10% preemption rate, and 264 under Tree Borrows all pass data race
Scale-out stress Writers over grafts of one map through a ZipperHead, with readers alongside, in 9 configurations up to 512 threads on 192 cores. Every old value is checked, everything is read back, and live allocations must return to their baseline each round all pass

With --features arena_compact,random ThreadSanitizer reports gxhash's read past short inputs, on base and head alike.

What CI runs, run by hand: cargo test --release (1,255 lib tests) and with arena_compact,random (1,421), cargo miri nextest run (1,252 of 1,252) and the doc tests under Miri, and fuzz_ab.py with 100,000 + 20,000 inputs (identical on base and head).

The loom model and the stress harness are not in this PR. I can add them if they are wanted in the repository.

Benchmarks

EPYC 9B45, bench_ab.py, 5 rounds, geomean per bench. The A/A column is #188's run of byte-identical binaries, as a measure of noise.

bench A/A default profile one codegen unit
prune −2.2% −15.3% −17.3%
write_zipper −8.2% −9.0%
binary_keys −0.1% −8.0% −7.5%
zipper_head_owned −0.3% −7.4% −5.8%
superdense_keys +1.0% −3.5% −3.3%
sparse_keys −0.1% −1.1% −1.7%
graft_masked_branches +0.3% −1.4% −0.9%

restrict, product_zipper, act_paths and oeis are flat. shakespeare, cities, pointcloud and byte_mask move as much in the A/A as here, so I read nothing into them.

89 cases are more than 5% faster in both builds. The largest, with one codegen unit:

  • set_val_at replacing a value: 43–49% faster (binary_set_val_at_short_replace/1: 19 ns → 11 ns)
  • superdense_insert: 34–48% faster; binary_insert: 10–30%; sparse_insert: 8–27%
  • Creating a zipper from a ZipperHead: 12–24% faster
  • prune: 10–23% faster per case
  • write_zipper over owned nodes: 14–24% faster

Threads (the stress harness, median of 3 runs of 15 rounds): threads that each write to nodes of their own through a ZipperHead are about 30% faster at every thread count from 1 to 192, 38 ns → 27 ns per write on one thread and 45 ns → 32 ns on 192. Over grafts of one map it is 5–18% faster at nine of the ten thread counts and within noise at the tenth.

On an M4, where a compare-exchange is cheap, the gain is smaller: write_zipper −4% (owned nodes −2 to −9%), short set_val_at −7 to −17%, prune flat over 24 rounds.

Five cases are more than 5% slower in both builds. None of them runs make_unique:

  • map_reader_is_shared (+13%, stable): the loop calls ReadZipperCore::is_shared out of line. Its code is byte-identical in both binaries but sits 224 bytes later, at a different offset in its cache line, because make_unique before it changed size.
  • sparse_k_path_iter/50, /100, /200 and shared_descend_to_rz/1 (+7 to +12%): equal on both sides when run on their own, and for sparse_k_path_iter the hot function is byte-identical and at the same address. In the full run they follow benches that this change made faster. My guess is that those run more iterations and leave the allocator in a different state; I did not confirm that.

🤖 Generated with Claude Code

make_unique took the refcount from 1 to 0 with a compare-exchange and stored 1
back when the node turned out to be unique: a locked instruction and a store
to the node's first cache line every time a node the trie already owns is
taken mutably.

Reading the count is enough.  A count of 1 seen through `&mut self` means no
other reference exists, and none can appear while the borrow lasts: a
reference is only ever cloned from an existing one, and there are no weak
references to upgrade, which is the reason `Arc::make_mut` needs the
compare-exchange.  `Arc::get_mut` checks its strong count the same way, and
triomphe's `Arc::make_mut`, which has no weak references either, is written
exactly like this.  The load stays `Acquire`, as the compare-exchange was, so
it still synchronizes with the `Release` decrement of the references that
were dropped.

On an EPYC 9B45 (one codegen unit, 5 rounds): prune -17%, write_zipper -9%,
binary_keys -7.5% with set_val_at replacing a value 43-49% faster,
superdense_insert 34-48% and sparse_insert 8-27% faster, creating a zipper
from a ZipperHead 12-24% faster.  Threads that each write to nodes of their
own through a ZipperHead are about 30% faster at every thread count from 1 to
192: 38 ns to 27 ns per write on one thread, 45 ns to 32 ns on 192.

Two tests take one shared node mutably from several threads at once, which no
existing test did: one on the node references themselves, one through write
zippers of a ZipperHead at paths that hold grafts of the same map.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant