Repository navigation
Check node uniqueness in make_unique by reading the refcount - #190
Open
adamv-symbolica wants to merge 1 commit into
Open
adamv-symbolica wants to merge 1 commit into
adamv-symbolica wants to merge 1 commit into
Conversation
make_unique took the refcount from 1 to 0 with a compare-exchange and stored 1 back when the node turned out to be unique: a locked instruction and a store to the node's first cache line every time a node the trie already owns is taken mutably. Reading the count is enough. A count of 1 seen through `&mut self` means no other reference exists, and none can appear while the borrow lasts: a reference is only ever cloned from an existing one, and there are no weak references to upgrade, which is the reason `Arc::make_mut` needs the compare-exchange. `Arc::get_mut` checks its strong count the same way, and triomphe's `Arc::make_mut`, which has no weak references either, is written exactly like this. The load stays `Acquire`, as the compare-exchange was, so it still synchronizes with the `Release` decrement of the references that were dropped. On an EPYC 9B45 (one codegen unit, 5 rounds): prune -17%, write_zipper -9%, binary_keys -7.5% with set_val_at replacing a value 43-49% faster, superdense_insert 34-48% and sparse_insert 8-27% faster, creating a zipper from a ZipperHead 12-24% faster. Threads that each write to nodes of their own through a ZipperHead are about 30% faster at every thread count from 1 to 192: 38 ns to 27 ns per write on one thread, 45 ns to 32 ns on 192. Two tests take one shared node mutably from several threads at once, which no existing test did: one on the node references themselves, one through write zippers of a ZipperHead at paths that hold grafts of the same map. Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
adamv-symbolica
force-pushed
the
adam/make-unique-load
branch
from
November 17, 2026 16:25
edfb431 to
04c0231
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Stacked on #188: the first three commits are that PR's, and only the last one is new here (
src/trie_node.rs,src/zipper_head.rs).The change
TrieNodeODRc::make_uniquetook the refcount from 1 to 0 with a compare-exchange and stored 1 back when the node turned out to be unique. That is a locked instruction and a store to the node's first cache line every time a node the trie already owns is taken mutably. It now reads the count and clones when it isn't 1:Why reading the count is enough
TrieNodeODRcs that point at the node. One is created only bynew_in(a fresh node, count 1) or byclone(fetch_add), and destroyed only bydrop. The fields are private to the module.clone,drop,refcount()andmake_uniqueare the only sites. There is no compare-exchange or "increment if non-zero" on a node's count anywhere insrc, so there are no weak references and no upgrade.TaggedNodeRef,TaggedNodePtr,SlimNodePtrandTinyRefNodenever touch it.&mut selfcannot rise while the borrow lasts: a new reference can only be cloned from an existing one, and the only existing one is mutably borrowed.&mutbegan, which is a happens-before edge, and a load cannot return a value older than a write that precedes it.Acquire, as the compare-exchange was. It pairs with theReleasedecrement indrop, so whatever a departed owner read from the node happens before the caller's write.Arc::make_mutneeds its compare-exchange to hold offWeak::upgrade.Arc::is_unique/get_mutread the strong count withload(Acquire), and triomphe'sArc::make_mut, which has no weak references, is written exactly like the new code.Through a
ZipperHeadthe exclusive access to aTrieNodeODRccomes from the path discipline rather than the borrow checker. That is unchanged, and the compare-exchange did not protect two zippers that reach the same reference either.Tests
Two new tests take one shared node mutably from several threads at once, which no existing test did:
make_unique_from_many_threads: threads that each hold a reference to one node callmake_muttogether and write. No thread may see another's write, and no two may end up with the same node.parallel_writes_to_shared_subtries_test: write zippers of aZipperHeadon separate threads, at paths that hold grafts of the same map.Attempts to break it
Each tool was also given a control with the bug put in on purpose,
load(Relaxed), to show it would notice.clone/drop/refcount/make_uniquewith the orderings as they are: 8 scenarios for the old and the new variant, searched exhaustively (up to 1,981 interleavings each)ZipperHead, with readers alongside, in 9 configurations up to 512 threads on 192 cores. Every old value is checked, everything is read back, and live allocations must return to their baseline each roundWith
--features arena_compact,randomThreadSanitizer reports gxhash's read past short inputs, on base and head alike.What CI runs, run by hand:
cargo test --release(1,255 lib tests) and witharena_compact,random(1,421),cargo miri nextest run(1,252 of 1,252) and the doc tests under Miri, andfuzz_ab.pywith 100,000 + 20,000 inputs (identical on base and head).The loom model and the stress harness are not in this PR. I can add them if they are wanted in the repository.
Benchmarks
EPYC 9B45,
bench_ab.py, 5 rounds, geomean per bench. The A/A column is #188's run of byte-identical binaries, as a measure of noise.restrict,product_zipper,act_pathsandoeisare flat.shakespeare,cities,pointcloudandbyte_maskmove as much in the A/A as here, so I read nothing into them.89 cases are more than 5% faster in both builds. The largest, with one codegen unit:
set_val_atreplacing a value: 43–49% faster (binary_set_val_at_short_replace/1: 19 ns → 11 ns)superdense_insert: 34–48% faster;binary_insert: 10–30%;sparse_insert: 8–27%ZipperHead: 12–24% fasterprune: 10–23% faster per casewrite_zipperover owned nodes: 14–24% fasterThreads (the stress harness, median of 3 runs of 15 rounds): threads that each write to nodes of their own through a
ZipperHeadare about 30% faster at every thread count from 1 to 192, 38 ns → 27 ns per write on one thread and 45 ns → 32 ns on 192. Over grafts of one map it is 5–18% faster at nine of the ten thread counts and within noise at the tenth.On an M4, where a compare-exchange is cheap, the gain is smaller:
write_zipper−4% (owned nodes −2 to −9%), shortset_val_at−7 to −17%,pruneflat over 24 rounds.Five cases are more than 5% slower in both builds. None of them runs
make_unique:map_reader_is_shared(+13%, stable): the loop callsReadZipperCore::is_sharedout of line. Its code is byte-identical in both binaries but sits 224 bytes later, at a different offset in its cache line, becausemake_uniquebefore it changed size.sparse_k_path_iter/50,/100,/200andshared_descend_to_rz/1(+7 to +12%): equal on both sides when run on their own, and forsparse_k_path_iterthe hot function is byte-identical and at the same address. In the full run they follow benches that this change made faster. My guess is that those run more iterations and leave the allocator in a different state; I did not confirm that.🤖 Generated with Claude Code