Repository navigation
feat(transfer): use native RustFS NIXL/UCX for RDMA transfers - #430
Merged
Merged
Conversation
houseme
approved these changes
Nov 21, 2026
| file.clone(), | ||
| ) | ||
| .await; | ||
| let bytes = match result { |
| }; | ||
| let upload = async { | ||
| if auto { | ||
| match staged.probe(path, &upload_id, part_size.min(length)).await { |
Comment on lines
+261
to
+262
| // Let already-exposed windows receive their receipts before aborting. | ||
| while in_flight.next().await.is_some() {} |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.

Related work
Companion server PR rustfs/rustfs#9542.
Summary
Use RustFS's native S3-over-NIXL protocol as the CLI's RDMA transport. The Linux build uses the public RustFS dependency and a checked-in lockfile. This removes the private gateway submodule and its dedicated build manifest. The regular CLI still builds independently and defaults to HTTP.
./scripts/build-rdma.sh --release. With a verified HTTPS alias, runrc --data-transport rdma put model.bin local/models/model.bin, or setRC_DATA_TRANSPORT=auto.BREAKING for custom RDMA builds. Callers of the previous
rdma/Cargo.tomlshould use./scripts/build-rdma.sh. Its default release output isnixl/target/release/rc. A custom Rust host passes itsControlExecutordirectly to the provider factory and callsrustfs_cli::run(). The executor returns the response status and headers.The S3 host signs control requests on its existing HTTPS client and applies the alias connection and read timeouts. Registration and window ownership stay in the NIXL adapter. Uncertain transfers retain their exposed windows and memory budget. Waiters switch pools when all windows in a registration are quarantined.
File uploads keep two transfers in flight. Each receipt makes room for the next part, and completion sends ETags in part-number order. Native control responses and SDK operations use the same S3 object error classification.
Verification
cargo test --workspace --locked,cargo test --manifest-path nixl/Cargo.toml --locked(including a two-agent memory transfer), andpython3 scripts/tests/test_build_rdma.py. Ran formatting checks and Clippy with-D warningsfor both Cargo manifests. Built the executable with./scripts/build-rdma.sh.nixl/tests/interop.shagainst native-enabled and HTTP-only RustFS instances built with the companion server change. Checked a 129 MiB random file byte for byte, small and empty objects, a range read, automatic HTTP fallback, and preservation of an existing object after a declined native overwrite.put,get,pipe,object show, remote-to-localobject move, and bothmirrordirections by comparing the transferred files.