Skip to content

feat(transfer): use native RustFS NIXL/UCX for RDMA transfers - #430

Merged
GatewayJ merged 5 commits into
rustfs:mainfrom
GatewayJ:feat/rustfs-nixl-transport
Nov 21, 2026
Merged

GatewayJ merged 5 commits into
rustfs:mainfrom
GatewayJ:feat/rustfs-nixl-transport

Conversation

@GatewayJ

@GatewayJ GatewayJ commented Nov 20, 2026 •

Copy link
Copy Markdown
Member

Related work

Companion server PR rustfs/rustfs#9542.

Summary

Use RustFS's native S3-over-NIXL protocol as the CLI's RDMA transport. The Linux build uses the public RustFS dependency and a checked-in lockfile. This removes the private gateway submodule and its dedicated build manifest. The regular CLI still builds independently and defaults to HTTP.

  • Build with ./scripts/build-rdma.sh --release. With a verified HTTPS alias, run rc --data-transport rdma put model.bin local/models/model.bin, or set RC_DATA_TRANSPORT=auto.
  • Sign bodyless S3 control requests, transfer object bytes through UCX, and verify the server's receipt and byte count. File transfers read and write registered windows directly, using two 1 MiB windows for small transfers or two 64 MiB windows for larger transfers. Range GETs keep the original version selector and compare each authenticated response version to HEAD under If-Match. Current-version reads require only the usual GetObject permission.
  • Publish nonempty uploads through multipart completion after RustFS confirms every part. Native writes require multipart permissions and produce multipart ETags even for a single part. The build guide covers supported options and HTTP fallback.

BREAKING for custom RDMA builds. Callers of the previous rdma/Cargo.toml should use ./scripts/build-rdma.sh. Its default release output is nixl/target/release/rc. A custom Rust host passes its ControlExecutor directly to the provider factory and calls rustfs_cli::run(). The executor returns the response status and headers.

The S3 host signs control requests on its existing HTTPS client and applies the alias connection and read timeouts. Registration and window ownership stay in the NIXL adapter. Uncertain transfers retain their exposed windows and memory budget. Waiters switch pools when all windows in a registration are quarantined.

File uploads keep two transfers in flight. Each receipt makes room for the next part, and completion sends ETags in part-number order. Native control responses and SDK operations use the same S3 object error classification.

Verification

  • Ran cargo test --workspace --locked, cargo test --manifest-path nixl/Cargo.toml --locked (including a two-agent memory transfer), and python3 scripts/tests/test_build_rdma.py. Ran formatting checks and Clippy with -D warnings for both Cargo manifests. Built the executable with ./scripts/build-rdma.sh.
  • Tested nixl/tests/interop.sh against native-enabled and HTTP-only RustFS instances built with the companion server change. Checked a 129 MiB random file byte for byte, small and empty objects, a range read, automatic HTTP fallback, and preservation of an existing object after a declined native overwrite.
  • Tested HTTP, forced native and automatic current-version reads against a versioned bucket using an account with GetObject and no GetObjectVersion permission. Tested a stalled HTTPS response with an alias read timeout of 150 ms.
  • Against the server revision before idle eviction, verified strict-native put, get, pipe, object show, remote-to-local object move, and both mirror directions by comparing the transferred files.
  • Read an explicit delete marker, the current delete marker and a missing version in HTTP, automatic and strict-native modes. All nine commands returned exit code 5 with the appropriate S3 error. Withheld the first part receipt of a 129 MiB upload to verify the third part started immediately after the second. On a part failure, verified the client waited for the outstanding receipt before aborting.
  • Added a Linux CI job that builds and tests the native binary with the NIXL revision from Cargo.lock and CPU-only UCX.
  • Interoperability ran over UCX TCP loopback. Physical RDMA bandwidth was not measured.

@GatewayJ GatewayJ changed the title feat(transfer): support native RustFS NIXL/UCX object transfers feat(transfer): use native RustFS NIXL/UCX for RDMA transfers Nov 20, 2026
@GatewayJ
GatewayJ requested a balanced review from Copilot November 21, 2026 02:40

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot was unable to review this pull request because the user who requested the review has insufficient remaining quota. Try requesting a review at a lower effort level.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

Unit coverage is missing for upload probe and fallback safety, direct-file download handling of invalid or partial output, and abort ordering while part receipts remain pending.

3 open issues

Review effort: Balanced

Comment thread crates/s3/src/client.rs
file.clone(),
)
.await;
let bytes = match result {
};
let upload = async {
if auto {
match staged.probe(path, &upload_id, part_size.min(length)).await {
Comment on lines +261 to +262
// Let already-exposed windows receive their receipts before aborting.
while in_flight.next().await.is_some() {}
@GatewayJ
GatewayJ merged commit a0abab2 into rustfs:main Nov 21, 2026
14 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants