Skip to content

Support RDMA for FlatBuffers RPC - #3746

Open
chenBright wants to merge 4 commits into
apache:masterfrom
chenBright:fb-rdma
Open

chenBright wants to merge 4 commits into
apache:masterfrom
chenBright:fb-rdma

Conversation

@chenBright

Copy link
Copy Markdown
Contributor

What problem does this PR solve?

Issue Number: resolve

Problem Summary:

The RDMA transport rejects fb_rpc. FlatBuffers also requires contiguous storage, which can exceed the RDMA memory pool's 2 MiB block limit.

What is changed and the side effects?

Changed:

  • Allow fb_rpc over RDMA when FlatBuffers support is enabled.
  • Register larger contiguous buffers separately for message construction and receive-side reassembly, with cleanup on failure and final reference release.

Side effects:

  • Performance effects:

  • Breaking backward compatibility:


Check List:

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

The change still has two active moderate issues that make the standard FlatBuffers test suite exit or fail on runners without RDMA hardware instead of skipping hardware-dependent coverage.

2 open issues
What changed in this PR

This pull request enables FlatBuffers RPC (fb_rpc) over RDMA and supports messages larger than the RDMA pool's 2 MiB limit through separately registered contiguous storage.

Changes:

  • Adds fb_rpc to RDMA-supported protocols when FlatBuffers is enabled.
  • Registers, retains, and cleans up oversized FlatBuffers builder and receive buffers.
  • Adds protocol, fallback, TCP, and RDMA integration tests.
File Description
test/​brpc_rdma_unittest.cpp Extends RDMA protocol-admission coverage to verify support for fb_rpc.
test/​brpc_flatbuffers_protocol_unittest.cpp Adds FlatBuffers TCP fallback, RDMA round-trip, large-payload, and transport tests.
src/​brpc/​rdma/​rdma_helper.cpp Allows fb_rpc as an RDMA-supported protocol when FlatBuffers is enabled.
src/​brpc/​flatbuffers/​message.cpp Adds allocation and registration of oversized contiguous FlatBuffers storage with safe cleanup and reference lifetime management.

Review effort: Lite

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Needs a closer look

Real RDMA hardware and runtime validation are required to verify negotiation, SGE lkeys, fallback behavior, and asynchronous registered-memory cleanup, which static review cannot establish.

No open issues 🎉

2 resolved since last review

Review effort: Lite

@chenBright

Copy link
Copy Markdown
Contributor Author

@wwbmmm PTAL.

@wwbmmm wwbmmm left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[replied by brpc-oncall robot] The test restructuring looks good, but the pool-vs-registered-memory fallback in AllocateStorage can abort the process in recoverable situations, and the new RDMA+FlatBuffers code branches are still never compiled by CI.

Comment thread src/brpc/flatbuffers/message.cpp Resolved
Comment thread src/brpc/rdma/rdma_helper.cpp Resolved
@chenBright
chenBright requested a review from wwbmmm November 17, 2026 09:20
@chenBright chenBright added the status/awaiting-maintainer Review or an author response needs a maintainer decision, verification, or approval. label Nov 18, 2026
@wasphin
wasphin requested a balanced review from Copilot November 19, 2026 05:56

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Needs a closer look

A critical teardown-order bug can crash at exit when a namespace-scope Message deregisters RDMA storage after GlobalRelease() has deleted the required mutex.

No open issues 🎉

1 previously missed issue

In code that hasn't changed since last review

High severity Avoid locking deleted RDMA mutex during global teardown

src/​brpc/​flatbuffers/​message.cpp:59

A namespace-scope Message constructed before RDMA initialization can crash at exit if it later owns this separately registered storage. GlobalRelease() runs first and deletes the mutex that DeregisterMemoryForRdma() unconditionally locks. Guard deregistration with rdma::IsRdmaAvailable() so the deleter only frees the buffer after global teardown has already deregistered it.

Review effort: Balanced

@chenBright chenBright added status/ready-to-merge Required approvals and checks are satisfied, with no blocking feedback remaining and removed status/awaiting-maintainer Review or an author response needs a maintainer decision, verification, or approval. labels Nov 19, 2026
@chenBright

Copy link
Copy Markdown
Contributor Author

@wwbmmm @wasphin PTAL.

@wwbmmm

wwbmmm commented Nov 21, 2026

Copy link
Copy Markdown
Contributor

[replied by brpc-oncall robot] Agree with the teardown-order issue above; it should be fixed before merge, so I'm withdrawing my earlier LGTM.

Guarding the deregistration with rdma::IsRdmaAvailable() is the right fix. GlobalRelease() stores g_rdma_available = false as its first step and then deregisters every MR in g_user_mrs itself before deleting the map and the mutex, and GlobalDisableRdma() is LOG(FATAL), so teardown is the only transition to "unavailable" — skipping DeregisterMemoryForRdma() there cannot leak a registration.

Please check the flag inside the deleter (at release time, as in free_storage) rather than at allocation time: storage is normally released long before teardown, and PoolExhaustionFallsBackToRegisteredStorage already relies on the real deregistration path to observe GetLKey(data) == 0 after the last reference is dropped, so that path must stay intact.

@chenBright

Copy link
Copy Markdown
Contributor Author

[replied by brpc-oncall robot] Agree with the teardown-order issue above; it should be fixed before merge, so I'm withdrawing my earlier LGTM.

Guarding the deregistration with rdma::IsRdmaAvailable() is the right fix. GlobalRelease() stores g_rdma_available = false as its first step and then deregisters every MR in g_user_mrs itself before deleting the map and the mutex, and GlobalDisableRdma() is LOG(FATAL), so teardown is the only transition to "unavailable" — skipping DeregisterMemoryForRdma() there cannot leak a registration.

Please check the flag inside the deleter (at release time, as in free_storage) rather than at allocation time: storage is normally released long before teardown, and PoolExhaustionFallsBackToRegisteredStorage already relies on the real deregistration path to observe GetLKey(data) == 0 after the last reference is dropped, so that path must stay intact.

@wwbmmm Fixed in aff9c68. In the RDMA helper, A process-lifetime mutex now synchronizes MR registration and deregistration with global cleanup. A Message destroyed after cleanup can safely free its buffer. If RDMA is disabled while the registry is still live, the last reference still deregisters the buffer.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

status/ready-to-merge Required approvals and checks are satisfied, with no blocking feedback remaining

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants