MetaDB is a high-performance, 100% pure-Rust, file-system structure-aware distributed transactional metadata database engine built for RustFS.
- Implementation guides (what this tree runs): English | Français | Deutsch | 日本語 | 简体中文
- Earlier operator guides: Documentation Hub — English | 日本語 | Français | 简体中文 | Deutsch
- Architecture & Engineering:
Architecture.md|docs/DEVELOPMENT_PLAN.md|docs/designs/SITEMAP.md|docs/designs/review_report.md
Academic & Architectural Attribution: The architecture of
rustfs/MetaDBis inspired by and adapted from Baidu's trillion-scale production metadata center paper: Zebin Ren (@ZebinRen), Biao Cao, Kang Chen (@kangchenPKU), Zhihui Yin, Qiushi Chen, Jielong Jian, Jiwei Xiong, Xinxing Wang, Yan Xing, Liguo Duan, Ran Zheng, Xianjun Meng, and Zhenyu Hou, "MetaDB: Putting File-System Structure Back into Distributed Databases", arXiv:2610.32247v1 (Vrije Universiteit Amsterdam, Baidu & Peking University, 2026). (Note: Trillion-record production figures cited fromarXiv:2610.32247v1are Baidu production metrics, not results of this repository. Current Implementation Status does not record a first-launch or trillion-scale pass.)Spec-Driven Development Methodology: Designed and stress-tested as a specification with Google's Elephant-Goldfish Model (EGM). That review is not a first-launch or scale-gate pass.
General-purpose distributed databases (such as TiDB, CockroachDB, and FoundationDB) treat file-system and object-storage metadata as generic flat key-value or relational rows. The three bottlenecks below are Baidu production observations reported in arXiv:2610.32247v1. None of the ratios or latencies is a measurement of this repository.
- Data Model Mismatch: The paper reports that flat hash or byte-midpoint sharding turns
>96%of localized single-directory operations (create,mkdir,unlink,listdir) into multi-shard Two-Phase Commit (2PC). That percentage is Baidu's workload figure, not a ratio counted in this tree. - Transaction Model Mismatch: The paper reports that synchronously locking the shared parent-directory attribute record collapses concurrent single-directory write throughput by
10x–36x(including a36.2xgap versus TiDB in that study). SynchronousQuotaTableandSecondaryIndexTableupdates are the paper's description of extra cross-shard 2PC, not a benchmark of this repository. - Storage Implementation Mismatch: The paper reports a double-tombstone penalty for short-lived (
<10ms) metadata transactions, andlistdirlatency moving from<100msto~6 secondsunder bulk TTL deletion (arXiv:2610.32247v1, §2.3.3 and §7.4). Those latencies are not measurements of this repository.
- Colocated Schema (
(PID, "_attr_")):MetaTableis keyed by(PID, name)withgroup_key = PID. A directory's own attributes are stored in a reserved child entry(PID, "_attr_")physically adjacent to its direct children(PID, child_name). - Epoch-Fenced
PID-Boundary Sharding & Jumbo Defense: Shards split strictly alonggroup_key = PIDboundaries underShardEpochfencing. In this tree, jumbo isolation is aJumboIsolatedroute flag. It does not migrate peer shards to a dedicated host.DualModeFineGrained(PID, name)splitting uses shard-localDirShardDeltarecords for1-RTTwrites and1-hopscatter-gatherdirstat. - Single-Shard Pushdown + Crash-Recoverable 2PC: Executes single-directory Read-Modify-Write (
create,unlink,setattr,lookup,dirstat,listdir, and colocatedmkdir/rmdir) in1 RTTon the shard leader, reserving Primary-Anchored 2PC (WriteIntent+TxnRecord) for cross-shardrenameand cross-shardmkdir/rmdir. The<3%share of cross-directoryrenameis a Baidu paper workload figure (arXiv:2610.32247v1, Table 2), not a benchmark of this repository.
- Conflict-Free Asynchronous Attribute Aggregation: Removes parent directory attribute locks from the foreground write path. Child mutations append deterministic deltas to
AsyncAttrAggregator, folded into(PID, "_attr_")at the L0 SSTable Coordinated Flush Barrier while Read-Path Reconciliation guarantees linearizabledirstatreads. - Exact-Once Asynchronous
AuxOutbox: PropagatesQuotaTableandSecondaryIndexTable(FID -> (PID, name)) updates in strict FIFO per-shard LSN batches with[0x07]local overflow spill so Raft log compaction is never blocked by remote partitions.
- WAL-less LSM + 3-Watermark OpenRaft: Disables local KV WAL files in favor of segmented CRC32C Raft logs (
openraft), clamping log purge tomin(state_flushed_index, aux_outbox_durable_index, last_snapshot_index). SnapshotGuardIn-Memory MVCC: Retains multi-version chains in memory for the5stransaction window and gates an unversioned L0 collapse-flush onActiveSnapshotRegistry(min_active_snapshot_hlc >= imm.max_hlc). That flush may keep only the latest value of a key. An open 2PC lease also stores the value captured atstart_hlcunder prefix0x0Din the same flush. A point read behind the flush watermark returns that capture, orTransactionTimeoutwhen it was not proved, and does not return a newer collapsed value. A range read behind the flush watermark returnsTransactionTimeoutfor the whole range. The~60%fewer KV writes figure is a paper/model claim fromarXiv:2610.32247v1, not a measurement of this repository. The same range-read limit is in docs/en/architecture.md.- Bottommost-Safe Locality-Aware Compaction: Tracks tombstone density per 9-byte prefix bucket and compacts hot deletion ranges down to the deepest overlapping SSTable level. A flat
listdirP99 under<4msis a design target inarXiv:2610.32247v1, not a measurement of this repository.
The design keeps one StorageBackend and one openraft state machine from an embedded single-binary node toward a multi-Cell cluster. Stage 3 is a design target. This binary does not deploy it.
- Stage 1 — Single-Node All-in-One (
1-VoterOpenRaft): Zero network coordination; sequential Raft log persistence withmetadbd --role all-in-oneor the persistent APIMetaDbEngine::open/PersistentMetaDbCluster.MetaDbEngine::new_single_nodeis a separate in-memory simulator: no data directory, no OpenRaft group, and no WAL-less LSM. - Stage 2 — Dynamic Multi-Shard Split: Splits shards along
group_key = PIDboundaries on a single host. - Stage 3 — Multi-Cell Distributed Cluster (design only): The design describes
AddLearner, SSTable snapshot catch-up, joint-consensus voter promotion, decentralizedHLCclocks, statelessMetaCache, and a two-tier control plane.be,proxy,cell-manager, andglobal-managerstart on a valid config. That startup is not a production MetaCache of every shard and not a disaggregated production deployment.
metadbd today runs Stage 1, plus in-process PID-boundary splits (Stage 2) on that same process. open_and_serve starts all-in-one, be, proxy, cell-manager, and global-manager when that role's configuration is valid. A second metadbd --role all-in-one joins the same cell when --join or --cell-manager names the current leader and the data directory is fresh (crates/metadb-server/tests/join_existing_group.rs). Join still has to reach that leader; a follower does not forward it. Without that join, a new data directory bootstraps its own cell. RegisterCell records another cell in GLOBAL_CELL_DIRECTORY after its routes cover the requested range, and a later request contained by that cell is forwarded. A three-metadbd voter rehearsal is not in this tree. See docs/en/operations.md.
| Crate | In this tree | Responsibility |
|---|---|---|
crates/metadb-core |
Present (phase label 1) | 0x01..0x07 CRC32C binary codecs, (PID, "_attr_") keys, ShardEpoch, CompositeKeyRange, HLC (+ drift/overflow guards). |
crates/metadb-proto |
Present (phase label 1) | Protobuf v1 schemas & tonic/prost gRPC contracts (MetadataService, TransactionService, RaftTransportService, ControlPlaneService). |
crates/metadb-storage |
Present (phase label 2) | StorageBackend trait, pure-Rust WAL-less SSTable LSM, MANIFEST, 16-shard LRU BlockCache, Bloom filter, SnapshotGuard ReadCut MVCC, LocalityCompactor. |
crates/metadb-raft |
Present (phase label 3) | openraft 0.10 Multi-Raft, segmented CRC32C .raftlog, 3-watermark purge gate, batched ReadIndex, SSTable snapshots. In-process 1->3 voter coverage is not a three-VM rehearsal. |
crates/metadb-tx |
Present (phase label 4) | Deterministic TxStateMachineHandler, 1-RTT pushdown, ReplicatedAttrAggregator, contiguous ShardAuxOutbox + [0x07] spill, crash-recoverable 2PC with in-flight lease renewal. |
crates/metadb-cluster |
Present (phase label 5) | 3-stage ExecuteSplit (PID & DualModeFineGrained), ShardEpoch fencing, a JumboIsolated route flag, DurableGlobalManager, DurableCellManager, CompositeMetaCache, DurableIdAllocator. |
crates/metadb-proxy |
Present (phase label 5) | Tokenizing SqlLexer & compiler for all 20 parameterized metadata SQL templates + StatelessProxyRouter. |
crates/metadb-server |
Present (phase label 6) | PersistentMetaDbCluster & MetaDbEngine SDKs + metadbd daemon + metadb-ctl operator CLI + Prometheus metrics. metadbd serves all-in-one, be, proxy, cell-manager, and global-manager when that role's configuration is valid. |
Phase labels below mean the crates and the tests described in docs/en/testing.md are in this tree. They do not mean a production MetaCache of every shard or a disaggregated production deployment, and they do not mean a first-launch or scale-gate pass. Trillion-scale figures in arXiv:2610.32247v1 are Baidu production metrics, not results of this repository. The listdir P99 <4ms figure is a design target from that paper, not a measurement of this tree. Applied memtable settings, in-flight 2PC lease renewal, and the seeded simulation catalog are on this tree. They are not Baidu production measurements.
- Phases 0–6 are in this tree; they are not a first-launch or scale-gate verification:
- Phase 0: Master
Architecture.md, EGM subsystem specs, zero-context Goldfish review & resolutions (B1–B18,N1–N5),DEVELOPMENT_PLAN.md, and 5-language usage guides (en,ja,fr,zh,de). The Goldfish review checked the specification. It is not a runtime acceptance test. - Phases 1–6: The 8 crates are a pure-Rust implementation (
zero C/C++ FFI). The commands checked into the workspace arecargo fmt --all -- --check,cargo clippy --workspace --all-targets -- -D warnings, andcargo test --workspace. Those suites include fault-injection coverage for SSTable/MANIFESTcrash recovery,SnapshotGuardReadCutisolation,LocalityCompactor, in-process1-Voter -> 3-Votersnapshot promotion and leader failover,[0x07]outbox spill/cleanup, 2PC crash recovery (including lease renewal), 3-wayDualModeFineGrainedscatter-gather, all 20 SQL templates, and the gRPC tests listed in docs/en/testing.md.crates/metadb-storage/tests/first_ship_storage_regressions.rscovers live-rowlistdirlimit pushdown and density rebuild on open. That file is not a launch pass. Process-restartlistdirand multi-process 3-voter failoverlistdirwithentries_examinedassertions are not in this tree.
- Phase 0: Master
The toolchain is pinned to Rust 1.100.0 in rust-toolchain.toml (rustfmt, clippy). Crate edition stays 2021.
cargo fmt --all -- --check
cargo clippy --workspace --all-targets -- -D warnings
cargo test --workspaceOperator and developer guides in English, French, German, Japanese, and Simplified Chinese are under docs/en/ (and docs/fr, docs/de, docs/ja, docs/zh).
cargo test --workspace includes the persistent topology suites and the stress smoke. T1–T6 still exercise the in-memory MetaDbEngine::new_single_node simulator. The persistent suites open a real data_dir through PersistentMetaDbCluster / DurableCellManager.
.github/workflows/ci.yml lists those three commands with --locked (tests also pass --no-fail-fast). This README does not say that workflow is enabled. See CI and release packages.
| Suite | Command |
|---|---|
| Single-node metadata, shutdown, and reopen | cargo test -p metadb-server --test topology_single_node |
| Single-node shard split and keyspace coverage | cargo test -p metadb-server --test topology_scale_out |
| 3-voter membership, metadata, leader change, and voter restart | cargo test -p metadb-server --test topology_distributed |
| Unflushed restart overlapping in-flight creates | cargo test -p metadb-server --test stress_crash_restart -- --nocapture |
| 3-voter leader loss and restart under create load | cargo test -p metadb-server --test topology_distributed three_voter_leader_loss_under_load -- --nocapture |
| Stress smoke (5s mixed workload, correctness, latency) | cargo test -p metadb-server --test stress_harness stress_smoke -- --nocapture |
The reusable harness is metadb_server::stress. The smoke profile is fixed (5 seconds, 4 clients, 4 directories) and does not read the environment, so CI stays bounded. The local profile is ignored by cargo test --workspace:
# Defaults: 30s, concurrency 8, 16 directories.
METADB_STRESS_DURATION_SECS=120 \
METADB_STRESS_CONCURRENCY=16 \
METADB_STRESS_KEYSPACE=32 \
METADB_STRESS_SEED=7 \
cargo test -p metadb-server --test stress_harness stress_local -- --ignored --nocapture
# Same harness inside a temporary all-in-one process:
cargo run -p metadb-server --bin metadb-ctl -- bench --embedded --profile smoke
cargo run -p metadb-server --bin metadb-ctl -- bench --embedded --profile localMETADB_STRESS_DURATION_SECS, METADB_STRESS_CONCURRENCY, METADB_STRESS_KEYSPACE, and METADB_STRESS_SEED apply to the local profile and to metadb-ctl bench --embedded --profile local. Explicit --duration-secs, --concurrency, --keyspace, and --seed override the profile.
stress_crash_restart stops a single-node process with PersistentMetaDbCluster::shutdown_without_flush while creates are still inside Raft propose, reopens the same data_dir, and checks listings, dirstat, quota, and a secondary index. A second stop covers a joined rename, unlink, and create tail. three_voter_leader_loss_under_load runs creates on a 3-voter cell, stops the leader without flushing, checks the new leader, and restarts the stopped voter while creates continue.
metadb-ctl bench --workload hotspot-create and --workload bulk-ttl-delete dial a remote gRPC endpoint and print JSON: a header, then phase reports with attempted, succeeded, failed, and latency percentiles. The process exits non-zero unless every requested operation succeeds. --workload mixed and --embedded run the in-process stress harness. Numbers from that command are for the machine that ran it; this repository does not record them as a result.
metadbd starts all-in-one, be, proxy, cell-manager, and global-manager when that role's configuration is valid. proxy dials the registered cell that contains a metadata call, ExecuteSql, or a user-key transaction, and fails closed otherwise. Those calls are not sent to [cluster] upstream. CommitPrimary2pc, ApplyAuxBatch, ForwardRouter, and GetRoutes may still use that upstream. AllocateIdBlock is refused. This is not a production MetaCache of every shard. topology_distributed still builds a 3-voter cell in-process. A second OS process can join that cell's Raft groups: cargo test -p metadb-server --test join_existing_group. [cluster] join does not create a second cell. RegisterCell records another cell in GLOBAL_CELL_DIRECTORY after its routes cover the requested range, and a later request contained by that cell is forwarded. The checked-in ci.yml lists cargo fmt, cargo clippy --locked, and cargo test --locked. Job rust-check-lint-test has timeout-minutes: 90. Job package has timeout-minutes: 45. The file sets cancel-in-progress for a newer push to the same ref and rejects a bare #[ignore] attribute. This README does not say that workflow is enabled.
.github/workflows/ci.yml is the test and package workflow file. The same directory also contains dependency-audit.yml and security-adversarial-regressions.yml. This section records timeouts checked into ci.yml. It does not say any workflow is enabled.
| Job | When it runs | What it does |
|---|---|---|
| Format, Clippy & Test | Every push, every pull request to main, and v* tags |
cargo fmt --all -- --check, cargo clippy --workspace --all-targets --locked -- -D warnings, cargo test --workspace --locked --no-fail-fast on ubuntu-24.04. A bare #[ignore] with no reason string fails the job. |
Package x86_64-unknown-linux-gnu |
Pull requests, pushes to main, and v* tags |
Native ubuntu-24.04 release build of metadbd and metadb-ctl. |
Package aarch64-unknown-linux-gnu |
Same as the x86_64 package job | Native ubuntu-24.04-arm release build. No cross or musl toolchain. |
| Publish GitHub Release | Push of a v* tag, after tests and both packages succeed |
Uploads the archives and checksums. contents: write is granted only on this job. |
The file sets cancel-in-progress for a newer push to the same ref. Checked-in timeout-minutes values are 90 for Format, Clippy & Test (rust-check-lint-test), 20 for the CI-scale prefix load (prefix-load-ci), 45 for each package job, and 15 for the release job.
Package artifacts (also attached to the GitHub Release for a v* tag):
metadb-<version>-x86_64-unknown-linux-gnu.tar.gzmetadb-<version>-aarch64-unknown-linux-gnu.tar.gz- a
.sha256sidecar for each archive
<version> is [workspace.package].version in Cargo.toml (currently 0.1.0). metadbd --version reports that same string. The Actions artifact name is the tarball basename, metadb-<version>-<target>. The tarball contains metadbd, metadb-ctl, LICENSE, and SHA256SUMS for those files. These are glibc builds. The workflow does not publish to crates.io. On a tag, the release also includes one SHA256SUMS covering both archives. A tag with a hyphen (for example v0.1.0-rc.1) is marked as a prerelease. The tag text is not required to equal the crate version; asset names follow Cargo.toml.
stress_local_mixed_workload stays #[ignore] and is not part of this CI job. The 5-second stress_smoke_mixed_workload still runs.
Licensed under the Apache License, Version 2.0.