Skip to content

feat(ingester): Add owned series tracking to prevent false throttling… - #7758

Open
anant10 wants to merge 26 commits into
cortexproject:masterfrom
anant10:feat/owned-series-tracking
Open

anant10 wants to merge 26 commits into
cortexproject:masterfrom
anant10:feat/owned-series-tracking

Conversation

@anant10

@anant10 anant10 commented Aug 11, 2026 •

Copy link
Copy Markdown

What this PR does

Adds owned series tracking to the ingester, so per-tenant and per-instance series limits stop throttling tenants for series that have already moved to other ingesters.

The problem. A tenant's local series limit is global limit × replication factor ÷ ingesters (the tenant's shard size when shuffle sharding). When ingesters are added, or a tenant's shard grows, that limit drops at once. The series that now belong to other ingesters stay in the old ingesters' TSDB heads until head compaction, up to about two hours. PreCreation checks Head().NumSeries(), so an old ingester compares everything it still holds against the new, lower limit and rejects new series the tenant is entitled to send.

The fix. An ingester counts as owned only the series the ring currently routes to it, and with enforcement enabled the limits use that count.

How it works

  • Each active series entry stores the series' ring token, the same hash the distributor routes with, computed once when the series is first seen.
  • Ownership is derived from the ring with the same replica-set walk Ring.Get uses (replicaSetAt, extracted from Ring.Get), so it can't drift from routing. It's precomputed per ring change as sorted tokens plus one bool per position: on push, ownership is a binary search and a bool read.
  • Shuffle sharding: a tenant whose shard is smaller than the ring gets ownership computed over its own subring, with the same ShuffleShard call the distributor uses, from the ingester's own copy of the ring (built from the ring the lifecycler reads). A tenant's subring is only re-evaluated when its own members, states or tokens change.
  • Owned series run on their own loop: ownership is checked every second, and every tenant is recounted in full on the active series update period. A new series counts as owned as soon as it arrives.
  • Restarts: series rebuilt from the WAL get tracker entries and are judged against the first ring the ingester reads, so leftovers restored from the WAL don't count as owned.
  • Limits: the per-tenant count is owned + series in the head without a tracker entry, capped at the head, so in-flight and just-restored series still count. The instance max_series check uses a snapshot taken by each recount, plus every series created since, capped at the series in memory.
  • Tracker entries are kept until the head deletes their series (not the 10-minute idle timeout), so that owned also counts idle series still held. This is the feature's memory cost; see the benchmarks.

Flags (experimental, both default off)

ingester:
  # Track owned series and export cortex_ingester_owned_series. No change to limits.
  # CLI flag: -ingester.owned-series-metrics-enabled
  [owned_series_metrics_enabled: <boolean> | default = false]

  # Use the owned count for the per-tenant and instance max_series limits.
  # Requires owned_series_metrics_enabled; without it this has no effect and a warning is logged.
  # CLI flag: -ingester.owned-series-limit-enforcement-enabled
  [owned_series_limit_enforcement_enabled: <boolean> | default = false]

With both off the behaviour is identical to master. Owned tracking is independent of -ingester.active-series-metrics-enabled.

Known properties

  • Owned briefly under-counts after a ring change. An old ingester stops counting a moved series at its next ring read; the new ingester counts it when its next sample arrives. The tenant's total owned can dip for about one write interval plus retries. This only ever lowers the count, so it can't cause throttling, but a tenant-level number built from it should be smoothed.
  • Up to one second between the limit and owned catching up. After an ingester reads a new ring, its local limit changes immediately and owned follows on the next one-second check. Master has the same mismatch for up to a head compaction cycle.
  • Per-metric, per-labelset and native histogram limits are unchanged here.

Testing

Unit and integration tests cover routing equivalence (whole ring and subrings), ring and shard-size changes, WAL restarts, concurrent creates during ring changes, the instance limit snapshot, and flag validation.

We also ran it in a multi-zone test cluster under steady write load, with a whole-ring tenant and a shuffle-sharded tenant:

  • After growing the ring, every pre-existing ingester held more of the whole-ring tenant in memory than its new local limit allowed, while its owned count stayed well under it. New series were accepted on all of them.
  • After growing the sharded tenant's shard, the original members kept the tenant's series in memory but owned only the share still routed to them.
  • Across every change, owned summed over ingesters matched the tenant's series × replication factor exactly.
  • As a control, with enforcement off (limits count the head, as on master) the same situation rejected the tenants' new series with per_user_series_limit on the old ingesters.
  • With owned tracking enabled, per-ingester CPU and heap rose only marginally and push latency was unchanged.

Benchmarks

AMD EPYC 9R14. Master is 3d636b064, the merge base when the benchmarks ran; the branch was later rebased onto two unrelated upstream commits with no change to this PR's diff. ActiveSeries_UpdateExistingSeries is added by this PR, since nothing upstream covers the per-sample update of a known series; on master it ran as an equivalent temporary copy. Comparison benchmarks were interleaved master/PR in two rounds of 5 (count 10 per side); PR-only benchmarks count 6. Large-ring cases use 1,000 instances and shard sizes 10, 100, 250 and 500.

Existing benchmarks (master vs PR): the push path is faster (IngesterPush −7.8%, UpdateExistingSeries −6% serial / −16% parallel). The once-a-minute active series purge is slower (Purge_once +5.9%, Purge_twice +4.3%), from the extra per-entry state. Ring benchmarks are within ~1% overall (geomean +0.84%); the largest changes are UpdateRingState with token updates (up to +3.5%) and ShuffleShardCached (up to +2.7%).

Ingester: master vs PR (benchstat)
│ master-ingester.txt │           pr-ingester.txt           │
                                              │       sec/op        │   sec/op     vs base                │
ActiveSeriesTest_single_series/50-32                    56.23n ± 0%   55.88n ± 1%   -0.62% (p=0.000 n=10)
ActiveSeriesTest_single_series/100-32                   53.74n ± 1%   53.77n ± 1%        ~ (p=0.868 n=10)
ActiveSeriesTest_single_series/500-32                   51.77n ± 1%   52.14n ± 0%   +0.71% (p=0.008 n=10)
ActiveSeries_Purge_once-32                              111.2µ ± 3%   117.7µ ± 1%   +5.88% (p=0.002 n=10)
ActiveSeries_Purge_twice-32                             115.0µ ± 1%   120.0µ ± 1%   +4.31% (p=0.000 n=10)
IngesterPush-32                                         163.7m ± 8%   151.0m ± 3%   -7.80% (p=0.009 n=10)
ActiveSeries_UpdateExistingSeries/serial-32             66.57n ± 6%   62.39n ± 2%   -6.29% (p=0.001 n=10)
ActiveSeries_UpdateExistingSeries/parallel-32           8.069n ± 1%   6.779n ± 3%  -15.99% (p=0.000 n=10)
geomean                                                 1.908µ        1.857µ        -2.71%

                                              │ master-ingester.txt │            pr-ingester.txt             │
                                              │        B/op         │     B/op       vs base                 │
ActiveSeriesTest_single_series/50-32                  0.000 ±  0%       0.000 ±  0%       ~ (p=1.000 n=10) ¹
ActiveSeriesTest_single_series/100-32                 0.000 ±  0%       0.000 ±  0%       ~ (p=1.000 n=10) ¹
ActiveSeriesTest_single_series/500-32                 0.000 ±  0%       0.000 ±  0%       ~ (p=1.000 n=10) ¹
ActiveSeries_Purge_once-32                            16.00 ±  0%       16.00 ±  0%       ~ (p=1.000 n=10) ¹
ActiveSeries_Purge_twice-32                           16.00 ±  0%       16.00 ±  0%       ~ (p=1.000 n=10) ¹
IngesterPush-32                                     13.93Mi ± 10%     12.94Mi ± 14%       ~ (p=0.190 n=10)
ActiveSeries_UpdateExistingSeries/serial-32           0.000 ±  0%       0.000 ±  0%       ~ (p=1.000 n=10) ¹
ActiveSeries_UpdateExistingSeries/parallel-32         0.000 ±  0%       0.000 ±  0%       ~ (p=1.000 n=10) ¹
geomean                                                           ²                  -0.92%                ²
¹ all samples are equal
² summaries must be >0 to compute geomean

                                              │ master-ingester.txt │            pr-ingester.txt            │
                                              │      allocs/op      │  allocs/op    vs base                 │
ActiveSeriesTest_single_series/50-32                  0.000 ±  0%      0.000 ±  0%       ~ (p=1.000 n=10) ¹
ActiveSeriesTest_single_series/100-32                 0.000 ±  0%      0.000 ±  0%       ~ (p=1.000 n=10) ¹
ActiveSeriesTest_single_series/500-32                 0.000 ±  0%      0.000 ±  0%       ~ (p=1.000 n=10) ¹
ActiveSeries_Purge_once-32                            1.000 ±  0%      1.000 ±  0%       ~ (p=1.000 n=10) ¹
ActiveSeries_Purge_twice-32                           2.000 ±  0%      2.000 ±  0%       ~ (p=1.000 n=10) ¹
IngesterPush-32                                      28.17k ± 18%     26.01k ± 27%       ~ (p=0.171 n=10)
ActiveSeries_UpdateExistingSeries/serial-32           0.000 ±  0%      0.000 ±  0%       ~ (p=1.000 n=10) ¹
ActiveSeries_UpdateExistingSeries/parallel-32         0.000 ±  0%      0.000 ±  0%       ~ (p=1.000 n=10) ¹
geomean                                                           ²                 -0.99%                ²
¹ all samples are equal
² summaries must be >0 to compute geomean
Ingester: owned-series benchmarks (PR only)
│ pr-owned-ingester.txt │
                                                                                 │        sec/op         │
ActiveSeries_UpdateSeries_Owned/ring_not_loaded-32                                          477.5n ± 14%
ActiveSeries_UpdateSeries_Owned/ring_loaded-32                                              476.2n ±  2%
ActiveSeries_UpdateMetrics/ring_unchanged-32                                                2.553m ±  3%
ActiveSeries_UpdateMetrics/ring_changed-32                                                  3.552m ±  1%
ActiveSeries_OwnedAndTracked-32                                                            0.9033n ±  0%
ActiveSeries_DeleteSeries-32                                                                33.97m ±  5%
ActiveSeries_MemoryPerEntry/owned_tracking_disabled-32                                      428.4m ±  2%
ActiveSeries_MemoryPerEntry/owned_tracking_enabled-32                                       430.5m ±  2%
Ingester_OwnedSeriesRecount/whole_ring/tenants=3/idle-32                                    127.0µ ±  1%
Ingester_OwnedSeriesRecount/whole_ring/tenants=3/state_change-32                            219.1m ±  2%
Ingester_OwnedSeriesRecount/whole_ring/tenants=3/topology_change-32                         417.7m ±  2%
Ingester_OwnedSeriesRecount/whole_ring/tenants=3/full_recount-32                            260.9µ ±  1%
Ingester_OwnedSeriesRecount/sharded/shard_size=100/tenants=1/idle-32                        141.4µ ±  1%
Ingester_OwnedSeriesRecount/sharded/shard_size=100/tenants=1/state_change-32                219.7m ±  3%
Ingester_OwnedSeriesRecount/sharded/shard_size=100/tenants=1/topology_change-32             423.0m ±  3%
Ingester_OwnedSeriesRecount/sharded/shard_size=100/tenants=1/full_recount-32                184.0µ ±  1%
Ingester_OwnedSeriesRecount/sharded/shard_size=100/tenants=10/idle-32                       269.4µ ±  3%
Ingester_OwnedSeriesRecount/sharded/shard_size=100/tenants=10/state_change-32               218.3m ±  2%
Ingester_OwnedSeriesRecount/sharded/shard_size=100/tenants=10/topology_change-32            509.4m ±  2%
Ingester_OwnedSeriesRecount/sharded/shard_size=100/tenants=10/full_recount-32               759.6µ ±  1%
Ingester_OwnedSeriesRecount/sharded/shard_size=100/tenants=50/idle-32                       825.7µ ±  2%
Ingester_OwnedSeriesRecount/sharded/shard_size=100/tenants=50/state_change-32               219.2m ±  3%
Ingester_OwnedSeriesRecount/sharded/shard_size=100/tenants=50/topology_change-32            783.1m ±  3%
Ingester_OwnedSeriesRecount/sharded/shard_size=100/tenants=50/full_recount-32               3.920m ±  4%
IngesterPush_OwnedSeries/owned_series_disabled/existing_series-32                           4.052m ±  3%
IngesterPush_OwnedSeries/owned_series_disabled/new_series-32                                43.58m ±  9%
IngesterPush_OwnedSeries/owned_series_enabled/existing_series-32                            3.921m ± 11%
IngesterPush_OwnedSeries/owned_series_enabled/new_series-32                                 41.23m ±  8%
IngesterPush_OwnedSeries/owned_series_enforced/existing_series-32                           3.993m ±  8%
IngesterPush_OwnedSeries/owned_series_enforced/new_series-32                                41.86m ±  5%
geomean                                                                                     4.205m

                                                                                 │ pr-owned-ingester.txt │
                                                                                 │         B/op          │
ActiveSeries_UpdateSeries_Owned/ring_not_loaded-32                                         235.5 ± 25%
ActiveSeries_UpdateSeries_Owned/ring_loaded-32                                             238.0 ±  1%
ActiveSeries_UpdateMetrics/ring_unchanged-32                                               0.000 ±  0%
ActiveSeries_UpdateMetrics/ring_changed-32                                                 0.000 ±  0%
ActiveSeries_OwnedAndTracked-32                                                            0.000 ±  0%
ActiveSeries_DeleteSeries-32                                                               0.000 ±  0%
ActiveSeries_MemoryPerEntry/owned_tracking_disabled-32                                   226.7Mi ±  0%
ActiveSeries_MemoryPerEntry/owned_tracking_enabled-32                                    234.3Mi ±  0%
Ingester_OwnedSeriesRecount/whole_ring/tenants=3/idle-32                                 18.52Ki ± 13%
Ingester_OwnedSeriesRecount/whole_ring/tenants=3/state_change-32                         5.055Mi ±  0%
Ingester_OwnedSeriesRecount/whole_ring/tenants=3/topology_change-32                      108.4Mi ±  7%
Ingester_OwnedSeriesRecount/whole_ring/tenants=3/full_recount-32                         21.39Ki ± 24%
Ingester_OwnedSeriesRecount/sharded/shard_size=100/tenants=1/idle-32                     23.31Ki ± 12%
Ingester_OwnedSeriesRecount/sharded/shard_size=100/tenants=1/state_change-32             9.602Mi ± 47%
Ingester_OwnedSeriesRecount/sharded/shard_size=100/tenants=1/topology_change-32          116.4Mi ±  7%
Ingester_OwnedSeriesRecount/sharded/shard_size=100/tenants=1/full_recount-32             25.24Ki ± 13%
Ingester_OwnedSeriesRecount/sharded/shard_size=100/tenants=10/idle-32                    54.02Ki ± 10%
Ingester_OwnedSeriesRecount/sharded/shard_size=100/tenants=10/state_change-32            14.16Mi ± 32%
Ingester_OwnedSeriesRecount/sharded/shard_size=100/tenants=10/topology_change-32         139.6Mi ±  8%
Ingester_OwnedSeriesRecount/sharded/shard_size=100/tenants=10/full_recount-32            88.21Ki ± 11%
Ingester_OwnedSeriesRecount/sharded/shard_size=100/tenants=50/idle-32                    184.8Ki ±  9%
Ingester_OwnedSeriesRecount/sharded/shard_size=100/tenants=50/state_change-32            21.05Mi ± 11%
Ingester_OwnedSeriesRecount/sharded/shard_size=100/tenants=50/topology_change-32         198.1Mi ±  3%
Ingester_OwnedSeriesRecount/sharded/shard_size=100/tenants=50/full_recount-32            420.6Ki ± 19%
IngesterPush_OwnedSeries/owned_series_disabled/existing_series-32                        56.25Ki ± 45%
IngesterPush_OwnedSeries/owned_series_disabled/new_series-32                             15.02Mi ±  4%
IngesterPush_OwnedSeries/owned_series_enabled/existing_series-32                         21.83Ki ± 43%
IngesterPush_OwnedSeries/owned_series_enabled/new_series-32                              14.91Mi ±  1%
IngesterPush_OwnedSeries/owned_series_enforced/existing_series-32                        20.77Ki ± 88%
IngesterPush_OwnedSeries/owned_series_enforced/new_series-32                             14.97Mi ±  5%
geomean                                                                                                ¹
¹ summaries must be >0 to compute geomean

                                                                                 │ pr-owned-ingester.txt │
                                                                                 │       allocs/op       │
ActiveSeries_UpdateSeries_Owned/ring_not_loaded-32                                         2.000 ±  0%
ActiveSeries_UpdateSeries_Owned/ring_loaded-32                                             2.000 ±  0%
ActiveSeries_UpdateMetrics/ring_unchanged-32                                               0.000 ±  0%
ActiveSeries_UpdateMetrics/ring_changed-32                                                 0.000 ±  0%
ActiveSeries_OwnedAndTracked-32                                                            0.000 ±  0%
ActiveSeries_DeleteSeries-32                                                               0.000 ±  0%
ActiveSeries_MemoryPerEntry/owned_tracking_disabled-32                                    2.016M ±  0%
ActiveSeries_MemoryPerEntry/owned_tracking_enabled-32                                     3.016M ±  0%
Ingester_OwnedSeriesRecount/whole_ring/tenants=3/idle-32                                   2.000 ± 50%
Ingester_OwnedSeriesRecount/whole_ring/tenants=3/state_change-32                          1.229k ±  0%
Ingester_OwnedSeriesRecount/whole_ring/tenants=3/topology_change-32                       10.23k ± 20%
Ingester_OwnedSeriesRecount/whole_ring/tenants=3/full_recount-32                           5.000 ± 20%
Ingester_OwnedSeriesRecount/sharded/shard_size=100/tenants=1/idle-32                       4.000 ± 25%
Ingester_OwnedSeriesRecount/sharded/shard_size=100/tenants=1/state_change-32              2.445k ± 50%
Ingester_OwnedSeriesRecount/sharded/shard_size=100/tenants=1/topology_change-32           12.71k ± 16%
Ingester_OwnedSeriesRecount/sharded/shard_size=100/tenants=1/full_recount-32               6.000 ± 17%
Ingester_OwnedSeriesRecount/sharded/shard_size=100/tenants=10/idle-32                      17.00 ± 12%
Ingester_OwnedSeriesRecount/sharded/shard_size=100/tenants=10/state_change-32             3.668k ± 33%
Ingester_OwnedSeriesRecount/sharded/shard_size=100/tenants=10/topology_change-32          21.90k ± 14%
Ingester_OwnedSeriesRecount/sharded/shard_size=100/tenants=10/full_recount-32              30.50 ±  8%
Ingester_OwnedSeriesRecount/sharded/shard_size=100/tenants=50/idle-32                      72.00 ±  6%
Ingester_OwnedSeriesRecount/sharded/shard_size=100/tenants=50/state_change-32             5.527k ± 11%
Ingester_OwnedSeriesRecount/sharded/shard_size=100/tenants=50/topology_change-32          50.88k ±  3%
Ingester_OwnedSeriesRecount/sharded/shard_size=100/tenants=50/full_recount-32              144.5 ± 12%
IngesterPush_OwnedSeries/owned_series_disabled/existing_series-32                          281.5 ±  4%
IngesterPush_OwnedSeries/owned_series_disabled/new_series-32                              144.9k ±  0%
IngesterPush_OwnedSeries/owned_series_enabled/existing_series-32                           268.5 ± 15%
IngesterPush_OwnedSeries/owned_series_enabled/new_series-32                               154.8k ±  0%
IngesterPush_OwnedSeries/owned_series_enforced/existing_series-32                          273.0 ± 14%
IngesterPush_OwnedSeries/owned_series_enforced/new_series-32                              154.9k ±  0%
geomean                                                                                                ¹
¹ summaries must be >0 to compute geomean

                             │ pr-owned-ingester.txt │
                             │      sec/series       │
ActiveSeries_DeleteSeries-32             339.8n ± 5%

                                                       │ pr-owned-ingester.txt │
                                                       │        B/entry        │
ActiveSeries_MemoryPerEntry/owned_tracking_disabled-32              155.9 ± 0%
ActiveSeries_MemoryPerEntry/owned_tracking_enabled-32               163.9 ± 0%
geomean                                                             159.8
Ring: owned-series benchmarks (PR only)
│ pr-owned-ring.txt │
                                                             │      sec/op       │
Ring_UpdateDesc/state_change-32                                      319.5µ ± 1%
Ring_UpdateDesc/topology_change-32                                   194.5m ± 1%
Ring_NewRingFromDescMemory-32                                        224.6m ± 2%
OwnershipMayHaveChanged/heartbeat_only-32                            352.7µ ± 1%
OwnershipMayHaveChanged/copy-32                                      646.0µ ± 4%
OwnedTokenPositions/whole_ring/instances=100-32                      16.55m ± 1%
OwnedTokenPositions/whole_ring/instances=1000-32                     220.0m ± 1%
OwnedTokenPositions/subring/instances=1000/shard_size=10-32          1.976m ± 0%
OwnedTokenPositions/subring/instances=1000/shard_size=100-32         19.75m ± 2%
OwnedTokenPositions/subring/instances=1000/shard_size=250-32         51.91m ± 1%
OwnedTokenPositions/subring/instances=1000/shard_size=500-32         110.2m ± 2%
Ring_OwnershipFingerprint/whole_ring/instances=1000-32               123.0µ ± 2%
Ring_OwnershipFingerprint/subring/shard_size=10-32                   1.232µ ± 1%
Ring_OwnershipFingerprint/subring/shard_size=100-32                  12.57µ ± 2%
Ring_OwnershipFingerprint/subring/shard_size=250-32                  36.39µ ± 3%
Ring_ShuffleShardMemory/shard_size=10-32                             233.6m ± 1%
Ring_ShuffleShardMemory/shard_size=100-32                            595.7m ± 1%
Ring_ShuffleShardMemory/shard_size=250-32                             1.439 ± 1%
geomean                                                              5.663m

                                                             │ pr-owned-ring.txt │
                                                             │       B/op        │
Ring_UpdateDesc/state_change-32                                   1.500Ki ± 0%
Ring_UpdateDesc/topology_change-32                                100.3Mi ± 0%
Ring_NewRingFromDescMemory-32                                     100.3Mi ± 0%
OwnershipMayHaveChanged/heartbeat_only-32                           0.000 ± 0%
OwnershipMayHaveChanged/copy-32                                   2.156Mi ± 0%
OwnedTokenPositions/whole_ring/instances=100-32                   56.00Ki ± 0%
OwnedTokenPositions/whole_ring/instances=1000-32                  504.0Ki ± 0%
OwnedTokenPositions/subring/instances=1000/shard_size=10-32       6.000Ki ± 0%
OwnedTokenPositions/subring/instances=1000/shard_size=100-32      56.00Ki ± 0%
OwnedTokenPositions/subring/instances=1000/shard_size=250-32      128.0Ki ± 0%
OwnedTokenPositions/subring/instances=1000/shard_size=500-32      256.0Ki ± 0%
Ring_OwnershipFingerprint/whole_ring/instances=1000-32            16.00Ki ± 0%
Ring_OwnershipFingerprint/subring/shard_size=10-32                  192.0 ± 0%
Ring_OwnershipFingerprint/subring/shard_size=100-32               1.750Ki ± 0%
Ring_OwnershipFingerprint/subring/shard_size=250-32               4.000Ki ± 1%
Ring_ShuffleShardMemory/shard_size=10-32                          103.7Mi ± 0%
Ring_ShuffleShardMemory/shard_size=100-32                         123.6Mi ± 0%
Ring_ShuffleShardMemory/shard_size=250-32                         156.2Mi ± 0%
geomean                                                                        ¹
¹ summaries must be >0 to compute geomean

                                                             │ pr-owned-ring.txt │
                                                             │     allocs/op     │
Ring_UpdateDesc/state_change-32                                     12.00 ± 0%
Ring_UpdateDesc/topology_change-32                                 8.191k ± 0%
Ring_NewRingFromDescMemory-32                                      8.577k ± 0%
OwnershipMayHaveChanged/heartbeat_only-32                           0.000 ± 0%
OwnershipMayHaveChanged/copy-32                                    1.005k ± 0%
OwnedTokenPositions/whole_ring/instances=100-32                     1.000 ± 0%
OwnedTokenPositions/whole_ring/instances=1000-32                    1.000 ± 0%
OwnedTokenPositions/subring/instances=1000/shard_size=10-32         1.000 ± 0%
OwnedTokenPositions/subring/instances=1000/shard_size=100-32        1.000 ± 0%
OwnedTokenPositions/subring/instances=1000/shard_size=250-32        1.000 ± 0%
OwnedTokenPositions/subring/instances=1000/shard_size=500-32        1.000 ± 0%
Ring_OwnershipFingerprint/whole_ring/instances=1000-32              1.000 ± 0%
Ring_OwnershipFingerprint/subring/shard_size=10-32                  1.000 ± 0%
Ring_OwnershipFingerprint/subring/shard_size=100-32                 1.000 ± 0%
Ring_OwnershipFingerprint/subring/shard_size=250-32                 1.000 ± 0%
Ring_ShuffleShardMemory/shard_size=10-32                           13.04k ± 0%
Ring_ShuffleShardMemory/shard_size=100-32                          31.49k ± 0%
Ring_ShuffleShardMemory/shard_size=250-32                          61.80k ± 0%
geomean                                                                        ¹
¹ summaries must be >0 to compute geomean

                              │ pr-owned-ring.txt │
                              │      B/ring       │
Ring_NewRingFromDescMemory-32        52.05Mi ± 0%

                                          │ pr-owned-ring.txt │
                                          │     B/subring     │
Ring_ShuffleShardMemory/shard_size=10-32         51.38Ki ± 0%
Ring_ShuffleShardMemory/shard_size=100-32        438.9Ki ± 0%
Ring_ShuffleShardMemory/shard_size=250-32        1.041Mi ± 0%
geomean                                          288.6Ki
Ring: master vs PR (benchstat)
│ master-ring.txt │            pr-ring.txt             │
                                                                                               │     sec/op      │   sec/op     vs base               │
UpdateRingState/num_instances_=_50,_num_zones_=_1,_num_tokens_=_128,_update_tokens_=_false-32       51.04µ ±  1%   50.83µ ± 1%       ~ (p=0.063 n=10)
UpdateRingState/num_instances_=_50,_num_zones_=_1,_num_tokens_=_128,_update_tokens_=_true-32        1.078m ±  0%   1.101m ± 1%  +2.17% (p=0.000 n=10)
UpdateRingState/num_instances_=_50,_num_zones_=_1,_num_tokens_=_256,_update_tokens_=_false-32       52.16µ ±  1%   52.45µ ± 1%       ~ (p=0.218 n=10)
UpdateRingState/num_instances_=_50,_num_zones_=_1,_num_tokens_=_256,_update_tokens_=_true-32        2.108m ±  1%   2.165m ± 2%  +2.71% (p=0.000 n=10)
UpdateRingState/num_instances_=_50,_num_zones_=_1,_num_tokens_=_512,_update_tokens_=_false-32       55.86µ ±  0%   55.51µ ± 0%  -0.64% (p=0.000 n=10)
UpdateRingState/num_instances_=_50,_num_zones_=_1,_num_tokens_=_512,_update_tokens_=_true-32        4.297m ±  3%   4.370m ± 2%       ~ (p=0.247 n=10)
UpdateRingState/num_instances_=_50,_num_zones_=_3,_num_tokens_=_128,_update_tokens_=_false-32       50.97µ ±  1%   50.82µ ± 1%       ~ (p=0.325 n=10)
UpdateRingState/num_instances_=_50,_num_zones_=_3,_num_tokens_=_128,_update_tokens_=_true-32        1.084m ±  1%   1.113m ± 1%  +2.73% (p=0.000 n=10)
UpdateRingState/num_instances_=_50,_num_zones_=_3,_num_tokens_=_256,_update_tokens_=_false-32       52.41µ ±  1%   52.55µ ± 1%       ~ (p=0.853 n=10)
UpdateRingState/num_instances_=_50,_num_zones_=_3,_num_tokens_=_256,_update_tokens_=_true-32        2.119m ±  1%   2.157m ± 1%  +1.81% (p=0.002 n=10)
UpdateRingState/num_instances_=_50,_num_zones_=_3,_num_tokens_=_512,_update_tokens_=_false-32       55.70µ ±  1%   55.41µ ± 0%  -0.52% (p=0.011 n=10)
UpdateRingState/num_instances_=_50,_num_zones_=_3,_num_tokens_=_512,_update_tokens_=_true-32        4.310m ±  2%   4.327m ± 1%       ~ (p=0.684 n=10)
UpdateRingState/num_instances_=_100,_num_zones_=_1,_num_tokens_=_128,_update_tokens_=_false-32      100.6µ ±  1%   100.4µ ± 0%       ~ (p=0.247 n=10)
UpdateRingState/num_instances_=_100,_num_zones_=_1,_num_tokens_=_128,_update_tokens_=_true-32       2.372m ±  2%   2.438m ± 2%  +2.78% (p=0.002 n=10)
UpdateRingState/num_instances_=_100,_num_zones_=_1,_num_tokens_=_256,_update_tokens_=_false-32      103.5µ ±  1%   103.4µ ± 1%       ~ (p=0.280 n=10)
UpdateRingState/num_instances_=_100,_num_zones_=_1,_num_tokens_=_256,_update_tokens_=_true-32       4.727m ±  1%   4.846m ± 2%  +2.50% (p=0.000 n=10)
UpdateRingState/num_instances_=_100,_num_zones_=_1,_num_tokens_=_512,_update_tokens_=_false-32      110.1µ ±  1%   109.5µ ± 0%  -0.54% (p=0.029 n=10)
UpdateRingState/num_instances_=_100,_num_zones_=_1,_num_tokens_=_512,_update_tokens_=_true-32       9.590m ±  3%   9.837m ± 3%  +2.57% (p=0.019 n=10)
UpdateRingState/num_instances_=_100,_num_zones_=_3,_num_tokens_=_128,_update_tokens_=_false-32      101.3µ ±  1%   100.4µ ± 1%  -0.88% (p=0.043 n=10)
UpdateRingState/num_instances_=_100,_num_zones_=_3,_num_tokens_=_128,_update_tokens_=_true-32       2.368m ±  2%   2.429m ± 2%  +2.55% (p=0.003 n=10)
UpdateRingState/num_instances_=_100,_num_zones_=_3,_num_tokens_=_256,_update_tokens_=_false-32      103.7µ ±  1%   102.7µ ± 0%  -1.01% (p=0.000 n=10)
UpdateRingState/num_instances_=_100,_num_zones_=_3,_num_tokens_=_256,_update_tokens_=_true-32       4.747m ±  2%   4.849m ± 2%  +2.14% (p=0.003 n=10)
UpdateRingState/num_instances_=_100,_num_zones_=_3,_num_tokens_=_512,_update_tokens_=_false-32      110.1µ ±  1%   109.6µ ± 1%       ~ (p=0.123 n=10)
UpdateRingState/num_instances_=_100,_num_zones_=_3,_num_tokens_=_512,_update_tokens_=_true-32       9.538m ±  2%   9.868m ± 5%  +3.47% (p=0.000 n=10)
UpdateRingState/num_instances_=_500,_num_zones_=_1,_num_tokens_=_128,_update_tokens_=_false-32      641.3µ ±  1%   640.5µ ± 1%       ~ (p=0.971 n=10)
UpdateRingState/num_instances_=_500,_num_zones_=_1,_num_tokens_=_128,_update_tokens_=_true-32       17.77m ±  2%   17.78m ± 3%       ~ (p=0.481 n=10)
UpdateRingState/num_instances_=_500,_num_zones_=_1,_num_tokens_=_256,_update_tokens_=_false-32      669.5µ ±  1%   666.6µ ± 1%       ~ (p=0.075 n=10)
UpdateRingState/num_instances_=_500,_num_zones_=_1,_num_tokens_=_256,_update_tokens_=_true-32       32.43m ±  1%   32.66m ± 1%  +0.70% (p=0.029 n=10)
UpdateRingState/num_instances_=_500,_num_zones_=_1,_num_tokens_=_512,_update_tokens_=_false-32      729.4µ ±  2%   724.4µ ± 1%  -0.69% (p=0.015 n=10)
UpdateRingState/num_instances_=_500,_num_zones_=_1,_num_tokens_=_512,_update_tokens_=_true-32       65.14m ±  2%   65.39m ± 2%       ~ (p=0.393 n=10)
UpdateRingState/num_instances_=_500,_num_zones_=_3,_num_tokens_=_128,_update_tokens_=_false-32      651.0µ ±  1%   640.6µ ± 1%  -1.60% (p=0.000 n=10)
UpdateRingState/num_instances_=_500,_num_zones_=_3,_num_tokens_=_128,_update_tokens_=_true-32       17.74m ±  1%   17.81m ± 2%       ~ (p=0.436 n=10)
UpdateRingState/num_instances_=_500,_num_zones_=_3,_num_tokens_=_256,_update_tokens_=_false-32      674.7µ ±  1%   664.9µ ± 1%  -1.46% (p=0.000 n=10)
UpdateRingState/num_instances_=_500,_num_zones_=_3,_num_tokens_=_256,_update_tokens_=_true-32       32.10m ±  1%   32.28m ± 2%       ~ (p=0.280 n=10)
UpdateRingState/num_instances_=_500,_num_zones_=_3,_num_tokens_=_512,_update_tokens_=_false-32      730.3µ ±  1%   723.4µ ± 1%  -0.95% (p=0.007 n=10)
UpdateRingState/num_instances_=_500,_num_zones_=_3,_num_tokens_=_512,_update_tokens_=_true-32       64.83m ±  1%   65.18m ± 3%       ~ (p=0.796 n=10)
Ring_ShuffleShard/num_instances_=_50,_num_zones_=_1,_shard_size_=_3-32                              24.21µ ±  1%   24.45µ ± 0%  +1.01% (p=0.000 n=10)
Ring_ShuffleShard/num_instances_=_50,_num_zones_=_1,_shard_size_=_10-32                             110.6µ ±  1%   111.8µ ± 0%  +1.10% (p=0.003 n=10)
Ring_ShuffleShard/num_instances_=_50,_num_zones_=_1,_shard_size_=_30-32                             452.9µ ±  0%   459.7µ ± 0%  +1.51% (p=0.002 n=10)
Ring_ShuffleShard/num_instances_=_50,_num_zones_=_3,_shard_size_=_3-32                              42.50µ ±  1%   42.82µ ± 1%  +0.75% (p=0.002 n=10)
Ring_ShuffleShard/num_instances_=_50,_num_zones_=_3,_shard_size_=_10-32                             134.6µ ±  1%   136.2µ ± 1%  +1.16% (p=0.000 n=10)
Ring_ShuffleShard/num_instances_=_50,_num_zones_=_3,_shard_size_=_30-32                             405.4µ ±  0%   412.0µ ± 0%  +1.63% (p=0.000 n=10)
Ring_ShuffleShard/num_instances_=_100,_num_zones_=_1,_shard_size_=_3-32                             23.63µ ±  1%   24.13µ ± 0%  +2.11% (p=0.000 n=10)
Ring_ShuffleShard/num_instances_=_100,_num_zones_=_1,_shard_size_=_10-32                            109.5µ ±  1%   111.1µ ± 1%  +1.49% (p=0.000 n=10)
Ring_ShuffleShard/num_instances_=_100,_num_zones_=_1,_shard_size_=_30-32                            450.7µ ±  0%   457.6µ ± 0%  +1.54% (p=0.000 n=10)
Ring_ShuffleShard/num_instances_=_100,_num_zones_=_3,_shard_size_=_3-32                             41.75µ ±  1%   42.24µ ± 0%  +1.19% (p=0.002 n=10)
Ring_ShuffleShard/num_instances_=_100,_num_zones_=_3,_shard_size_=_10-32                            133.6µ ±  0%   135.0µ ± 0%  +1.03% (p=0.000 n=10)
Ring_ShuffleShard/num_instances_=_100,_num_zones_=_3,_shard_size_=_30-32                            404.3µ ±  0%   410.0µ ± 0%  +1.40% (p=0.000 n=10)
Ring_ShuffleShard/num_instances_=_1000,_num_zones_=_1,_shard_size_=_3-32                            21.96µ ±  1%   22.12µ ± 0%  +0.71% (p=0.001 n=10)
Ring_ShuffleShard/num_instances_=_1000,_num_zones_=_1,_shard_size_=_10-32                           106.9µ ±  1%   108.0µ ± 0%  +1.02% (p=0.011 n=10)
Ring_ShuffleShard/num_instances_=_1000,_num_zones_=_1,_shard_size_=_30-32                           445.7µ ±  0%   451.9µ ± 1%  +1.38% (p=0.000 n=10)
Ring_ShuffleShard/num_instances_=_1000,_num_zones_=_3,_shard_size_=_3-32                            39.04µ ±  0%   39.17µ ± 0%       ~ (p=0.072 n=10)
Ring_ShuffleShard/num_instances_=_1000,_num_zones_=_3,_shard_size_=_10-32                           128.7µ ±  0%   129.8µ ± 0%  +0.82% (p=0.001 n=10)
Ring_ShuffleShard/num_instances_=_1000,_num_zones_=_3,_shard_size_=_30-32                           396.8µ ±  0%   401.5µ ± 0%  +1.19% (p=0.000 n=10)
Ring_ShuffleShardCached/num_instances_=_50,_num_zones_=_1,_shard_size_=_3-32                        212.2n ±  2%   213.1n ± 1%       ~ (p=0.566 n=10)
Ring_ShuffleShardCached/num_instances_=_50,_num_zones_=_1,_shard_size_=_10-32                       458.4n ± 14%   449.9n ± 3%       ~ (p=0.190 n=10)
Ring_ShuffleShardCached/num_instances_=_50,_num_zones_=_1,_shard_size_=_30-32                       1.109µ ±  5%   1.117µ ± 5%       ~ (p=0.271 n=10)
Ring_ShuffleShardCached/num_instances_=_50,_num_zones_=_3,_shard_size_=_3-32                        210.9n ±  2%   212.0n ± 1%       ~ (p=0.424 n=10)
Ring_ShuffleShardCached/num_instances_=_50,_num_zones_=_3,_shard_size_=_10-32                       505.1n ±  3%   519.0n ± 3%  +2.74% (p=0.015 n=10)
Ring_ShuffleShardCached/num_instances_=_50,_num_zones_=_3,_shard_size_=_30-32                       1.114µ ±  5%   1.118µ ± 2%       ~ (p=0.956 n=10)
Ring_ShuffleShardCached/num_instances_=_100,_num_zones_=_1,_shard_size_=_3-32                       211.1n ±  3%   212.3n ± 1%       ~ (p=0.643 n=10)
Ring_ShuffleShardCached/num_instances_=_100,_num_zones_=_1,_shard_size_=_10-32                      449.4n ±  3%   463.3n ± 3%       ~ (p=0.052 n=10)
Ring_ShuffleShardCached/num_instances_=_100,_num_zones_=_1,_shard_size_=_30-32                      1.109µ ±  2%   1.126µ ± 2%       ~ (p=0.143 n=10)
Ring_ShuffleShardCached/num_instances_=_100,_num_zones_=_3,_shard_size_=_3-32                       211.4n ±  1%   211.8n ± 2%       ~ (p=1.000 n=10)
Ring_ShuffleShardCached/num_instances_=_100,_num_zones_=_3,_shard_size_=_10-32                      515.8n ±  2%   519.9n ± 3%       ~ (p=0.425 n=10)
Ring_ShuffleShardCached/num_instances_=_100,_num_zones_=_3,_shard_size_=_30-32                      1.104µ ±  4%   1.124µ ± 9%       ~ (p=0.138 n=10)
Ring_ShuffleShardCached/num_instances_=_1000,_num_zones_=_1,_shard_size_=_3-32                      215.6n ±  0%   215.6n ± 2%       ~ (p=0.925 n=10)
Ring_ShuffleShardCached/num_instances_=_1000,_num_zones_=_1,_shard_size_=_10-32                     453.5n ±  4%   465.6n ± 2%       ~ (p=0.075 n=10)
Ring_ShuffleShardCached/num_instances_=_1000,_num_zones_=_1,_shard_size_=_30-32                     1.122µ ±  5%   1.145µ ± 2%       ~ (p=0.066 n=10)
Ring_ShuffleShardCached/num_instances_=_1000,_num_zones_=_3,_shard_size_=_3-32                      215.4n ±  2%   215.5n ± 2%       ~ (p=0.671 n=10)
Ring_ShuffleShardCached/num_instances_=_1000,_num_zones_=_3,_shard_size_=_10-32                     510.4n ±  2%   523.4n ± 3%  +2.55% (p=0.011 n=10)
Ring_ShuffleShardCached/num_instances_=_1000,_num_zones_=_3,_shard_size_=_30-32                     1.119µ ±  4%   1.141µ ± 2%       ~ (p=0.093 n=10)
Ring_ShuffleShard_512Tokens-32                                                                      298.8µ ±  1%   301.0µ ± 0%  +0.73% (p=0.011 n=10)
Ring_Get-32                                                                                         592.0n ±  1%   591.2n ± 1%       ~ (p=0.928 n=10)
geomean                                                                                             84.10µ         84.81µ       +0.84%

                                                                                               │ master-ring.txt │              pr-ring.txt              │
                                                                                               │      B/op       │     B/op      vs base                 │
UpdateRingState/num_instances_=_50,_num_zones_=_1,_num_tokens_=_128,_update_tokens_=_false-32     20.34Ki ± 0%     20.34Ki ± 0%       ~ (p=0.060 n=10)
UpdateRingState/num_instances_=_50,_num_zones_=_1,_num_tokens_=_128,_update_tokens_=_true-32      859.9Ki ± 0%     860.0Ki ± 0%  +0.01% (p=0.006 n=10)
UpdateRingState/num_instances_=_50,_num_zones_=_1,_num_tokens_=_256,_update_tokens_=_false-32     20.38Ki ± 0%     20.38Ki ± 0%       ~ (p=0.705 n=10)
UpdateRingState/num_instances_=_50,_num_zones_=_1,_num_tokens_=_256,_update_tokens_=_true-32      1.644Mi ± 0%     1.644Mi ± 0%       ~ (p=0.494 n=10)
UpdateRingState/num_instances_=_50,_num_zones_=_1,_num_tokens_=_512,_update_tokens_=_false-32     20.46Ki ± 0%     20.46Ki ± 0%  -0.00% (p=0.024 n=10)
UpdateRingState/num_instances_=_50,_num_zones_=_1,_num_tokens_=_512,_update_tokens_=_true-32      3.242Mi ± 0%     3.242Mi ± 0%       ~ (p=0.670 n=10)
UpdateRingState/num_instances_=_50,_num_zones_=_3,_num_tokens_=_128,_update_tokens_=_false-32     20.34Ki ± 0%     20.34Ki ± 0%       ~ (p=1.000 n=10)
UpdateRingState/num_instances_=_50,_num_zones_=_3,_num_tokens_=_128,_update_tokens_=_true-32      860.0Ki ± 0%     860.0Ki ± 0%       ~ (p=0.565 n=10)
UpdateRingState/num_instances_=_50,_num_zones_=_3,_num_tokens_=_256,_update_tokens_=_false-32     20.38Ki ± 0%     20.38Ki ± 0%       ~ (p=1.000 n=10)
UpdateRingState/num_instances_=_50,_num_zones_=_3,_num_tokens_=_256,_update_tokens_=_true-32      1.644Mi ± 0%     1.644Mi ± 0%       ~ (p=0.954 n=10)
UpdateRingState/num_instances_=_50,_num_zones_=_3,_num_tokens_=_512,_update_tokens_=_false-32     20.46Ki ± 0%     20.46Ki ± 0%       ~ (p=0.185 n=10)
UpdateRingState/num_instances_=_50,_num_zones_=_3,_num_tokens_=_512,_update_tokens_=_true-32      3.241Mi ± 0%     3.242Mi ± 0%       ~ (p=0.239 n=10)
UpdateRingState/num_instances_=_100,_num_zones_=_1,_num_tokens_=_128,_update_tokens_=_false-32    40.90Ki ± 0%     40.90Ki ± 0%       ~ (p=0.259 n=10)
UpdateRingState/num_instances_=_100,_num_zones_=_1,_num_tokens_=_128,_update_tokens_=_true-32     1.689Mi ± 0%     1.689Mi ± 0%       ~ (p=0.739 n=10)
UpdateRingState/num_instances_=_100,_num_zones_=_1,_num_tokens_=_256,_update_tokens_=_false-32    41.05Ki ± 0%     41.05Ki ± 0%       ~ (p=0.362 n=10)
UpdateRingState/num_instances_=_100,_num_zones_=_1,_num_tokens_=_256,_update_tokens_=_true-32     3.288Mi ± 0%     3.287Mi ± 0%       ~ (p=0.424 n=10)
UpdateRingState/num_instances_=_100,_num_zones_=_1,_num_tokens_=_512,_update_tokens_=_false-32    41.37Ki ± 0%     41.37Ki ± 0%  -0.01% (p=0.024 n=10)
UpdateRingState/num_instances_=_100,_num_zones_=_1,_num_tokens_=_512,_update_tokens_=_true-32     6.484Mi ± 0%     6.483Mi ± 0%       ~ (p=0.971 n=10)
UpdateRingState/num_instances_=_100,_num_zones_=_3,_num_tokens_=_128,_update_tokens_=_false-32    40.90Ki ± 0%     40.90Ki ± 0%  -0.00% (p=0.032 n=10)
UpdateRingState/num_instances_=_100,_num_zones_=_3,_num_tokens_=_128,_update_tokens_=_true-32     1.689Mi ± 0%     1.690Mi ± 0%       ~ (p=0.240 n=10)
UpdateRingState/num_instances_=_100,_num_zones_=_3,_num_tokens_=_256,_update_tokens_=_false-32    41.05Ki ± 0%     41.04Ki ± 0%  -0.01% (p=0.000 n=10)
UpdateRingState/num_instances_=_100,_num_zones_=_3,_num_tokens_=_256,_update_tokens_=_true-32     3.288Mi ± 0%     3.288Mi ± 0%       ~ (p=0.481 n=10)
UpdateRingState/num_instances_=_100,_num_zones_=_3,_num_tokens_=_512,_update_tokens_=_false-32    41.37Ki ± 0%     41.37Ki ± 0%       ~ (p=0.168 n=10)
UpdateRingState/num_instances_=_100,_num_zones_=_3,_num_tokens_=_512,_update_tokens_=_true-32     6.483Mi ± 0%     6.484Mi ± 0%       ~ (p=0.105 n=10)
UpdateRingState/num_instances_=_500,_num_zones_=_1,_num_tokens_=_128,_update_tokens_=_false-32    326.7Ki ± 0%     326.7Ki ± 0%       ~ (p=0.985 n=10)
UpdateRingState/num_instances_=_500,_num_zones_=_1,_num_tokens_=_128,_update_tokens_=_true-32     13.14Mi ± 0%     13.14Mi ± 0%       ~ (p=0.739 n=10)
UpdateRingState/num_instances_=_500,_num_zones_=_1,_num_tokens_=_256,_update_tokens_=_false-32    335.2Ki ± 0%     335.0Ki ± 0%       ~ (p=0.393 n=10)
UpdateRingState/num_instances_=_500,_num_zones_=_1,_num_tokens_=_256,_update_tokens_=_true-32     25.66Mi ± 0%     25.66Mi ± 0%       ~ (p=0.403 n=10)
UpdateRingState/num_instances_=_500,_num_zones_=_1,_num_tokens_=_512,_update_tokens_=_false-32    352.2Ki ± 0%     351.9Ki ± 0%  -0.07% (p=0.035 n=10)
UpdateRingState/num_instances_=_500,_num_zones_=_1,_num_tokens_=_512,_update_tokens_=_true-32     50.69Mi ± 0%     50.69Mi ± 0%       ~ (p=0.670 n=10)
UpdateRingState/num_instances_=_500,_num_zones_=_3,_num_tokens_=_128,_update_tokens_=_false-32    326.8Ki ± 0%     326.6Ki ± 0%  -0.05% (p=0.002 n=10)
UpdateRingState/num_instances_=_500,_num_zones_=_3,_num_tokens_=_128,_update_tokens_=_true-32     13.14Mi ± 0%     13.14Mi ± 0%       ~ (p=0.988 n=10)
UpdateRingState/num_instances_=_500,_num_zones_=_3,_num_tokens_=_256,_update_tokens_=_false-32    335.3Ki ± 0%     335.1Ki ± 0%  -0.06% (p=0.050 n=10)
UpdateRingState/num_instances_=_500,_num_zones_=_3,_num_tokens_=_256,_update_tokens_=_true-32     25.65Mi ± 0%     25.65Mi ± 0%       ~ (p=1.000 n=10)
UpdateRingState/num_instances_=_500,_num_zones_=_3,_num_tokens_=_512,_update_tokens_=_false-32    352.4Ki ± 0%     351.9Ki ± 0%  -0.13% (p=0.000 n=10)
UpdateRingState/num_instances_=_500,_num_zones_=_3,_num_tokens_=_512,_update_tokens_=_true-32     50.68Mi ± 0%     50.69Mi ± 0%       ~ (p=0.193 n=10)
Ring_ShuffleShard/num_instances_=_50,_num_zones_=_1,_shard_size_=_3-32                            11.20Ki ± 0%     11.20Ki ± 0%       ~ (p=1.000 n=10) ¹
Ring_ShuffleShard/num_instances_=_50,_num_zones_=_1,_shard_size_=_10-32                           21.30Ki ± 0%     21.30Ki ± 0%       ~ (p=1.000 n=10) ¹
Ring_ShuffleShard/num_instances_=_50,_num_zones_=_1,_shard_size_=_30-32                           51.73Ki ± 0%     51.73Ki ± 0%       ~ (p=1.000 n=10) ¹
Ring_ShuffleShard/num_instances_=_50,_num_zones_=_3,_shard_size_=_3-32                            21.77Ki ± 0%     21.77Ki ± 0%       ~ (p=1.000 n=10) ¹
Ring_ShuffleShard/num_instances_=_50,_num_zones_=_3,_shard_size_=_10-32                           33.57Ki ± 0%     33.57Ki ± 0%       ~ (p=1.000 n=10) ¹
Ring_ShuffleShard/num_instances_=_50,_num_zones_=_3,_shard_size_=_30-32                           62.68Ki ± 0%     62.68Ki ± 0%       ~ (p=1.000 n=10) ¹
Ring_ShuffleShard/num_instances_=_100,_num_zones_=_1,_shard_size_=_3-32                           11.20Ki ± 0%     11.20Ki ± 0%       ~ (p=1.000 n=10) ¹
Ring_ShuffleShard/num_instances_=_100,_num_zones_=_1,_shard_size_=_10-32                          21.30Ki ± 0%     21.30Ki ± 0%       ~ (p=1.000 n=10) ¹
Ring_ShuffleShard/num_instances_=_100,_num_zones_=_1,_shard_size_=_30-32                          51.73Ki ± 0%     51.73Ki ± 0%       ~ (p=1.000 n=10) ¹
Ring_ShuffleShard/num_instances_=_100,_num_zones_=_3,_shard_size_=_3-32                           21.77Ki ± 0%     21.77Ki ± 0%       ~ (p=1.000 n=10) ¹
Ring_ShuffleShard/num_instances_=_100,_num_zones_=_3,_shard_size_=_10-32                          33.57Ki ± 0%     33.57Ki ± 0%       ~ (p=1.000 n=10) ¹
Ring_ShuffleShard/num_instances_=_100,_num_zones_=_3,_shard_size_=_30-32                          62.68Ki ± 0%     62.68Ki ± 0%       ~ (p=1.000 n=10) ¹
Ring_ShuffleShard/num_instances_=_1000,_num_zones_=_1,_shard_size_=_3-32                          11.20Ki ± 0%     11.20Ki ± 0%       ~ (p=1.000 n=10) ¹
Ring_ShuffleShard/num_instances_=_1000,_num_zones_=_1,_shard_size_=_10-32                         21.30Ki ± 0%     21.30Ki ± 0%       ~ (p=1.000 n=10) ¹
Ring_ShuffleShard/num_instances_=_1000,_num_zones_=_1,_shard_size_=_30-32                         51.73Ki ± 0%     51.73Ki ± 0%       ~ (p=1.000 n=10) ¹
Ring_ShuffleShard/num_instances_=_1000,_num_zones_=_3,_shard_size_=_3-32                          21.77Ki ± 0%     21.77Ki ± 0%       ~ (p=1.000 n=10) ¹
Ring_ShuffleShard/num_instances_=_1000,_num_zones_=_3,_shard_size_=_10-32                         33.57Ki ± 0%     33.57Ki ± 0%       ~ (p=1.000 n=10) ¹
Ring_ShuffleShard/num_instances_=_1000,_num_zones_=_3,_shard_size_=_30-32                         62.68Ki ± 0%     62.68Ki ± 0%       ~ (p=1.000 n=10) ¹
Ring_ShuffleShardCached/num_instances_=_50,_num_zones_=_1,_shard_size_=_3-32                        0.000 ± 0%       0.000 ± 0%       ~ (p=1.000 n=10) ¹
Ring_ShuffleShardCached/num_instances_=_50,_num_zones_=_1,_shard_size_=_10-32                       0.000 ± 0%       0.000 ± 0%       ~ (p=1.000 n=10) ¹
Ring_ShuffleShardCached/num_instances_=_50,_num_zones_=_1,_shard_size_=_30-32                       0.000 ± 0%       0.000 ± 0%       ~ (p=1.000 n=10) ¹
Ring_ShuffleShardCached/num_instances_=_50,_num_zones_=_3,_shard_size_=_3-32                        0.000 ± 0%       0.000 ± 0%       ~ (p=1.000 n=10) ¹
Ring_ShuffleShardCached/num_instances_=_50,_num_zones_=_3,_shard_size_=_10-32                       0.000 ± 0%       0.000 ± 0%       ~ (p=1.000 n=10) ¹
Ring_ShuffleShardCached/num_instances_=_50,_num_zones_=_3,_shard_size_=_30-32                       0.000 ± 0%       0.000 ± 0%       ~ (p=1.000 n=10) ¹
Ring_ShuffleShardCached/num_instances_=_100,_num_zones_=_1,_shard_size_=_3-32                       0.000 ± 0%       0.000 ± 0%       ~ (p=1.000 n=10) ¹
Ring_ShuffleShardCached/num_instances_=_100,_num_zones_=_1,_shard_size_=_10-32                      0.000 ± 0%       0.000 ± 0%       ~ (p=1.000 n=10) ¹
Ring_ShuffleShardCached/num_instances_=_100,_num_zones_=_1,_shard_size_=_30-32                      0.000 ± 0%       0.000 ± 0%       ~ (p=1.000 n=10) ¹
Ring_ShuffleShardCached/num_instances_=_100,_num_zones_=_3,_shard_size_=_3-32                       0.000 ± 0%       0.000 ± 0%       ~ (p=1.000 n=10) ¹
Ring_ShuffleShardCached/num_instances_=_100,_num_zones_=_3,_shard_size_=_10-32                      0.000 ± 0%       0.000 ± 0%       ~ (p=1.000 n=10) ¹
Ring_ShuffleShardCached/num_instances_=_100,_num_zones_=_3,_shard_size_=_30-32                      0.000 ± 0%       0.000 ± 0%       ~ (p=1.000 n=10) ¹
Ring_ShuffleShardCached/num_instances_=_1000,_num_zones_=_1,_shard_size_=_3-32                      0.000 ± 0%       0.000 ± 0%       ~ (p=1.000 n=10) ¹
Ring_ShuffleShardCached/num_instances_=_1000,_num_zones_=_1,_shard_size_=_10-32                     0.000 ± 0%       0.000 ± 0%       ~ (p=1.000 n=10) ¹
Ring_ShuffleShardCached/num_instances_=_1000,_num_zones_=_1,_shard_size_=_30-32                     0.000 ± 0%       0.000 ± 0%       ~ (p=1.000 n=10) ¹
Ring_ShuffleShardCached/num_instances_=_1000,_num_zones_=_3,_shard_size_=_3-32                      0.000 ± 0%       0.000 ± 0%       ~ (p=1.000 n=10) ¹
Ring_ShuffleShardCached/num_instances_=_1000,_num_zones_=_3,_shard_size_=_10-32                     0.000 ± 0%       0.000 ± 0%       ~ (p=1.000 n=10) ¹
Ring_ShuffleShardCached/num_instances_=_1000,_num_zones_=_3,_shard_size_=_30-32                     0.000 ± 0%       0.000 ± 0%       ~ (p=1.000 n=10) ¹
Ring_ShuffleShard_512Tokens-32                                                                    57.12Ki ± 0%     57.12Ki ± 0%       ~ (p=1.000 n=10) ¹
Ring_Get-32                                                                                         0.000 ± 0%       0.000 ± 0%       ~ (p=1.000 n=10) ¹
geomean                                                                                                        ²                 -0.00%                ²
¹ all samples are equal
² summaries must be >0 to compute geomean

                                                                                               │ master-ring.txt │             pr-ring.txt              │
                                                                                               │    allocs/op    │  allocs/op   vs base                 │
UpdateRingState/num_instances_=_50,_num_zones_=_1,_num_tokens_=_128,_update_tokens_=_false-32       54.00 ± 0%      54.00 ± 0%       ~ (p=1.000 n=10) ¹
UpdateRingState/num_instances_=_50,_num_zones_=_1,_num_tokens_=_128,_update_tokens_=_true-32        537.0 ± 0%      537.0 ± 0%       ~ (p=1.000 n=10)
UpdateRingState/num_instances_=_50,_num_zones_=_1,_num_tokens_=_256,_update_tokens_=_false-32       54.00 ± 0%      54.00 ± 0%       ~ (p=1.000 n=10) ¹
UpdateRingState/num_instances_=_50,_num_zones_=_1,_num_tokens_=_256,_update_tokens_=_true-32        572.0 ± 0%      573.0 ± 0%       ~ (p=0.179 n=10)
UpdateRingState/num_instances_=_50,_num_zones_=_1,_num_tokens_=_512,_update_tokens_=_false-32       54.00 ± 0%      54.00 ± 0%       ~ (p=1.000 n=10) ¹
UpdateRingState/num_instances_=_50,_num_zones_=_1,_num_tokens_=_512,_update_tokens_=_true-32        644.0 ± 0%      644.0 ± 0%       ~ (p=1.000 n=10) ¹
UpdateRingState/num_instances_=_50,_num_zones_=_3,_num_tokens_=_128,_update_tokens_=_false-32       54.00 ± 0%      54.00 ± 0%       ~ (p=1.000 n=10) ¹
UpdateRingState/num_instances_=_50,_num_zones_=_3,_num_tokens_=_128,_update_tokens_=_true-32        537.0 ± 0%      537.0 ± 0%       ~ (p=1.000 n=10) ¹
UpdateRingState/num_instances_=_50,_num_zones_=_3,_num_tokens_=_256,_update_tokens_=_false-32       54.00 ± 0%      54.00 ± 0%       ~ (p=1.000 n=10) ¹
UpdateRingState/num_instances_=_50,_num_zones_=_3,_num_tokens_=_256,_update_tokens_=_true-32        572.5 ± 0%      573.0 ± 0%       ~ (p=0.350 n=10)
UpdateRingState/num_instances_=_50,_num_zones_=_3,_num_tokens_=_512,_update_tokens_=_false-32       54.00 ± 0%      54.00 ± 0%       ~ (p=1.000 n=10) ¹
UpdateRingState/num_instances_=_50,_num_zones_=_3,_num_tokens_=_512,_update_tokens_=_true-32        644.0 ± 0%      644.0 ± 0%       ~ (p=1.000 n=10)
UpdateRingState/num_instances_=_100,_num_zones_=_1,_num_tokens_=_128,_update_tokens_=_false-32      66.00 ± 0%      66.00 ± 0%       ~ (p=1.000 n=10) ¹
UpdateRingState/num_instances_=_100,_num_zones_=_1,_num_tokens_=_128,_update_tokens_=_true-32       996.0 ± 0%      996.0 ± 0%       ~ (p=0.526 n=10)
UpdateRingState/num_instances_=_100,_num_zones_=_1,_num_tokens_=_256,_update_tokens_=_false-32      66.00 ± 0%      66.00 ± 0%       ~ (p=1.000 n=10) ¹
UpdateRingState/num_instances_=_100,_num_zones_=_1,_num_tokens_=_256,_update_tokens_=_true-32      1.074k ± 0%     1.075k ± 0%  +0.09% (p=0.003 n=10)
UpdateRingState/num_instances_=_100,_num_zones_=_1,_num_tokens_=_512,_update_tokens_=_false-32      66.00 ± 0%      66.00 ± 0%       ~ (p=1.000 n=10) ¹
UpdateRingState/num_instances_=_100,_num_zones_=_1,_num_tokens_=_512,_update_tokens_=_true-32      1.230k ± 0%     1.232k ± 1%  +0.12% (p=0.029 n=10)
UpdateRingState/num_instances_=_100,_num_zones_=_3,_num_tokens_=_128,_update_tokens_=_false-32      66.00 ± 0%      66.00 ± 0%       ~ (p=1.000 n=10) ¹
UpdateRingState/num_instances_=_100,_num_zones_=_3,_num_tokens_=_128,_update_tokens_=_true-32       996.0 ± 0%      996.0 ± 0%       ~ (p=1.000 n=10) ¹
UpdateRingState/num_instances_=_100,_num_zones_=_3,_num_tokens_=_256,_update_tokens_=_false-32      66.00 ± 0%      66.00 ± 0%       ~ (p=1.000 n=10) ¹
UpdateRingState/num_instances_=_100,_num_zones_=_3,_num_tokens_=_256,_update_tokens_=_true-32      1.074k ± 0%     1.075k ± 0%       ~ (p=0.132 n=10)
UpdateRingState/num_instances_=_100,_num_zones_=_3,_num_tokens_=_512,_update_tokens_=_false-32      66.00 ± 0%      66.00 ± 0%       ~ (p=1.000 n=10) ¹
UpdateRingState/num_instances_=_100,_num_zones_=_3,_num_tokens_=_512,_update_tokens_=_true-32      1.230k ± 0%     1.232k ± 1%  +0.20% (p=0.000 n=10)
UpdateRingState/num_instances_=_500,_num_zones_=_1,_num_tokens_=_128,_update_tokens_=_false-32      122.0 ± 0%      122.0 ± 0%       ~ (p=1.000 n=10) ¹
UpdateRingState/num_instances_=_500,_num_zones_=_1,_num_tokens_=_128,_update_tokens_=_true-32      5.231k ± 1%     5.218k ± 1%       ~ (p=0.837 n=10)
UpdateRingState/num_instances_=_500,_num_zones_=_1,_num_tokens_=_256,_update_tokens_=_false-32      124.0 ± 1%      124.0 ± 1%       ~ (p=0.628 n=10)
UpdateRingState/num_instances_=_500,_num_zones_=_1,_num_tokens_=_256,_update_tokens_=_true-32      6.242k ± 2%     6.210k ± 2%       ~ (p=0.847 n=10)
UpdateRingState/num_instances_=_500,_num_zones_=_1,_num_tokens_=_512,_update_tokens_=_false-32      126.0 ± 1%      126.0 ± 0%       ~ (p=0.211 n=10)
UpdateRingState/num_instances_=_500,_num_zones_=_1,_num_tokens_=_512,_update_tokens_=_true-32      8.055k ± 3%     8.055k ± 3%       ~ (p=0.675 n=10)
UpdateRingState/num_instances_=_500,_num_zones_=_3,_num_tokens_=_128,_update_tokens_=_false-32      122.0 ± 0%      122.0 ± 0%       ~ (p=1.000 n=10)
UpdateRingState/num_instances_=_500,_num_zones_=_3,_num_tokens_=_128,_update_tokens_=_true-32      5.226k ± 1%     5.236k ± 1%       ~ (p=0.725 n=10)
UpdateRingState/num_instances_=_500,_num_zones_=_3,_num_tokens_=_256,_update_tokens_=_false-32      124.5 ± 0%      124.0 ± 1%       ~ (p=0.350 n=10)
UpdateRingState/num_instances_=_500,_num_zones_=_3,_num_tokens_=_256,_update_tokens_=_true-32      6.111k ± 1%     6.111k ± 2%       ~ (p=0.901 n=10)
UpdateRingState/num_instances_=_500,_num_zones_=_3,_num_tokens_=_512,_update_tokens_=_false-32      126.0 ± 1%      126.0 ± 0%       ~ (p=0.263 n=10)
UpdateRingState/num_instances_=_500,_num_zones_=_3,_num_tokens_=_512,_update_tokens_=_true-32      8.055k ± 0%     8.055k ± 3%       ~ (p=0.477 n=10)
Ring_ShuffleShard/num_instances_=_50,_num_zones_=_1,_shard_size_=_3-32                              31.00 ± 0%      31.00 ± 0%       ~ (p=1.000 n=10) ¹
Ring_ShuffleShard/num_instances_=_50,_num_zones_=_1,_shard_size_=_10-32                             63.00 ± 0%      63.00 ± 0%       ~ (p=1.000 n=10) ¹
Ring_ShuffleShard/num_instances_=_50,_num_zones_=_1,_shard_size_=_30-32                             144.0 ± 0%      144.0 ± 0%       ~ (p=1.000 n=10) ¹
Ring_ShuffleShard/num_instances_=_50,_num_zones_=_3,_shard_size_=_3-32                              43.00 ± 0%      43.00 ± 0%       ~ (p=1.000 n=10) ¹
Ring_ShuffleShard/num_instances_=_50,_num_zones_=_3,_shard_size_=_10-32                             87.00 ± 0%      87.00 ± 0%       ~ (p=1.000 n=10) ¹
Ring_ShuffleShard/num_instances_=_50,_num_zones_=_3,_shard_size_=_30-32                             165.0 ± 0%      165.0 ± 0%       ~ (p=1.000 n=10) ¹
Ring_ShuffleShard/num_instances_=_100,_num_zones_=_1,_shard_size_=_3-32                             31.00 ± 0%      31.00 ± 0%       ~ (p=1.000 n=10) ¹
Ring_ShuffleShard/num_instances_=_100,_num_zones_=_1,_shard_size_=_10-32                            63.00 ± 0%      63.00 ± 0%       ~ (p=1.000 n=10) ¹
Ring_ShuffleShard/num_instances_=_100,_num_zones_=_1,_shard_size_=_30-32                            144.0 ± 0%      144.0 ± 0%       ~ (p=1.000 n=10) ¹
Ring_ShuffleShard/num_instances_=_100,_num_zones_=_3,_shard_size_=_3-32                             43.00 ± 0%      43.00 ± 0%       ~ (p=1.000 n=10) ¹
Ring_ShuffleShard/num_instances_=_100,_num_zones_=_3,_shard_size_=_10-32                            87.00 ± 0%      87.00 ± 0%       ~ (p=1.000 n=10) ¹
Ring_ShuffleShard/num_instances_=_100,_num_zones_=_3,_shard_size_=_30-32                            165.0 ± 0%      165.0 ± 0%       ~ (p=1.000 n=10) ¹
Ring_ShuffleShard/num_instances_=_1000,_num_zones_=_1,_shard_size_=_3-32                            31.00 ± 0%      31.00 ± 0%       ~ (p=1.000 n=10) ¹
Ring_ShuffleShard/num_instances_=_1000,_num_zones_=_1,_shard_size_=_10-32                           63.00 ± 0%      63.00 ± 0%       ~ (p=1.000 n=10) ¹
Ring_ShuffleShard/num_instances_=_1000,_num_zones_=_1,_shard_size_=_30-32                           144.0 ± 0%      144.0 ± 0%       ~ (p=1.000 n=10) ¹
Ring_ShuffleShard/num_instances_=_1000,_num_zones_=_3,_shard_size_=_3-32                            43.00 ± 0%      43.00 ± 0%       ~ (p=1.000 n=10) ¹
Ring_ShuffleShard/num_instances_=_1000,_num_zones_=_3,_shard_size_=_10-32                           87.00 ± 0%      87.00 ± 0%       ~ (p=1.000 n=10) ¹
Ring_ShuffleShard/num_instances_=_1000,_num_zones_=_3,_shard_size_=_30-32                           165.0 ± 0%      165.0 ± 0%       ~ (p=1.000 n=10) ¹
Ring_ShuffleShardCached/num_instances_=_50,_num_zones_=_1,_shard_size_=_3-32                        0.000 ± 0%      0.000 ± 0%       ~ (p=1.000 n=10) ¹
Ring_ShuffleShardCached/num_instances_=_50,_num_zones_=_1,_shard_size_=_10-32                       0.000 ± 0%      0.000 ± 0%       ~ (p=1.000 n=10) ¹
Ring_ShuffleShardCached/num_instances_=_50,_num_zones_=_1,_shard_size_=_30-32                       0.000 ± 0%      0.000 ± 0%       ~ (p=1.000 n=10) ¹
Ring_ShuffleShardCached/num_instances_=_50,_num_zones_=_3,_shard_size_=_3-32                        0.000 ± 0%      0.000 ± 0%       ~ (p=1.000 n=10) ¹
Ring_ShuffleShardCached/num_instances_=_50,_num_zones_=_3,_shard_size_=_10-32                       0.000 ± 0%      0.000 ± 0%       ~ (p=1.000 n=10) ¹
Ring_ShuffleShardCached/num_instances_=_50,_num_zones_=_3,_shard_size_=_30-32                       0.000 ± 0%      0.000 ± 0%       ~ (p=1.000 n=10) ¹
Ring_ShuffleShardCached/num_instances_=_100,_num_zones_=_1,_shard_size_=_3-32                       0.000 ± 0%      0.000 ± 0%       ~ (p=1.000 n=10) ¹
Ring_ShuffleShardCached/num_instances_=_100,_num_zones_=_1,_shard_size_=_10-32                      0.000 ± 0%      0.000 ± 0%       ~ (p=1.000 n=10) ¹
Ring_ShuffleShardCached/num_instances_=_100,_num_zones_=_1,_shard_size_=_30-32                      0.000 ± 0%      0.000 ± 0%       ~ (p=1.000 n=10) ¹
Ring_ShuffleShardCached/num_instances_=_100,_num_zones_=_3,_shard_size_=_3-32                       0.000 ± 0%      0.000 ± 0%       ~ (p=1.000 n=10) ¹
Ring_ShuffleShardCached/num_instances_=_100,_num_zones_=_3,_shard_size_=_10-32                      0.000 ± 0%      0.000 ± 0%       ~ (p=1.000 n=10) ¹
Ring_ShuffleShardCached/num_instances_=_100,_num_zones_=_3,_shard_size_=_30-32                      0.000 ± 0%      0.000 ± 0%       ~ (p=1.000 n=10) ¹
Ring_ShuffleShardCached/num_instances_=_1000,_num_zones_=_1,_shard_size_=_3-32                      0.000 ± 0%      0.000 ± 0%       ~ (p=1.000 n=10) ¹
Ring_ShuffleShardCached/num_instances_=_1000,_num_zones_=_1,_shard_size_=_10-32                     0.000 ± 0%      0.000 ± 0%       ~ (p=1.000 n=10) ¹
Ring_ShuffleShardCached/num_instances_=_1000,_num_zones_=_1,_shard_size_=_30-32                     0.000 ± 0%      0.000 ± 0%       ~ (p=1.000 n=10) ¹
Ring_ShuffleShardCached/num_instances_=_1000,_num_zones_=_3,_shard_size_=_3-32                      0.000 ± 0%      0.000 ± 0%       ~ (p=1.000 n=10) ¹
Ring_ShuffleShardCached/num_instances_=_1000,_num_zones_=_3,_shard_size_=_10-32                     0.000 ± 0%      0.000 ± 0%       ~ (p=1.000 n=10) ¹
Ring_ShuffleShardCached/num_instances_=_1000,_num_zones_=_3,_shard_size_=_30-32                     0.000 ± 0%      0.000 ± 0%       ~ (p=1.000 n=10) ¹
Ring_ShuffleShard_512Tokens-32                                                                      75.00 ± 0%      75.00 ± 0%       ~ (p=1.000 n=10) ¹
Ring_Get-32                                                                                         0.000 ± 0%      0.000 ± 0%       ~ (p=1.000 n=10) ¹
geomean                                                                                                        ²                -0.00%                ²
¹ all samples are equal
² summaries must be >0 to compute geomean

Which issue(s) this PR fixes: Fixes #7509

Checklist

  • Tests updated
  • Documentation added
  • CHANGELOG.md updated

Comment thread pkg/ingester/active_series.go Resolved
Comment thread pkg/ingester/active_series_test.go Resolved
Comment thread pkg/ingester/active_series.go Resolved
Comment thread CHANGELOG.md Resolved
@anant10
anant10 requested a review from a team as a code owner October 7, 2026 01:04
@anant10
anant10 requested a review from yeya24 October 7, 2026 01:04
@anant10
anant10 force-pushed the feat/owned-series-tracking branch 2 times, most recently from 640b980 to 3720c7b Compare October 10, 2026 01:14

@sandy2008 sandy2008 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the detailed write-up. I traced the ownership path and probed a few cases. Ownership matches routing for the default strategy but not for shuffle-sharded tenants. The instance-wide max_series check misses series created between update cycles. And an entry created during a ring change can keep the old ownership. Details inline.

Comment thread pkg/ingester/ingester.go Outdated
userDB.activeSeries.Purge(purgeTime)
if i.cfg.OwnedSeriesMetricsEnabled {
// Use UpdateMetrics which handles both purge AND ownership re-evaluation.
ringTokens, ownedPositions, ringFingerprint := i.lifecycler.GetOwnedTokenPositions()

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

With shuffle sharding this ownership doesn't match routing. GetOwnedTokenPositions is one bitmap for the whole ring, but for a tenant with ingestion_tenant_shard_size > 0 the distributor routes within ShuffleShard(userID, shardSize), whose replica sets differ from the full ring's. I checked with a quick probe in pkg/ring (30 instances, 3 zones, RF 3, 2000 random keys): 0 of 6000 placements from Ring.Get are reported as not owned, but 4756 of 6000 placements from a size-6 tenant subring's Get are. With -ingester.owned-series-limit-enforcement-enabled, such tenants would be checked against a fraction of their real series, while the limiter already scales their local limit to the shard size. Could ownership be computed against the tenant's subring when shuffle sharding applies, or could enforcement fall back to the head series count for those tenants?

Comment thread pkg/ingester/ingester.go Outdated
// off. Series without a tracker entry, such as those restored from the WAL,
// are already counted as owned by the cycle. A count of zero is a real count
// and is used as-is.
if owned := u.instanceOwnedCount.Load(); u.ownedSeriesLimitEnabled && owned >= 0 {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

With enforcement on, this check uses instanceOwnedCount, which is only stored once per updateActiveSeries cycle (1m by default), while PostCreation only increments instanceSeriesCount. So series created since the last cycle aren't counted at all. The staleness is conservative after a ring change, but not for new series. I checked with a probe: instance max_series: 2, one series pushed, one cycle (snapshot 1), then three new series. All three were accepted and the head ended at 4, whereas with enforcement off the second and third were rejected with "max series limit reached". Since this limit is the OOM guard, could the check add the growth since the snapshot? For example, record the in-memory series count when the snapshot is taken and add max(0, current - recorded).

Comment thread pkg/ingester/active_series.go Outdated

c.stripes[stripeID].updateSeriesTimestamp(now, series, hash, nativeHistogram, labelsCopy)
// Load ring state atomically — readers on the push path always see a consistent snapshot.
state := c.ring.Load()

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The ring state is loaded here, before findOrCreateEntryForSeries takes the stripe lock and computes owned. If a ring change lands in between, UpdateMetrics publishes the new ring and rescans the stripe before the entry exists. The entry is then created with the old ownership, and later cycles see tokensChanged == false and never re-evaluate it. In a stress probe (32 goroutines creating new series, one ring change from nothing owned to everything owned partway through), 224 of 300 trials ended with entries stuck as not owned, about two per trial, and they stay that way until the next ring change. Loading c.ring inside findOrCreateEntryForSeries, after s.mu.Lock(), should close this. updateTokens stores the new ring before the scan locks each stripe, so an insert that lands after its stripe's scan sees the new ring, and one that lands before it gets re-evaluated by the scan.

Comment thread pkg/ingester/ingester.go Outdated
@@ -1682,7 +1824,13 @@ func (i *Ingester) Push(ctx context.Context, req *cortexpb.WriteRequest) (*corte
isNHAppended := succeededHistogramsCount > oldSucceededHistogramsCount
shouldUpdateSeries := (succeededSamplesCount > oldSucceededSamplesCount) || isNHAppended
if i.cfg.ActiveSeriesMetricsEnabled && shouldUpdateSeries {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Owned tracking also depends on -ingester.active-series-metrics-enabled. With it off, nothing is recorded here, and updateLoop never runs updateActiveSeries. So with both owned flags on, cortex_ingester_owned_series is never emitted and both limits quietly fall back to the head counts. A probe push left head=1, Tracked()=0 and the instance snapshot at -1. Could Validate reject that combination, or could the flag help say it requires active series metrics?

Comment thread CHANGELOG.md Outdated
* [CHANGE] Alertmanager: Remove the obsolete startup migration of local state files into per-tenant directories (scheduled for removal in 1.11.0). Upgrading from a release older than 1.9.0 with a persisted local state directory now requires upgrading to an intermediate release first, so the migration can run. #7513
* [CHANGE] Cache: Setting `-blocks-storage.bucket-store.metadata-cache.bucket-index-content-ttl` to 0 will disable the bucket-index cache. #7446
* [CHANGE] HA Tracker: Move `-distributor.ha-tracker.failover-timeout` from a global config to a per-tenant runtime config. The flag name and default value (30s) remain the same. #7481
* [FEATURE] Ingester: Add experimental owned series tracking to prevent false throttling during ingester scale-up and ring resharding. The ingester counts only the series the ring assigns to it, rather than everything left in the TSDB head, closing a window of up to one head compaction cycle in which a tenant could be throttled against a limit that had already shrunk. Enabled via `-ingester.owned-series-metrics-enabled` and `-ingester.owned-series-limit-enforcement-enabled`. #7509

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This entry is under ## 1.22.0 in progress. New entries go under ## master / unreleased and end with the PR number (#7758) rather than the issue number.

… during ring changes

When ingesters scale up, the per-ingester local series limit drops immediately
but stale series data remains in TSDB head for up to 2 hours. This causes
PreCreation() to incorrectly reject new writes.

This PR introduces owned series tracking in ActiveSeries:
- Each series stores its ring token (computed via TokenForLabels)
- Ownership is evaluated against current ring state on each update cycle
- When ring changes, unowned series are excluded from limit enforcement

Two feature flags for safe rollout:
- owned_series_metrics_enabled: enables cortex_ingester_owned_series metric
- owned_series_limit_enforcement_enabled: switches PreCreation to use owned
  count for both per-user and instance-level max_series limits

Key design decisions:
- Zone-local ownership check via SearchToken + instance token map lookup
- Ring state stored behind atomic.Pointer[ringState] for lock-free reads
  on the hot push path
- instanceOwnedCount recalculated every ~1 min (not incremental) to avoid
  drift from edge cases
- Startup fallback: when instanceOwnedCount==0, uses instanceSeriesCount

Code consolidation:
- FNV hash functions consolidated into pkg/util/fnv.go (single source)
- Sharding functions moved to pkg/ring/token.go (eliminates duplication
  between distributor and ring packages)

Validated under sustained write load in a multi-zone deployment: after a
scale-up, the owned count on the existing ingesters dropped in proportion
to the redistribution while their head series count stayed high, and no
writes were rejected.

Fixes cortexproject#7509

Signed-off-by: Anant Shanbhag <anantvas@amazon.com>
Ring.Get determined the replica set for a key by inlining the ring walk
directly in its body. The walk is the single definition of "which
instances hold this data", and the owned-series work needs to ask that
same question from a second place: for every token position, rather than
for one key.

Duplicating the walk there would let the two copies disagree, and a
disagreement is exactly the bug that matters, because an ingester would
then count series against its limit that the distributor does not route
to it.

Move the walk into ringTopology.replicaSetAt, keyed on a token position
rather than a key, and have Ring.Get call it after resolving the key to a
position with SearchToken. ringTopology is a read-only view that copies
only slice and map headers, so it can be built per call.

replicaSetAt also returns the instance IDs alongside the descriptors.
InstanceDesc carries no ID of its own and an instance's ID is not
interchangeable with its Addr, so a caller asking "am I in this set?" can
only answer that correctly by comparing IDs.

The per-zone cap is now skipped when the ring reports no zones at all,
rather than dividing by the zone count unconditionally. Ring.Get cannot
reach that state for a non-empty ring, so this is a robustness change for
the new caller, not a behaviour change here.

No behaviour change. The existing pkg/ring suite passes unmodified, and
BenchmarkRing_Get is unchanged at ~607 ns/op and 0 allocs/op.

Signed-off-by: Anant Shanbhag <anantvas@amazon.com>
Add OwnedTokenPositions, which returns the ring's sorted token list plus a
parallel bitmap of the positions owned by a given instance. Ownership is
computed with replicaSetAt, the same walk Ring.Get uses, so an instance's
view of what it owns cannot drift from where the ring routes.

This replaces the "am I the first instance in this token range, among the
token list for my own zone?" check the owned-series tracking previously
used. That check has exactly one owner per token range, which is only the
same thing as the replica set in one configuration: zone awareness enabled
with a replication factor equal to the zone count. In every other shape,
including the default of zone awareness disabled, it undercounts by
roughly the replication factor.

Ownership deliberately does not apply the replication strategy's health
filter. It has to stay stable across heartbeat flapping, because a series
does not stop being this instance's responsibility when a peer misses a
heartbeat.

The lookup is O(tokens x replicationFactor) and is intended to be called
once per ring change, so that the per-series question becomes
owned[SearchToken(tokens, key)]: a binary search plus an array index.
BenchmarkOwnedTokenPositions covers that precompute, at ~29.7 ms for 100
instances holding 512 tokens each.

Tests assert the load-bearing property directly: across ten ring shapes
covering zone awareness on and off and replication factors above, below,
equal to and not divisible by the zone count, an instance owns a token
position if and only if Ring.Get for a key in that position includes it.
A conservation test asserts that every position is owned by exactly
replicationFactor instances, which is what makes the per-ingester counts
sum correctly against a limit that is itself scaled by the replication
factor.

One test builds a ring whose instance IDs and addresses deliberately
differ. The shared fixtures set Addr equal to the instance ID, so an
implementation comparing addresses would pass every other test here and
then own nothing at all in a real ring.

Signed-off-by: Anant Shanbhag <anantvas@amazon.com>
Replace the ownership check used by owned-series tracking. It previously
asked "is the first token at or after my series' token, within the token
list for my own zone, one of my tokens?". That has exactly one owner per
token range, which only coincides with the ring's replica set when zone
awareness is enabled and the replication factor equals the zone count.

In every other configuration, including the default of zone awareness
disabled, it is wrong. With zone awareness off, every instance falls into
one unnamed zone, the "zone-filtered" token list becomes the whole ring,
and the check degenerates to "am I the single global primary?". That counts
roughly 1/replicationFactor of the series the instance actually holds,
while the limit it is compared against is scaled up by the replication
factor. The net effect is a limit that is too permissive by about the
replication factor, so an ingester runs out of memory instead of
throttling. That is a worse failure than the stale-count problem this
feature exists to fix.

Ownership now comes from ring.OwnedTokenPositions, which walks the ring
with the same function Ring.Get uses, so the set of series an ingester
counts is by construction the set the distributor routes to it. Because
the walk honours the operation's replica-set extension rules, instances in
JOINING, LEAVING and READONLY are handled exactly as the write path
handles them, rather than via a separate READONLY special case.

The lifecycler computes the bitmap in updateCounters, alongside the
healthy-instance count that the limiter already uses as the denominator of
the same comparison. Both therefore come from one ring snapshot taken at
one instant under one lock, so numerator and denominator cannot be drawn
from different views of the ring.

The walk is too expensive to repeat on every heartbeat, and heartbeats are
by far the most common reason updateCounters runs, so it is gated on a
fingerprint covering instances, zones, states and tokens. Desc.RingCompare
cannot serve here: it reports a state change and a timestamp change as the
same result, and ownership depends on state. The recompute deliberately
runs outside countersLock, since HealthyInstancesCount is on the push path
and must not block behind it.

Per-series cost drops from a map lookup to a binary search plus an array
index, and isOwned no longer indexes the token slice without checking its
length, which panicked on an empty ring.

Tested with the upstream CI invocation, which matters here:
go test -tags "netgo slicelabels" -race. Without the slicelabels tag
pkg/ingester panics in cortexpb.CopyLabels, and without -race the
//go:build !race file is compiled in and panics too; both reproduce on
unmodified upstream.

Signed-off-by: Anant Shanbhag <anantvas@amazon.com>
Owned-series tracking previously refused to track a series the ring did not
assign to this ingester, and deleted entries when a ring change took
ownership away. That made cortex_ingester_active_series silently drop those
series, changing the meaning of a long-standing metric, and it made owned
and active identical by construction, which left the staged rollout across
the two feature flags unable to show any difference.

Ownership now gates exactly one counter. Every series is tracked and counts
towards active exactly as before; only owned is conditional. Losing
ownership moves a series out of owned and leaves it active, which is
accurate, because the series is still held in this ingester's head.

Retention is now two-tier. The periodic cycle recounts but removes nothing,
and entries are released by a purge after head compaction, keyed on the
head's new minimum time. An entry therefore lives for as long as the series
it describes is in memory.

This is what makes the feature work for a tenant with high churn. Tying
owned to the idle timeout meant a tenant with a 3M-series head but only
500K recently active series reported owned as 500K, so a 3M limit would
admit 2.5M more series on top of the 3M already resident: more permissive
than the unmodified code it replaces. Anchoring to the head instead makes
owned a measure of what the ingester is actually storing, which is what a
series limit exists to bound. The consequence is that owned may exceed
active for such a tenant, which is intended and is the reason the two
counts are maintained separately rather than one being derived from the
other.

Totals are cached per tenant and maintained as series are created, because
PreCreation consults them for every new series and previously took a read
lock on all 512 stripes to do so. The periodic cycle recomputes them
authoritatively, so any drift is corrected within one cycle.

Limit enforcement follows. The per-tenant check no longer needs its
"owned > 0" guard: the cached total is accurate from the first sample
rather than only after the first periodic cycle, so there is no startup
window to paper over, and the guard had a cliff where a genuinely empty
tenant silently fell back to the head count. The instance-level check keeps
a fallback, because the instance total really is unavailable until the
first cycle runs, but it is now keyed on an explicit negative sentinel
rather than on zero, so a true zero is no longer confused with "not yet
computed".

Measured on 100k series: the periodic recount is 1.67ms with no
allocations when the ring is unchanged and 2.42ms when every entry's
ownership is re-evaluated. The push path is unchanged within noise,
~440ns/op with the ring loaded against ~400-457ns/op without it.

Signed-off-by: Anant Shanbhag <anantvas@amazon.com>
The owned-series work replaced this file wholesale, removing 139 lines:
three tests covering the active count and purge behaviour, and four
benchmarks covering the push and purge paths. Removing the benchmarks meant
there was no longer a baseline to compare the new per-series ownership cost
against, which is what the review asked about.

All of them are restored, with the ring token argument added. They pass a
token of zero with no ring loaded, so ownership is unknown and every series
counts as owned, which means they exercise precisely the behaviour that
predates this feature.

Restoring them surfaced that BenchmarkActiveSeries_UpdateSeries does not
run at all. It calls b.Loop() in two separate loops, which is not allowed,
and sizes its series slice from b.N before b.Loop() has established an
iteration count. It panics with "index out of range [1] with length 1", or
fails with "B.Loop called with timer stopped" when an explicit -benchtime is
given. This reproduces on unmodified upstream at this branch's merge base,
and the same code is present verbatim on master, so it has been broken since
the b.Loop() migration rather than by this change. It is converted to the
b.N form, which is the correct shape when the iteration count is needed
before the timed loop begins.

New coverage for the owned-series semantics:

  - active is unaffected by ownership. The same series are pushed into two
    trackers, one with a ring where half the tokens belong elsewhere and one
    with no ring, and the active counts must agree exactly. This is the
    regression test for the metric's meaning being preserved.
  - owned exceeds active for a tenant whose series have gone idle but are
    still held in the head, which is the behaviour the limit fix depends on.
  - the periodic cycle retains expired entries, repeated cycles are stable
    rather than progressively dropping entries, and only a purge releases
    them.
  - losing and regaining ownership moves series between owned and not-owned
    without touching active and without deleting anything, so ownership can
    return without the series being re-pushed.
  - the cached totals match the per-stripe counters after creation, after a
    periodic cycle, after a purge and after a clear, since those totals are
    what limits are enforced against.
  - ownership change detection fires when the bitmap changes even though the
    token list does not, which is the case a token-list hash cannot see.
  - isOwned handles an empty ring and a mismatched bitmap without panicking,
    which the previous ownership check did not.

Two benchmarks are added for the new paths: the push path with and without
a ring loaded, to show the per-series ownership cost, and the periodic
recount separating the common unchanged-ring case from a full ownership
re-evaluation.

Signed-off-by: Anant Shanbhag <anantvas@amazon.com>
Ownership is computed from the raw replica-set walk and deliberately does
not apply the replication strategy's health filter. That makes ownership
differ from routing while an instance is transitioning, and until now that
difference was only described in a comment rather than pinned by a test.

For each of JOINING, LEAVING and READONLY, assert that the transitioning
instance still owns its token ranges, that those ranges end up with more
owners than the replication factor because the replica set is extended, and
that writes still land on exactly replicationFactor healthy instances and
never on the transitioning one.

Both the transitioning instance and the instance the walk extends to really
are holding that data, so both counting it is correct rather than double
counting: the departing instance still has the series in its head, and the
distributor really is writing to the extension.

Filtering for health here would instead make an instance's own series count
depend on whether its peers happened to miss a heartbeat, which is why the
asymmetry with Ring.Get is intentional.

Note that the aggregate invariant asserted by
TestOwnedTokenPositions_Conservation, that every position has exactly
replicationFactor owners, holds only for a ring where every instance is
ACTIVE. This test covers the complementary case.

Signed-off-by: Anant Shanbhag <anantvas@amazon.com>
The CHANGELOG carried three entries for this change. The two ENHANCEMENT
lines described internal refactoring, moving the sharding helpers into
pkg/ring/token.go and the FNV helpers into pkg/util/fnv.go, which is not
something an operator reading release notes needs. Collapsed to the single
FEATURE entry, reworded to describe the effect rather than the mechanism,
marked experimental to match how the flags are listed in v1-guarantees, and
with the "a up-to-2-hour" typo removed. It now says "up to one head
compaction cycle", which is what the window actually is rather than assuming
the default block range.

The two new flags were never added to the generated configuration
documentation, so `make check-doc` fails: it regenerates the docs and asserts
`git diff --exit-code` over the config reference, the blocks-storage pages
and the JSON config schema. Regenerated, which adds the twelve expected lines
to docs/configuration/config-file-reference.md and the matching entries to
schemas/cortex-config-schema.json. No other generated file changes.

Also listed the feature under experimental features in
docs/configuration/v1-guarantees.md, alongside the other opt-in ingester
metrics, so the stability expectation is explicit while it is behind flags.

Signed-off-by: Anant Shanbhag <anantvas@amazon.com>
Head-anchored retention compares two different clocks. An entry's timestamp
is the wall-clock time a sample for it arrived, while Head().MinTime() is a
sample timestamp. They agree for real-time ingestion, which is why this works
in the common case, but they diverge for a tenant whose samples are backdated
or whose clock is skewed.

Only one of the two directions is dangerous. A cutoff that lands too early
retains entries for longer than necessary, which over-counts owned series and
errs towards enforcing limits sooner: wasteful, not harmful. A cutoff that
lands too late drops entries for series which are still resident, which
under-counts owned series and lets a tenant exceed its limit. A tenant whose
sample timestamps run ahead of real time would hit exactly that, and in the
extreme every entry would be released on each compaction, since no arrival
time can be in the future.

Clamp the cutoff so it is never more recent than one block range ago.
Anything still in the head must have arrived within roughly that window, so
this cannot release an entry for a resident series however far ahead a
tenant's timestamps run. Backdated timestamps are left alone, since they
already err in the safe direction.

This is a bound rather than a translation between the clocks. Making the
comparison exact would mean recording each series' last sample timestamp on
its entry, which costs memory per series on a structure whose retention cost
is already the open question here, so it is deliberately left for the design
discussion on the pull request rather than decided unilaterally.

The cutoff is computed by a free function so the behaviour can be asserted
without standing up a TSDB. Tests cover each direction of skew and state the
property the clamp exists for: a series which arrived within the last block
range is never released, whatever the head reports.

Signed-off-by: Anant Shanbhag <anantvas@amazon.com>
Retaining entries past the idle window introduced one way for the active
count to behave differently from before, which I had not handled.

Previously a series going idle had its entry removed, so a later sample
created a fresh entry and the active count reflected the series at once. Now
the entry survives and merely stops being counted, so a later sample updates
it in place and creates nothing. The cached total is only bumped on creation,
so a returning series went unnoticed until the next periodic recount,
under-reporting active for up to one update period, a minute by default.

The exported cortex_ingester_active_series gauge was not affected, because it
is set inside the periodic loop immediately after the recount. The two other
readers were: the native histogram limit consulted by PreCreation on every
new series, and the active series figure in the per-tenant stats endpoint.
Both would have read low, making that limit marginally too permissive.

UpdateMetrics now publishes the idle cutoff it used, and the push path uses it
to notice an entry crossing back into the window. The compare-and-swap that
moves the timestamp across the cutoff decides which caller counts the series,
so concurrent samples for the same returning series count it once. Whether
the series is a native histogram is taken from the entry rather than from the
incoming sample, since an entry's kind is fixed when it is created.

Only the active counts change on reactivation. An idle series never stopped
being owned, because owned deliberately ignores the window.

The cutoff is published as zero until a cycle has run with one, so when
entries are released by the idle timeout rather than retained, the
reactivation path never engages and the original behaviour of creating a
fresh entry is preserved exactly.

Tests assert active reflects the returning series before any recount, that a
second sample inside the window does not count it twice, that the subsequent
authoritative recount agrees with what the push path did, and that the cached
totals still match the per-stripe counters. Verified to fail without the
change. A concurrency test pushes the same returning series from 64
goroutines and asserts it is counted once. The push path is unchanged within
noise at ~470ns/op.

Signed-off-by: Anant Shanbhag <anantvas@amazon.com>
Three behaviours were only visible by reading the implementation.

Enabling owned series tracking is documented as turning on a metric, but it
also changes how long active series entries are retained: they are released at
head compaction rather than after the idle timeout, so that the owned count
reflects what is held in memory rather than only what is recently active. That
increases ingester memory roughly in proportion to the ratio between series in
the TSDB head and recently active series. An operator enabling what reads like
an observability flag should not have to discover that from a heap profile.

The owned count may exceed the active count for a tenant with high churn,
because it includes idle series that are still held. This is intended, and
cortex_ingester_active_series keeps its existing value, but anything asserting
that owned is a subset of active will be surprised.

Limit enforcement applies to the instance-wide max_series limit as well as to
per-tenant limits, which a single flag name does not convey. The consequence
worth stating is that max_series then stops counting series reassigned to
another ingester which are still resident until the next head compaction, so
it no longer reflects everything the ingester is holding.

No behaviour change. Flag help text, the regenerated configuration reference
and schema, and the experimental features list.

Signed-off-by: Anant Shanbhag <anantvas@amazon.com>
cortex_ingester_owned_series carries a user label but was absent from both
places that clean up per-tenant metrics: deletePerUserMetrics, and the TSDB
close path. Its three sibling per-tenant gauges are removed in both.

A stale gauge therefore remained for every tenant whose TSDB was ever closed,
by idle cleanup or deletion. On an ingester that sees tenants come and go the
label cardinality of its own /metrics grows without bound, and dashboards keep
reporting owned series for tenants that no longer exist.

Signed-off-by: Anant Shanbhag <anantvas@amazon.com>
numZones was derived as len(getTokensByZone()), which merge-sorts every token
in the ring into one slice per zone just to take the length of the resulting
map. Counting distinct Zone values over the instances is proportional to the
number of instances rather than the number of tokens, and allocates nothing.

For 100 instances holding 512 tokens each this takes the once-per-ring-change
precompute from 29.7ms to 26.4ms and from 715 allocations to 489.

It also removes one of the two calls that sort each instance's token slice in
place on the shared Desc as a side effect.

Signed-off-by: Anant Shanbhag <anantvas@amazon.com>
…aching

The previous revision cached one cross-stripe total per counter and had the
push path increment it, while the periodic recount summed the stripes and
stored the result. That store is not atomic with respect to the scan it was
computed from, so a series created between a stripe being scanned and the total
being stored was lost, and one created just before its stripe was scanned could
be counted twice. The next complete recount repaired it, so the drift was
transient, but the counts were only eventually correct rather than correct.

It also diverged from the behaviour it replaced even with both feature flags
off. Previously Active and ActiveNativeHistogram summed the stripes under a
read lock and were exact at all times; with the cache they could drift during
each purge pass, which is a change nobody opting out of this feature asked for.

Make the per-stripe counters atomics and total them in the accessors. Every
mutation still happens while holding that stripe's lock, so nothing can be
lost, and readers no longer need to acquire 512 locks: totalling 512 atomics
costs about 290ns with no allocations. That is cheaper than the read-locked
version it replaces as well as being exact, so the cache was solving a problem
that did not need solving at the cost of one that did.

The cross-stripe cache, and the plumbing that reported back from
updateSeriesTimestamp purely to maintain it, are gone. Reactivation still has
to be counted explicitly, because an entry crossing back into the active window
is not a creation, but it now increments the stripe counter under the stripe
lock like every other mutation.

A test asserts the counts stay exact while a recount runs concurrently with
sixteen writers. It does not reproduce the transient drift of the cached
design, since that repairs itself before any assertion can observe it; it pins
the steady-state invariant.

Signed-off-by: Anant Shanbhag <anantvas@amazon.com>
purge was given one cutoff and used it both to decide what to delete and to
decide what counts as active. That is correct for its original caller, which
passes the idle timeout, but owned-series tracking added a second caller that
passes a cutoff derived from the head's minimum time, which can be hours old.
Every retained entry then counted as active, so after each head compaction the
active count jumped to the owned count until the next periodic recount.

Two readers see that window, and neither waits for the recount:

  - PreCreation consults ActiveNativeHistogram on every new series, so a tenant
    with max_native_histogram_series_per_user set would get spurious
    limit-exceeded errors for up to a minute after each head compaction.
  - The per-tenant stats endpoint reports the inflated active count, which also
    contradicts the documentation added in this revision stating that the active
    series value is unchanged.

Take the two cutoffs separately: deleteBefore decides what is still held,
activeCutoff decides what is inside the idle window. The original caller passes
the idle timeout for both, which is exactly its previous behaviour, so nothing
changes with the feature disabled.

The regression test fails if the cutoffs are conflated again.

Signed-off-by: Anant Shanbhag <anantvas@amazon.com>
Active series entries are only created when a sample arrives: UpdateSeries is
called from the push path and nowhere else, and nothing seeds the tracker from a
restored head. An ingester that has just replayed its WAL therefore has a full
head and an empty tracker, so Owned() reads zero.

The previous revision removed the "owned > 0" guard that had been covering this,
on the reasoning that the count is maintained as series are created and so is
accurate from the first sample. That is true for series created by a push and
false for every series restored from the WAL. With the guard gone, a tenant could
create a full limit's worth of series on top of everything already resident,
roughly doubling the intended memory, until enough of the restored series
happened to receive a sample. Unmodified Cortex is not exposed to this, because
it compares the limit against Head().NumSeries(), which is correct the moment
replay finishes.

The instance-wide max_series limit had the same hole, and the negative sentinel
added in this revision did not close it: one cycle after a restart the periodic
loop stored a genuine zero, so the OOM guard passed everything.

Track the number of entries held, and only use the owned count when it covers
what is in memory:

  - per tenant, use Owned() only when Tracked() is at least Head().NumSeries()
  - instance wide, publish the sentinel rather than a total whenever the trackers
    between them hold fewer entries than there are series in memory

Falling back to the in-memory series count over-counts, which delays a scale-up
from taking effect but cannot let a tenant past its limit. Guarding on coverage
rather than on the count being zero also removes the cliff the old guard had,
where a genuinely empty tenant was indistinguishable from an unknown one.

Tracked is a useful number in its own right: it is the size of the structure
whose retention this feature extends, which was otherwise not observable.

Signed-off-by: Anant Shanbhag <anantvas@amazon.com>
Head-anchored retention decided which active series entries to release at
head compaction by comparing each entry's timestamp against a cutoff derived
from the head's minimum time. Those are two different clocks: an entry's
timestamp is the wall-clock time a sample arrived, while the head's minimum
time is a sample timestamp. An earlier change clamped the cutoff to at most
one block range before now, to avoid releasing entries for resident series
when sample timestamps run ahead of real time.

In practice the clamp binds on every compaction and over-counts
substantially. Compaction runs about an hour after a block boundary, so the
head is truncated to that boundary while the cutoff is pinned an hour earlier.
Every series that went idle in between leaves the head but keeps its entry.
For a tenant with high churn, owned then exceeds the head substantially
after each compaction and stays there until the next one. That makes the
limit stricter than with the feature off, which is the same class of false
throttling the feature exists to remove.

Narrowing the clamp to the creation grace period would shrink the error but
not remove it, and would depend on a per-tenant setting and on distributor and
ingester clocks agreeing. Instead, stop estimating. The head already reports
exactly which series it deleted, through the PostDeletion lifecycle callback
Cortex uses to keep its in-memory series count. Release precisely those
entries there.

Each entry now records the head series reference it describes, refreshed on
every sample. On deletion, an entry is removed only if it still carries the
deleted reference. The head invokes the callback after releasing its own lock,
so a sample can re-create the same labels as a new head series before the
callback runs; that sample moves the entry to the new reference, and the stale
deletion leaves it alone. References are never reused within a TSDB. The push
path now keeps the reference returned by every successful append, rather than
discarding it when a cached reference was used, so a series re-created between
GetRef and Append is still recorded correctly.

The time-based purge of retained entries and its cutoff function are removed.
A backstop purge remains at compaction for entries idle longer than two block
ranges, longer than any series can stay in the head without a sample, so it
cannot under-count and only guards against an entry outliving its series by a
path the callback does not cover.

With owned-series tracking disabled nothing changes: references are not
recorded or allocated, the callback does not touch the tracker, and entries
are still released by the idle-timeout purge.

TestIngester_OwnedSeriesFollowsHeadThroughCompaction drives a real head
compaction with explicit sample timestamps, including a series idle for five
minutes before the truncation point, and asserts the tracker holds exactly the
series left in the head. It fails on the previous code with 3 entries for 1
head series, reproducing the over-count, and passes with this change. Unit
tests cover release on a matching reference, retention across re-creation,
unknown references, and adopting a reference late.

Signed-off-by: Anant Shanbhag <anantvas@amazon.com>
…ding owned

The limit check used the owned count only when the tracker held an entry
for every series in the head, and otherwise fell back to the head count.
That guard was meant for the window after a restart, but every new series
enters the head a moment before its tracker entry is written. Under
concurrent pushes the guard therefore failed during any burst of series
creation, and an ingester that had just lost ownership of most of its head
was throttled against the stale head count: the false throttling the owned
count exists to prevent.

Count series in the head without a tracker entry as owned instead. After a
restart this equals the head count, as before; in normal running it adds
only the series in flight. Apply the same to the instance-level count.

Without this, a large share of new series on the existing ingesters can
be rejected after a scale-up. The new concurrent test rejects 300 of 640
series without this change and none with it.

Signed-off-by: Anant Shanbhag <anantvas@amazon.com>
Reports bytes per entry with owned-series tracking disabled and enabled,
so the memory cost of retaining entries until the head deletes their
series can be estimated as this per-entry cost times the number of idle
series still in the head.

Signed-off-by: Anant Shanbhag <anantvas@amazon.com>
- Use go.uber.org/atomic instead of sync/atomic in active_series.go and
  owned_series_limit_test.go. faillint blocks sync/atomic, which is what
  failed the lint job on the previous revision.
- Keep Lifecycler.getTokens unexported, as on master. Nothing outside
  pkg/ring needs it, and master's lifecycler tests call getTokens.
- Move the CHANGELOG entry under "master / unreleased", reference cortexproject#7758,
  and drop the leftover per-change entries so the feature has one entry.

Signed-off-by: Anant Shanbhag <anantvas@amazon.com>
Instance max_series missed series created between cycles. With enforcement
on, the check read an owned count taken once per updateActiveSeries cycle,
so series created after it were not counted at all. The snapshot now
records how many series had been created when it was taken, and the check
adds every series created since. It counts creations rather than the change
in the in-memory count, because head compaction can delete as many series
as were created meanwhile, which would leave the new ones uncounted. The
result is capped at the in-memory count, so it is never stricter than
counting everything in memory.

A new series could keep stale ownership. The ring state was read before
taking the stripe lock, so a series created during a ring change could be
given the old ownership after its stripe had been recounted, and keep it
until the next ring change. The state is now read under the stripe lock,
which the recount also takes.

Owned-series tracking depended on -ingester.active-series-metrics-enabled.
With that flag off, series were not recorded on push, the periodic recount
did not run and cortex_ingester_owned_series was not registered, so both
owned limits quietly fell back to the head count. The tracker now runs
whenever either flag is set, and the active series flag only decides
whether the active series metrics are exported.

Enabling -ingester.owned-series-limit-enforcement-enabled without
-ingester.owned-series-metrics-enabled still has no effect, but is now
logged once at startup, and the flag help says so.

Signed-off-by: Anant Shanbhag <anantvas@amazon.com>
Add Ring.OwnedTokenPositions, which returns the ring's sorted tokens and a
bitmap of the positions an instance owns, from the same replicaSetAt walk
Ring.Get routes with. Being a method on Ring, it answers for the subring
ShuffleShard returns as well as for the whole ring. A subring picks
different replica sets from the whole ring, so ownership computed over the
whole ring miscounts the series of a shuffle-sharded tenant. It reuses the
ring's token list and instance lookup, so the bitmap is all it allocates.

Add Ring.OwnershipFingerprint, which changes whenever ownership may have.
Tokens, zones and the set of instances only change together with the ring
topology, which updateRingState already timestamps, so the fingerprint
hashes that timestamp and each instance's state rather than every token.
On a subring it covers only the subring's members, so a change elsewhere
in the ring leaves it unchanged.

Add NewRingFromDesc and Ring.UpdateDesc, for a Ring that is updated with a
Desc another component has already read instead of watching the KV store.
A process can then use its own view of a ring it already reads, including
shuffle-shard subrings, without a second watch. With a KV store that is
polled, that watch re-reads the whole ring on every poll. An update
that only changes instance states stays cheap and keeps the cached
subrings, as it does on a watched ring.

The Desc-based OwnedTokenPositions is kept, deprecated, until the
lifecycler stops using it in the next commit.

Signed-off-by: Anant Shanbhag <anantvas@amazon.com>
Owned series were computed over the whole ring for every tenant. For a
tenant with a shuffle-shard size smaller than the ring, the distributor
routes within the tenant's subring instead, which picks different replica
sets. Whole-ring ownership then counts the wrong series: an ingester in a
tenant's shard counts as owned only the fraction the whole ring would also
send it, so with limit enforcement the tenant is checked against a fraction
of the series it holds.

Each tenant's ownership is now computed over the ring the distributor
routes it within: the tenant's subring when shuffle sharding applies and
the shard is smaller than the ring, and the whole ring otherwise. Whole-ring
tenants share one ownership bitmap, as before. Each sharded tenant on this
ingester gets its own, built from its subring, and an ingester outside a
tenant's shard owns none of its series. Until this ingester has joined the
ring, ownership is unknown and every series counts as owned.

The rings are built from the ring the lifecycler already reads on every
heartbeat, not by watching the KV store a second time from every ingester.
The lifecycler keeps a copy of it once asked to, replaced only when the
instances, their tokens, zones or states change, so heartbeats alone copy
nothing. This is also the view of the ring the limiter computes local
limits from, so a ring change reaches the local limit and the owned count
together.

Ownership is recomputed when the ring changes and when a tenant's shard
size changes. The shard size matters on its own: a larger shard moves some
of the tenant's series to the new members while the old ones still hold
them, and the shard size can change at any time through the runtime
overrides.

Changes are applied without waiting for the next active series update.
Every -ingester.owned-series-recompute-min-interval (default 15s) the
ingester checks whether the ring or any tenant's shard size changed. The
check costs about a millisecond with 50 sharded tenants in a ring of 1000
instances. Work is kept to what actually changed, because during a rolling
restart instance states change every few seconds:

- A sharded tenant is recomputed only when an instance in its own shard
  changed, not on every change elsewhere in the ring.
- When a recomputed bitmap equals the previous one, the tenant keeps its
  ownership, and its series are not re-evaluated.

Ownership moves out of the lifecycler, which computed it on every
heartbeat that changed the ring even though it was used once a minute. The
Desc-based OwnedTokenPositions is removed.

Signed-off-by: Anant Shanbhag <anantvas@amazon.com>
A new series now counts as owned as soon as it is created. It reached this
ingester because the distributor routed it here, which is the question
ownership answers. Judging it by this ingester's own view of the ring could
count it as owned by nobody: that view is only refreshed when the
lifecycler heartbeats, while the distributors see ring changes sooner, so
after this ingester gains token ranges, on a scale-down or while a peer
restarts, new series arrive in ranges its stale view does not give it, and
they escaped both series limits until the next heartbeat. Existing series
are still re-evaluated against the ring whenever it changes. A new entry
also no longer reads the ring state at all, so no ordering against a
concurrent recount can leave it with stale ownership.

For the same reason an ingester outside a tenant's shard owns none of the
series it holds for the tenant: if it and the distributors disagree about
the shard, new series that arrive still count, as they are created.

A recomputed bitmap is compared with the previous one by value. A topology
change rebuilds every subring with a new token list, even for a shard whose
members did not change, and comparing the lists by identity made every
sharded tenant re-evaluate every series it held on each topology change.
When the bitmap is unchanged the tracker keeps its ownership and adopts the
new slices, so the previous ring's can be freed.

Ownership is computed with the operation the distributor writes with, which
follows -distributor.extend-writes, passed to the ingester the way
-distributor.shard-by-all-labels is.

A series' ring token is computed only when its tracker entry is created,
and outside the stripe lock, instead of for every series on every push. If
it cannot be computed, as for a series with no metric name when sharding by
metric name, the series is tracked as owned instead of failing the push.

The per-tenant owned count is capped at the head. The head deletes a series
before telling the tracker, so for a moment the tracker can hold more
entries than the head holds series.

The frequent recount compares each tenant's ownership with the one its
tracker actually has, rather than with the one last applied to the tenant:
a TSDB closed while idle and opened again has a new tracker with no
ownership, which the recount used to skip until the next active series
update. A tenant whose ownership cannot be computed is left alone and
retried on the next check. When a tenant's shard size changes, the subring
cached for the previous size is dropped.

The ring built from the lifecycler's copy no longer computes detailed ring
metrics, which nothing exports. -ingester.active-series-metrics-update-period
must be positive when owned or active series are tracked. The flag help and
v1-guarantees now say that new series count as owned on arrival, that ring
changes are seen on this ingester's heartbeat, and what the ring copy costs
in memory. Stale comments are corrected, and the unused clear() and
HashAddUint and HashAddUint32 are removed.

Test series names are scrambled, so that they spread over the token space
rather than hashing into clusters, which made the scale-up assertions
depend on luck.

Signed-off-by: Anant Shanbhag <anantvas@amazon.com>
This ingester's view of the ring is the one its lifecycler reads on each
heartbeat, which is also the view its local limits are computed from. Two
problems remained with using it for owned series.

The owned count lagged the limits. After a scale-up the local limit dropped
as soon as the lifecycler read the new ring, while the owned count waited
for the next recount, up to -ingester.owned-series-recompute-min-interval
later, and the tenant could be throttled in between. The ingester now
checks every second whether the lifecycler has read a changed ring, a
pointer comparison, and recounts straight away, so the two change together.

A stale view could un-count series routed here. Distributors refresh the
ring more often than the lifecycler, so after this ingester gains token
ranges new series arrive before its view shows them as its own. They counted
as owned on arrival, but the next recount judged them by the stale view: the
first ownership applied to a new or reopened tenant, or a shard size change.
A tenant could then fill its limit again. The lifecycler now numbers its
ring reads, each number taken before the read starts, and a series records
the number current when it arrived. A recount only judges a series by a view
read after it arrived; until then it stays owned, and the next read judges
it, even if the ring did not change. A series older than the view is judged
as before, so series which moved away still stop counting.

A subring's ownership fingerprint now hashes its instances, zones and tokens
instead of the ring's topology timestamp. An instance that restarts with a
new address is a topology change to the ring, but changes no replica set, so
it no longer recomputes the ownership of every sharded tenant.

Also:
- Rename -ingester.owned-series-recompute-min-interval to
  -ingester.owned-series-recompute-interval: it is a period, not a minimum.
  It now only governs shard size changes.
- Require a positive lifecycler heartbeat period, the source of the ring
  view, when owned series are tracked.
- Ring.UpdateDesc does nothing on a ring watching the KV store, where it
  would race with the watch.
- Treat a ring with instances but no tokens yet as unknown ownership.
- Remove the tracker's atomic ring state, which nothing reads concurrently
  any more, and the test of the race it guarded, which can no longer fail.
- Keep HashNew32, HashAdd32 and HashAddByte32 in pkg/ingester/client as
  deprecated wrappers, so that code importing them from there still builds.
- Document that ingesters need the distributors' sharding strategy,
  shard-by-all-labels and extend-writes settings, that a new series counts
  as owned on arrival, and the memory the ring copy takes.

Signed-off-by: Anant Shanbhag <anantvas@amazon.com>
…ries

Series restored from the WAL after a restart had no tracker entries, so they
all counted as owned until head compaction dropped them. A rolling restart
soon after a scale-up therefore brought back the throttling owned series is
meant to prevent, on every restarted ingester, for up to a head compaction
cycle. After the WAL replay and before the ingester joins the ring, every
series in each tenant's head is now given a tracker entry: owned, not
active, since it has had no sample since the restart, and with its kind set
by its first sample. They predate every ring read of the process, so the
first view of the ring judges them, even after a crash, when the instance
is still ACTIVE in the ring and its next read is the first heartbeat.

All owned-series work now runs on its own goroutine instead of the
ingester's update loop. Every second it recounts the tenants whose
ownership changed, because the lifecycler read a changed ring or a tenant's
shuffle-shard size changed, which is cheap when nothing did; every active
series update period it recounts every tracker. On the update loop, a ring
walk of a few hundred milliseconds in a large ring delayed the instance
ingestion rate tick, which assumes a steady interval and so over-reported
the rate, and the recount could wait behind the once-a-minute update. A
shard size change was only checked every
-ingester.owned-series-recompute-interval, although it changes the tenant's
local limit at once, so the tenant could be throttled until then. That flag
is removed: the check runs every second.

Ownership now records the lifecycler read it was computed from. A recount
for which ownership could not be computed no longer judges waiting series by
the tracker's previous bitmap, nor records the newer read as applied.

The lifecycler keeps only the newest of overlapping ring reads, as its first
heartbeat can overlap another read, and its copy of the ring is no longer
replaced when only an instance's address changes.

Active series counts match those without owned-series tracking in two more
cases:
- A retained entry which becomes active again takes the kind of the sample
  that reactivated it. Without retention it would have been released when
  idle and re-created with that sample's kind, so a series which switched
  between float and native histogram samples was counted as its old kind,
  and could trip the native histogram series limit, until head compaction.
- The backstop purge after head compaction counts active series against the
  cutoff the last recount published, which the push and deletion paths also
  use. Counting against a newer one put the counters out of step with them
  until the next recount: a reactivation in between was missed, and a
  deletion in between could take the active count below zero.

Benchmarking against the merge base found three costs, two of them paid
with the feature off:

- Updating a series the tracker already holds, which runs for every
  sample, was about 17% slower than without this change, and more than
  that under contention. Scanning a stripe copied each entry, the ring
  token was passed by value through calls that do not inline, and the
  active cutoff was loaded for every sample. The scan now takes the
  address of each entry, the token and the cutoff are passed by
  reference, and the cutoff is read only once the timestamp has moved,
  since a sample that merely advances a timestamp needs neither.
- Every ring computed a topology fingerprint on every rebuild, including
  the rings in distributors, queriers and store-gateways, which never ask
  about ownership: building an uncached shuffle shard was 4-10% slower
  and applying a topology change 2-5% slower, fleet-wide. It is now
  computed by the first call to OwnershipFingerprint after a topology
  change, so a ring nobody asks does not compute it at all.
- The per-tenant limit read the owned and tracked counts for every new
  series, each totalling 512 stripe counters: about 630ns, more than the
  rest of creating a series put together, and 14% of a benchmark that
  only creates series. Each counter is now also held as a cross-stripe
  total, which the accessors read, at 0.9ns. Every mutation applies its
  delta to the stripe and to the total while holding that stripe's lock,
  so a total stays equal to the sum of the stripes rather than being
  recomputed from a scan. An earlier revision of this file cached the
  totals and had the recount store an absolute sum computed from its
  scan, which lost any increment landing after the scan had read a
  stripe; deltas cannot lose one, because a stripe's own mutations are
  serialised by its lock and deltas from different stripes commute.

With the feature off the result is now within noise of the merge base, and
updating a tracked series is slightly faster than it. With enforcement on,
creating a series costs what it does with the feature off.

Also:
- Build the ring test fixtures for subrings with tokens unique across
  instances: shared tokens made a subring test fail occasionally.
- Test that a push records the ring read number, and the cutoff of the
  backstop purge after head compaction, now a function of its own.
- Check the cross-stripe totals against the sum of the stripes after
  creations, deletions, native histograms, a backstop purge and a
  changing ring. This replaces a test which compared the per-stripe
  totals with themselves, and is what would catch a counter mutated
  without its delta.
- Benchmark the owned-series work on a large ring of
  1000 instances of 512 tokens across three zones: the ownership walk
  over the whole ring and over subrings, the per-second check, a state
  change, a topology change and the recount at 1 to 50 sharded tenants,
  the count the limit reads per new series, DeleteSeries, and updating an
  already tracked series serially and in parallel, which nothing covered
  although every sample takes that path.
- Correct comments left from replaced designs, say in the OwnedTokenPositions
  doc that ownership applies no health filter, and document the memory the
  ring copy takes and that ownership is checked every second.

Signed-off-by: Anant Shanbhag <anantvas@amazon.com>
@anant10
anant10 force-pushed the feat/owned-series-tracking branch from 3720c7b to f678073 Compare November 17, 2026 18:51

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Ingester: Add owned series tracking to exclude stale data from per-ingester limit calculations

3 participants