Repository navigation
Conversation
97d8157 to
305bc32
Compare
640b980 to
3720c7b
Compare
sandy2008
left a comment
There was a problem hiding this comment.
Thanks for the detailed write-up. I traced the ownership path and probed a few cases. Ownership matches routing for the default strategy but not for shuffle-sharded tenants. The instance-wide max_series check misses series created between update cycles. And an entry created during a ring change can keep the old ownership. Details inline.
| userDB.activeSeries.Purge(purgeTime) | ||
| if i.cfg.OwnedSeriesMetricsEnabled { | ||
| // Use UpdateMetrics which handles both purge AND ownership re-evaluation. | ||
| ringTokens, ownedPositions, ringFingerprint := i.lifecycler.GetOwnedTokenPositions() |
There was a problem hiding this comment.
With shuffle sharding this ownership doesn't match routing. GetOwnedTokenPositions is one bitmap for the whole ring, but for a tenant with ingestion_tenant_shard_size > 0 the distributor routes within ShuffleShard(userID, shardSize), whose replica sets differ from the full ring's. I checked with a quick probe in pkg/ring (30 instances, 3 zones, RF 3, 2000 random keys): 0 of 6000 placements from Ring.Get are reported as not owned, but 4756 of 6000 placements from a size-6 tenant subring's Get are. With -ingester.owned-series-limit-enforcement-enabled, such tenants would be checked against a fraction of their real series, while the limiter already scales their local limit to the shard size. Could ownership be computed against the tenant's subring when shuffle sharding applies, or could enforcement fall back to the head series count for those tenants?
| // off. Series without a tracker entry, such as those restored from the WAL, | ||
| // are already counted as owned by the cycle. A count of zero is a real count | ||
| // and is used as-is. | ||
| if owned := u.instanceOwnedCount.Load(); u.ownedSeriesLimitEnabled && owned >= 0 { |
There was a problem hiding this comment.
With enforcement on, this check uses instanceOwnedCount, which is only stored once per updateActiveSeries cycle (1m by default), while PostCreation only increments instanceSeriesCount. So series created since the last cycle aren't counted at all. The staleness is conservative after a ring change, but not for new series. I checked with a probe: instance max_series: 2, one series pushed, one cycle (snapshot 1), then three new series. All three were accepted and the head ended at 4, whereas with enforcement off the second and third were rejected with "max series limit reached". Since this limit is the OOM guard, could the check add the growth since the snapshot? For example, record the in-memory series count when the snapshot is taken and add max(0, current - recorded).
|
|
||
| c.stripes[stripeID].updateSeriesTimestamp(now, series, hash, nativeHistogram, labelsCopy) | ||
| // Load ring state atomically — readers on the push path always see a consistent snapshot. | ||
| state := c.ring.Load() |
There was a problem hiding this comment.
The ring state is loaded here, before findOrCreateEntryForSeries takes the stripe lock and computes owned. If a ring change lands in between, UpdateMetrics publishes the new ring and rescans the stripe before the entry exists. The entry is then created with the old ownership, and later cycles see tokensChanged == false and never re-evaluate it. In a stress probe (32 goroutines creating new series, one ring change from nothing owned to everything owned partway through), 224 of 300 trials ended with entries stuck as not owned, about two per trial, and they stay that way until the next ring change. Loading c.ring inside findOrCreateEntryForSeries, after s.mu.Lock(), should close this. updateTokens stores the new ring before the scan locks each stripe, so an insert that lands after its stripe's scan sees the new ring, and one that lands before it gets re-evaluated by the scan.
| @@ -1682,7 +1824,13 @@ func (i *Ingester) Push(ctx context.Context, req *cortexpb.WriteRequest) (*corte | |||
| isNHAppended := succeededHistogramsCount > oldSucceededHistogramsCount | |||
| shouldUpdateSeries := (succeededSamplesCount > oldSucceededSamplesCount) || isNHAppended | |||
| if i.cfg.ActiveSeriesMetricsEnabled && shouldUpdateSeries { | |||
There was a problem hiding this comment.
Owned tracking also depends on -ingester.active-series-metrics-enabled. With it off, nothing is recorded here, and updateLoop never runs updateActiveSeries. So with both owned flags on, cortex_ingester_owned_series is never emitted and both limits quietly fall back to the head counts. A probe push left head=1, Tracked()=0 and the instance snapshot at -1. Could Validate reject that combination, or could the flag help say it requires active series metrics?
| * [CHANGE] Alertmanager: Remove the obsolete startup migration of local state files into per-tenant directories (scheduled for removal in 1.11.0). Upgrading from a release older than 1.9.0 with a persisted local state directory now requires upgrading to an intermediate release first, so the migration can run. #7513 | ||
| * [CHANGE] Cache: Setting `-blocks-storage.bucket-store.metadata-cache.bucket-index-content-ttl` to 0 will disable the bucket-index cache. #7446 | ||
| * [CHANGE] HA Tracker: Move `-distributor.ha-tracker.failover-timeout` from a global config to a per-tenant runtime config. The flag name and default value (30s) remain the same. #7481 | ||
| * [FEATURE] Ingester: Add experimental owned series tracking to prevent false throttling during ingester scale-up and ring resharding. The ingester counts only the series the ring assigns to it, rather than everything left in the TSDB head, closing a window of up to one head compaction cycle in which a tenant could be throttled against a limit that had already shrunk. Enabled via `-ingester.owned-series-metrics-enabled` and `-ingester.owned-series-limit-enforcement-enabled`. #7509 |
There was a problem hiding this comment.
This entry is under ## 1.22.0 in progress. New entries go under ## master / unreleased and end with the PR number (#7758) rather than the issue number.
… during ring changes When ingesters scale up, the per-ingester local series limit drops immediately but stale series data remains in TSDB head for up to 2 hours. This causes PreCreation() to incorrectly reject new writes. This PR introduces owned series tracking in ActiveSeries: - Each series stores its ring token (computed via TokenForLabels) - Ownership is evaluated against current ring state on each update cycle - When ring changes, unowned series are excluded from limit enforcement Two feature flags for safe rollout: - owned_series_metrics_enabled: enables cortex_ingester_owned_series metric - owned_series_limit_enforcement_enabled: switches PreCreation to use owned count for both per-user and instance-level max_series limits Key design decisions: - Zone-local ownership check via SearchToken + instance token map lookup - Ring state stored behind atomic.Pointer[ringState] for lock-free reads on the hot push path - instanceOwnedCount recalculated every ~1 min (not incremental) to avoid drift from edge cases - Startup fallback: when instanceOwnedCount==0, uses instanceSeriesCount Code consolidation: - FNV hash functions consolidated into pkg/util/fnv.go (single source) - Sharding functions moved to pkg/ring/token.go (eliminates duplication between distributor and ring packages) Validated under sustained write load in a multi-zone deployment: after a scale-up, the owned count on the existing ingesters dropped in proportion to the redistribution while their head series count stayed high, and no writes were rejected. Fixes cortexproject#7509 Signed-off-by: Anant Shanbhag <anantvas@amazon.com>
Ring.Get determined the replica set for a key by inlining the ring walk directly in its body. The walk is the single definition of "which instances hold this data", and the owned-series work needs to ask that same question from a second place: for every token position, rather than for one key. Duplicating the walk there would let the two copies disagree, and a disagreement is exactly the bug that matters, because an ingester would then count series against its limit that the distributor does not route to it. Move the walk into ringTopology.replicaSetAt, keyed on a token position rather than a key, and have Ring.Get call it after resolving the key to a position with SearchToken. ringTopology is a read-only view that copies only slice and map headers, so it can be built per call. replicaSetAt also returns the instance IDs alongside the descriptors. InstanceDesc carries no ID of its own and an instance's ID is not interchangeable with its Addr, so a caller asking "am I in this set?" can only answer that correctly by comparing IDs. The per-zone cap is now skipped when the ring reports no zones at all, rather than dividing by the zone count unconditionally. Ring.Get cannot reach that state for a non-empty ring, so this is a robustness change for the new caller, not a behaviour change here. No behaviour change. The existing pkg/ring suite passes unmodified, and BenchmarkRing_Get is unchanged at ~607 ns/op and 0 allocs/op. Signed-off-by: Anant Shanbhag <anantvas@amazon.com>
Add OwnedTokenPositions, which returns the ring's sorted token list plus a parallel bitmap of the positions owned by a given instance. Ownership is computed with replicaSetAt, the same walk Ring.Get uses, so an instance's view of what it owns cannot drift from where the ring routes. This replaces the "am I the first instance in this token range, among the token list for my own zone?" check the owned-series tracking previously used. That check has exactly one owner per token range, which is only the same thing as the replica set in one configuration: zone awareness enabled with a replication factor equal to the zone count. In every other shape, including the default of zone awareness disabled, it undercounts by roughly the replication factor. Ownership deliberately does not apply the replication strategy's health filter. It has to stay stable across heartbeat flapping, because a series does not stop being this instance's responsibility when a peer misses a heartbeat. The lookup is O(tokens x replicationFactor) and is intended to be called once per ring change, so that the per-series question becomes owned[SearchToken(tokens, key)]: a binary search plus an array index. BenchmarkOwnedTokenPositions covers that precompute, at ~29.7 ms for 100 instances holding 512 tokens each. Tests assert the load-bearing property directly: across ten ring shapes covering zone awareness on and off and replication factors above, below, equal to and not divisible by the zone count, an instance owns a token position if and only if Ring.Get for a key in that position includes it. A conservation test asserts that every position is owned by exactly replicationFactor instances, which is what makes the per-ingester counts sum correctly against a limit that is itself scaled by the replication factor. One test builds a ring whose instance IDs and addresses deliberately differ. The shared fixtures set Addr equal to the instance ID, so an implementation comparing addresses would pass every other test here and then own nothing at all in a real ring. Signed-off-by: Anant Shanbhag <anantvas@amazon.com>
Replace the ownership check used by owned-series tracking. It previously asked "is the first token at or after my series' token, within the token list for my own zone, one of my tokens?". That has exactly one owner per token range, which only coincides with the ring's replica set when zone awareness is enabled and the replication factor equals the zone count. In every other configuration, including the default of zone awareness disabled, it is wrong. With zone awareness off, every instance falls into one unnamed zone, the "zone-filtered" token list becomes the whole ring, and the check degenerates to "am I the single global primary?". That counts roughly 1/replicationFactor of the series the instance actually holds, while the limit it is compared against is scaled up by the replication factor. The net effect is a limit that is too permissive by about the replication factor, so an ingester runs out of memory instead of throttling. That is a worse failure than the stale-count problem this feature exists to fix. Ownership now comes from ring.OwnedTokenPositions, which walks the ring with the same function Ring.Get uses, so the set of series an ingester counts is by construction the set the distributor routes to it. Because the walk honours the operation's replica-set extension rules, instances in JOINING, LEAVING and READONLY are handled exactly as the write path handles them, rather than via a separate READONLY special case. The lifecycler computes the bitmap in updateCounters, alongside the healthy-instance count that the limiter already uses as the denominator of the same comparison. Both therefore come from one ring snapshot taken at one instant under one lock, so numerator and denominator cannot be drawn from different views of the ring. The walk is too expensive to repeat on every heartbeat, and heartbeats are by far the most common reason updateCounters runs, so it is gated on a fingerprint covering instances, zones, states and tokens. Desc.RingCompare cannot serve here: it reports a state change and a timestamp change as the same result, and ownership depends on state. The recompute deliberately runs outside countersLock, since HealthyInstancesCount is on the push path and must not block behind it. Per-series cost drops from a map lookup to a binary search plus an array index, and isOwned no longer indexes the token slice without checking its length, which panicked on an empty ring. Tested with the upstream CI invocation, which matters here: go test -tags "netgo slicelabels" -race. Without the slicelabels tag pkg/ingester panics in cortexpb.CopyLabels, and without -race the //go:build !race file is compiled in and panics too; both reproduce on unmodified upstream. Signed-off-by: Anant Shanbhag <anantvas@amazon.com>
Owned-series tracking previously refused to track a series the ring did not assign to this ingester, and deleted entries when a ring change took ownership away. That made cortex_ingester_active_series silently drop those series, changing the meaning of a long-standing metric, and it made owned and active identical by construction, which left the staged rollout across the two feature flags unable to show any difference. Ownership now gates exactly one counter. Every series is tracked and counts towards active exactly as before; only owned is conditional. Losing ownership moves a series out of owned and leaves it active, which is accurate, because the series is still held in this ingester's head. Retention is now two-tier. The periodic cycle recounts but removes nothing, and entries are released by a purge after head compaction, keyed on the head's new minimum time. An entry therefore lives for as long as the series it describes is in memory. This is what makes the feature work for a tenant with high churn. Tying owned to the idle timeout meant a tenant with a 3M-series head but only 500K recently active series reported owned as 500K, so a 3M limit would admit 2.5M more series on top of the 3M already resident: more permissive than the unmodified code it replaces. Anchoring to the head instead makes owned a measure of what the ingester is actually storing, which is what a series limit exists to bound. The consequence is that owned may exceed active for such a tenant, which is intended and is the reason the two counts are maintained separately rather than one being derived from the other. Totals are cached per tenant and maintained as series are created, because PreCreation consults them for every new series and previously took a read lock on all 512 stripes to do so. The periodic cycle recomputes them authoritatively, so any drift is corrected within one cycle. Limit enforcement follows. The per-tenant check no longer needs its "owned > 0" guard: the cached total is accurate from the first sample rather than only after the first periodic cycle, so there is no startup window to paper over, and the guard had a cliff where a genuinely empty tenant silently fell back to the head count. The instance-level check keeps a fallback, because the instance total really is unavailable until the first cycle runs, but it is now keyed on an explicit negative sentinel rather than on zero, so a true zero is no longer confused with "not yet computed". Measured on 100k series: the periodic recount is 1.67ms with no allocations when the ring is unchanged and 2.42ms when every entry's ownership is re-evaluated. The push path is unchanged within noise, ~440ns/op with the ring loaded against ~400-457ns/op without it. Signed-off-by: Anant Shanbhag <anantvas@amazon.com>
The owned-series work replaced this file wholesale, removing 139 lines:
three tests covering the active count and purge behaviour, and four
benchmarks covering the push and purge paths. Removing the benchmarks meant
there was no longer a baseline to compare the new per-series ownership cost
against, which is what the review asked about.
All of them are restored, with the ring token argument added. They pass a
token of zero with no ring loaded, so ownership is unknown and every series
counts as owned, which means they exercise precisely the behaviour that
predates this feature.
Restoring them surfaced that BenchmarkActiveSeries_UpdateSeries does not
run at all. It calls b.Loop() in two separate loops, which is not allowed,
and sizes its series slice from b.N before b.Loop() has established an
iteration count. It panics with "index out of range [1] with length 1", or
fails with "B.Loop called with timer stopped" when an explicit -benchtime is
given. This reproduces on unmodified upstream at this branch's merge base,
and the same code is present verbatim on master, so it has been broken since
the b.Loop() migration rather than by this change. It is converted to the
b.N form, which is the correct shape when the iteration count is needed
before the timed loop begins.
New coverage for the owned-series semantics:
- active is unaffected by ownership. The same series are pushed into two
trackers, one with a ring where half the tokens belong elsewhere and one
with no ring, and the active counts must agree exactly. This is the
regression test for the metric's meaning being preserved.
- owned exceeds active for a tenant whose series have gone idle but are
still held in the head, which is the behaviour the limit fix depends on.
- the periodic cycle retains expired entries, repeated cycles are stable
rather than progressively dropping entries, and only a purge releases
them.
- losing and regaining ownership moves series between owned and not-owned
without touching active and without deleting anything, so ownership can
return without the series being re-pushed.
- the cached totals match the per-stripe counters after creation, after a
periodic cycle, after a purge and after a clear, since those totals are
what limits are enforced against.
- ownership change detection fires when the bitmap changes even though the
token list does not, which is the case a token-list hash cannot see.
- isOwned handles an empty ring and a mismatched bitmap without panicking,
which the previous ownership check did not.
Two benchmarks are added for the new paths: the push path with and without
a ring loaded, to show the per-series ownership cost, and the periodic
recount separating the common unchanged-ring case from a full ownership
re-evaluation.
Signed-off-by: Anant Shanbhag <anantvas@amazon.com>
Ownership is computed from the raw replica-set walk and deliberately does not apply the replication strategy's health filter. That makes ownership differ from routing while an instance is transitioning, and until now that difference was only described in a comment rather than pinned by a test. For each of JOINING, LEAVING and READONLY, assert that the transitioning instance still owns its token ranges, that those ranges end up with more owners than the replication factor because the replica set is extended, and that writes still land on exactly replicationFactor healthy instances and never on the transitioning one. Both the transitioning instance and the instance the walk extends to really are holding that data, so both counting it is correct rather than double counting: the departing instance still has the series in its head, and the distributor really is writing to the extension. Filtering for health here would instead make an instance's own series count depend on whether its peers happened to miss a heartbeat, which is why the asymmetry with Ring.Get is intentional. Note that the aggregate invariant asserted by TestOwnedTokenPositions_Conservation, that every position has exactly replicationFactor owners, holds only for a ring where every instance is ACTIVE. This test covers the complementary case. Signed-off-by: Anant Shanbhag <anantvas@amazon.com>
The CHANGELOG carried three entries for this change. The two ENHANCEMENT lines described internal refactoring, moving the sharding helpers into pkg/ring/token.go and the FNV helpers into pkg/util/fnv.go, which is not something an operator reading release notes needs. Collapsed to the single FEATURE entry, reworded to describe the effect rather than the mechanism, marked experimental to match how the flags are listed in v1-guarantees, and with the "a up-to-2-hour" typo removed. It now says "up to one head compaction cycle", which is what the window actually is rather than assuming the default block range. The two new flags were never added to the generated configuration documentation, so `make check-doc` fails: it regenerates the docs and asserts `git diff --exit-code` over the config reference, the blocks-storage pages and the JSON config schema. Regenerated, which adds the twelve expected lines to docs/configuration/config-file-reference.md and the matching entries to schemas/cortex-config-schema.json. No other generated file changes. Also listed the feature under experimental features in docs/configuration/v1-guarantees.md, alongside the other opt-in ingester metrics, so the stability expectation is explicit while it is behind flags. Signed-off-by: Anant Shanbhag <anantvas@amazon.com>
Head-anchored retention compares two different clocks. An entry's timestamp is the wall-clock time a sample for it arrived, while Head().MinTime() is a sample timestamp. They agree for real-time ingestion, which is why this works in the common case, but they diverge for a tenant whose samples are backdated or whose clock is skewed. Only one of the two directions is dangerous. A cutoff that lands too early retains entries for longer than necessary, which over-counts owned series and errs towards enforcing limits sooner: wasteful, not harmful. A cutoff that lands too late drops entries for series which are still resident, which under-counts owned series and lets a tenant exceed its limit. A tenant whose sample timestamps run ahead of real time would hit exactly that, and in the extreme every entry would be released on each compaction, since no arrival time can be in the future. Clamp the cutoff so it is never more recent than one block range ago. Anything still in the head must have arrived within roughly that window, so this cannot release an entry for a resident series however far ahead a tenant's timestamps run. Backdated timestamps are left alone, since they already err in the safe direction. This is a bound rather than a translation between the clocks. Making the comparison exact would mean recording each series' last sample timestamp on its entry, which costs memory per series on a structure whose retention cost is already the open question here, so it is deliberately left for the design discussion on the pull request rather than decided unilaterally. The cutoff is computed by a free function so the behaviour can be asserted without standing up a TSDB. Tests cover each direction of skew and state the property the clamp exists for: a series which arrived within the last block range is never released, whatever the head reports. Signed-off-by: Anant Shanbhag <anantvas@amazon.com>
Retaining entries past the idle window introduced one way for the active count to behave differently from before, which I had not handled. Previously a series going idle had its entry removed, so a later sample created a fresh entry and the active count reflected the series at once. Now the entry survives and merely stops being counted, so a later sample updates it in place and creates nothing. The cached total is only bumped on creation, so a returning series went unnoticed until the next periodic recount, under-reporting active for up to one update period, a minute by default. The exported cortex_ingester_active_series gauge was not affected, because it is set inside the periodic loop immediately after the recount. The two other readers were: the native histogram limit consulted by PreCreation on every new series, and the active series figure in the per-tenant stats endpoint. Both would have read low, making that limit marginally too permissive. UpdateMetrics now publishes the idle cutoff it used, and the push path uses it to notice an entry crossing back into the window. The compare-and-swap that moves the timestamp across the cutoff decides which caller counts the series, so concurrent samples for the same returning series count it once. Whether the series is a native histogram is taken from the entry rather than from the incoming sample, since an entry's kind is fixed when it is created. Only the active counts change on reactivation. An idle series never stopped being owned, because owned deliberately ignores the window. The cutoff is published as zero until a cycle has run with one, so when entries are released by the idle timeout rather than retained, the reactivation path never engages and the original behaviour of creating a fresh entry is preserved exactly. Tests assert active reflects the returning series before any recount, that a second sample inside the window does not count it twice, that the subsequent authoritative recount agrees with what the push path did, and that the cached totals still match the per-stripe counters. Verified to fail without the change. A concurrency test pushes the same returning series from 64 goroutines and asserts it is counted once. The push path is unchanged within noise at ~470ns/op. Signed-off-by: Anant Shanbhag <anantvas@amazon.com>
Three behaviours were only visible by reading the implementation. Enabling owned series tracking is documented as turning on a metric, but it also changes how long active series entries are retained: they are released at head compaction rather than after the idle timeout, so that the owned count reflects what is held in memory rather than only what is recently active. That increases ingester memory roughly in proportion to the ratio between series in the TSDB head and recently active series. An operator enabling what reads like an observability flag should not have to discover that from a heap profile. The owned count may exceed the active count for a tenant with high churn, because it includes idle series that are still held. This is intended, and cortex_ingester_active_series keeps its existing value, but anything asserting that owned is a subset of active will be surprised. Limit enforcement applies to the instance-wide max_series limit as well as to per-tenant limits, which a single flag name does not convey. The consequence worth stating is that max_series then stops counting series reassigned to another ingester which are still resident until the next head compaction, so it no longer reflects everything the ingester is holding. No behaviour change. Flag help text, the regenerated configuration reference and schema, and the experimental features list. Signed-off-by: Anant Shanbhag <anantvas@amazon.com>
cortex_ingester_owned_series carries a user label but was absent from both places that clean up per-tenant metrics: deletePerUserMetrics, and the TSDB close path. Its three sibling per-tenant gauges are removed in both. A stale gauge therefore remained for every tenant whose TSDB was ever closed, by idle cleanup or deletion. On an ingester that sees tenants come and go the label cardinality of its own /metrics grows without bound, and dashboards keep reporting owned series for tenants that no longer exist. Signed-off-by: Anant Shanbhag <anantvas@amazon.com>
numZones was derived as len(getTokensByZone()), which merge-sorts every token in the ring into one slice per zone just to take the length of the resulting map. Counting distinct Zone values over the instances is proportional to the number of instances rather than the number of tokens, and allocates nothing. For 100 instances holding 512 tokens each this takes the once-per-ring-change precompute from 29.7ms to 26.4ms and from 715 allocations to 489. It also removes one of the two calls that sort each instance's token slice in place on the shared Desc as a side effect. Signed-off-by: Anant Shanbhag <anantvas@amazon.com>
…aching The previous revision cached one cross-stripe total per counter and had the push path increment it, while the periodic recount summed the stripes and stored the result. That store is not atomic with respect to the scan it was computed from, so a series created between a stripe being scanned and the total being stored was lost, and one created just before its stripe was scanned could be counted twice. The next complete recount repaired it, so the drift was transient, but the counts were only eventually correct rather than correct. It also diverged from the behaviour it replaced even with both feature flags off. Previously Active and ActiveNativeHistogram summed the stripes under a read lock and were exact at all times; with the cache they could drift during each purge pass, which is a change nobody opting out of this feature asked for. Make the per-stripe counters atomics and total them in the accessors. Every mutation still happens while holding that stripe's lock, so nothing can be lost, and readers no longer need to acquire 512 locks: totalling 512 atomics costs about 290ns with no allocations. That is cheaper than the read-locked version it replaces as well as being exact, so the cache was solving a problem that did not need solving at the cost of one that did. The cross-stripe cache, and the plumbing that reported back from updateSeriesTimestamp purely to maintain it, are gone. Reactivation still has to be counted explicitly, because an entry crossing back into the active window is not a creation, but it now increments the stripe counter under the stripe lock like every other mutation. A test asserts the counts stay exact while a recount runs concurrently with sixteen writers. It does not reproduce the transient drift of the cached design, since that repairs itself before any assertion can observe it; it pins the steady-state invariant. Signed-off-by: Anant Shanbhag <anantvas@amazon.com>
purge was given one cutoff and used it both to decide what to delete and to
decide what counts as active. That is correct for its original caller, which
passes the idle timeout, but owned-series tracking added a second caller that
passes a cutoff derived from the head's minimum time, which can be hours old.
Every retained entry then counted as active, so after each head compaction the
active count jumped to the owned count until the next periodic recount.
Two readers see that window, and neither waits for the recount:
- PreCreation consults ActiveNativeHistogram on every new series, so a tenant
with max_native_histogram_series_per_user set would get spurious
limit-exceeded errors for up to a minute after each head compaction.
- The per-tenant stats endpoint reports the inflated active count, which also
contradicts the documentation added in this revision stating that the active
series value is unchanged.
Take the two cutoffs separately: deleteBefore decides what is still held,
activeCutoff decides what is inside the idle window. The original caller passes
the idle timeout for both, which is exactly its previous behaviour, so nothing
changes with the feature disabled.
The regression test fails if the cutoffs are conflated again.
Signed-off-by: Anant Shanbhag <anantvas@amazon.com>
Active series entries are only created when a sample arrives: UpdateSeries is
called from the push path and nowhere else, and nothing seeds the tracker from a
restored head. An ingester that has just replayed its WAL therefore has a full
head and an empty tracker, so Owned() reads zero.
The previous revision removed the "owned > 0" guard that had been covering this,
on the reasoning that the count is maintained as series are created and so is
accurate from the first sample. That is true for series created by a push and
false for every series restored from the WAL. With the guard gone, a tenant could
create a full limit's worth of series on top of everything already resident,
roughly doubling the intended memory, until enough of the restored series
happened to receive a sample. Unmodified Cortex is not exposed to this, because
it compares the limit against Head().NumSeries(), which is correct the moment
replay finishes.
The instance-wide max_series limit had the same hole, and the negative sentinel
added in this revision did not close it: one cycle after a restart the periodic
loop stored a genuine zero, so the OOM guard passed everything.
Track the number of entries held, and only use the owned count when it covers
what is in memory:
- per tenant, use Owned() only when Tracked() is at least Head().NumSeries()
- instance wide, publish the sentinel rather than a total whenever the trackers
between them hold fewer entries than there are series in memory
Falling back to the in-memory series count over-counts, which delays a scale-up
from taking effect but cannot let a tenant past its limit. Guarding on coverage
rather than on the count being zero also removes the cliff the old guard had,
where a genuinely empty tenant was indistinguishable from an unknown one.
Tracked is a useful number in its own right: it is the size of the structure
whose retention this feature extends, which was otherwise not observable.
Signed-off-by: Anant Shanbhag <anantvas@amazon.com>
Head-anchored retention decided which active series entries to release at head compaction by comparing each entry's timestamp against a cutoff derived from the head's minimum time. Those are two different clocks: an entry's timestamp is the wall-clock time a sample arrived, while the head's minimum time is a sample timestamp. An earlier change clamped the cutoff to at most one block range before now, to avoid releasing entries for resident series when sample timestamps run ahead of real time. In practice the clamp binds on every compaction and over-counts substantially. Compaction runs about an hour after a block boundary, so the head is truncated to that boundary while the cutoff is pinned an hour earlier. Every series that went idle in between leaves the head but keeps its entry. For a tenant with high churn, owned then exceeds the head substantially after each compaction and stays there until the next one. That makes the limit stricter than with the feature off, which is the same class of false throttling the feature exists to remove. Narrowing the clamp to the creation grace period would shrink the error but not remove it, and would depend on a per-tenant setting and on distributor and ingester clocks agreeing. Instead, stop estimating. The head already reports exactly which series it deleted, through the PostDeletion lifecycle callback Cortex uses to keep its in-memory series count. Release precisely those entries there. Each entry now records the head series reference it describes, refreshed on every sample. On deletion, an entry is removed only if it still carries the deleted reference. The head invokes the callback after releasing its own lock, so a sample can re-create the same labels as a new head series before the callback runs; that sample moves the entry to the new reference, and the stale deletion leaves it alone. References are never reused within a TSDB. The push path now keeps the reference returned by every successful append, rather than discarding it when a cached reference was used, so a series re-created between GetRef and Append is still recorded correctly. The time-based purge of retained entries and its cutoff function are removed. A backstop purge remains at compaction for entries idle longer than two block ranges, longer than any series can stay in the head without a sample, so it cannot under-count and only guards against an entry outliving its series by a path the callback does not cover. With owned-series tracking disabled nothing changes: references are not recorded or allocated, the callback does not touch the tracker, and entries are still released by the idle-timeout purge. TestIngester_OwnedSeriesFollowsHeadThroughCompaction drives a real head compaction with explicit sample timestamps, including a series idle for five minutes before the truncation point, and asserts the tracker holds exactly the series left in the head. It fails on the previous code with 3 entries for 1 head series, reproducing the over-count, and passes with this change. Unit tests cover release on a matching reference, retention across re-creation, unknown references, and adopting a reference late. Signed-off-by: Anant Shanbhag <anantvas@amazon.com>
…ding owned The limit check used the owned count only when the tracker held an entry for every series in the head, and otherwise fell back to the head count. That guard was meant for the window after a restart, but every new series enters the head a moment before its tracker entry is written. Under concurrent pushes the guard therefore failed during any burst of series creation, and an ingester that had just lost ownership of most of its head was throttled against the stale head count: the false throttling the owned count exists to prevent. Count series in the head without a tracker entry as owned instead. After a restart this equals the head count, as before; in normal running it adds only the series in flight. Apply the same to the instance-level count. Without this, a large share of new series on the existing ingesters can be rejected after a scale-up. The new concurrent test rejects 300 of 640 series without this change and none with it. Signed-off-by: Anant Shanbhag <anantvas@amazon.com>
Reports bytes per entry with owned-series tracking disabled and enabled, so the memory cost of retaining entries until the head deletes their series can be estimated as this per-entry cost times the number of idle series still in the head. Signed-off-by: Anant Shanbhag <anantvas@amazon.com>
- Use go.uber.org/atomic instead of sync/atomic in active_series.go and owned_series_limit_test.go. faillint blocks sync/atomic, which is what failed the lint job on the previous revision. - Keep Lifecycler.getTokens unexported, as on master. Nothing outside pkg/ring needs it, and master's lifecycler tests call getTokens. - Move the CHANGELOG entry under "master / unreleased", reference cortexproject#7758, and drop the leftover per-change entries so the feature has one entry. Signed-off-by: Anant Shanbhag <anantvas@amazon.com>
Instance max_series missed series created between cycles. With enforcement on, the check read an owned count taken once per updateActiveSeries cycle, so series created after it were not counted at all. The snapshot now records how many series had been created when it was taken, and the check adds every series created since. It counts creations rather than the change in the in-memory count, because head compaction can delete as many series as were created meanwhile, which would leave the new ones uncounted. The result is capped at the in-memory count, so it is never stricter than counting everything in memory. A new series could keep stale ownership. The ring state was read before taking the stripe lock, so a series created during a ring change could be given the old ownership after its stripe had been recounted, and keep it until the next ring change. The state is now read under the stripe lock, which the recount also takes. Owned-series tracking depended on -ingester.active-series-metrics-enabled. With that flag off, series were not recorded on push, the periodic recount did not run and cortex_ingester_owned_series was not registered, so both owned limits quietly fell back to the head count. The tracker now runs whenever either flag is set, and the active series flag only decides whether the active series metrics are exported. Enabling -ingester.owned-series-limit-enforcement-enabled without -ingester.owned-series-metrics-enabled still has no effect, but is now logged once at startup, and the flag help says so. Signed-off-by: Anant Shanbhag <anantvas@amazon.com>
Add Ring.OwnedTokenPositions, which returns the ring's sorted tokens and a bitmap of the positions an instance owns, from the same replicaSetAt walk Ring.Get routes with. Being a method on Ring, it answers for the subring ShuffleShard returns as well as for the whole ring. A subring picks different replica sets from the whole ring, so ownership computed over the whole ring miscounts the series of a shuffle-sharded tenant. It reuses the ring's token list and instance lookup, so the bitmap is all it allocates. Add Ring.OwnershipFingerprint, which changes whenever ownership may have. Tokens, zones and the set of instances only change together with the ring topology, which updateRingState already timestamps, so the fingerprint hashes that timestamp and each instance's state rather than every token. On a subring it covers only the subring's members, so a change elsewhere in the ring leaves it unchanged. Add NewRingFromDesc and Ring.UpdateDesc, for a Ring that is updated with a Desc another component has already read instead of watching the KV store. A process can then use its own view of a ring it already reads, including shuffle-shard subrings, without a second watch. With a KV store that is polled, that watch re-reads the whole ring on every poll. An update that only changes instance states stays cheap and keeps the cached subrings, as it does on a watched ring. The Desc-based OwnedTokenPositions is kept, deprecated, until the lifecycler stops using it in the next commit. Signed-off-by: Anant Shanbhag <anantvas@amazon.com>
Owned series were computed over the whole ring for every tenant. For a tenant with a shuffle-shard size smaller than the ring, the distributor routes within the tenant's subring instead, which picks different replica sets. Whole-ring ownership then counts the wrong series: an ingester in a tenant's shard counts as owned only the fraction the whole ring would also send it, so with limit enforcement the tenant is checked against a fraction of the series it holds. Each tenant's ownership is now computed over the ring the distributor routes it within: the tenant's subring when shuffle sharding applies and the shard is smaller than the ring, and the whole ring otherwise. Whole-ring tenants share one ownership bitmap, as before. Each sharded tenant on this ingester gets its own, built from its subring, and an ingester outside a tenant's shard owns none of its series. Until this ingester has joined the ring, ownership is unknown and every series counts as owned. The rings are built from the ring the lifecycler already reads on every heartbeat, not by watching the KV store a second time from every ingester. The lifecycler keeps a copy of it once asked to, replaced only when the instances, their tokens, zones or states change, so heartbeats alone copy nothing. This is also the view of the ring the limiter computes local limits from, so a ring change reaches the local limit and the owned count together. Ownership is recomputed when the ring changes and when a tenant's shard size changes. The shard size matters on its own: a larger shard moves some of the tenant's series to the new members while the old ones still hold them, and the shard size can change at any time through the runtime overrides. Changes are applied without waiting for the next active series update. Every -ingester.owned-series-recompute-min-interval (default 15s) the ingester checks whether the ring or any tenant's shard size changed. The check costs about a millisecond with 50 sharded tenants in a ring of 1000 instances. Work is kept to what actually changed, because during a rolling restart instance states change every few seconds: - A sharded tenant is recomputed only when an instance in its own shard changed, not on every change elsewhere in the ring. - When a recomputed bitmap equals the previous one, the tenant keeps its ownership, and its series are not re-evaluated. Ownership moves out of the lifecycler, which computed it on every heartbeat that changed the ring even though it was used once a minute. The Desc-based OwnedTokenPositions is removed. Signed-off-by: Anant Shanbhag <anantvas@amazon.com>
A new series now counts as owned as soon as it is created. It reached this ingester because the distributor routed it here, which is the question ownership answers. Judging it by this ingester's own view of the ring could count it as owned by nobody: that view is only refreshed when the lifecycler heartbeats, while the distributors see ring changes sooner, so after this ingester gains token ranges, on a scale-down or while a peer restarts, new series arrive in ranges its stale view does not give it, and they escaped both series limits until the next heartbeat. Existing series are still re-evaluated against the ring whenever it changes. A new entry also no longer reads the ring state at all, so no ordering against a concurrent recount can leave it with stale ownership. For the same reason an ingester outside a tenant's shard owns none of the series it holds for the tenant: if it and the distributors disagree about the shard, new series that arrive still count, as they are created. A recomputed bitmap is compared with the previous one by value. A topology change rebuilds every subring with a new token list, even for a shard whose members did not change, and comparing the lists by identity made every sharded tenant re-evaluate every series it held on each topology change. When the bitmap is unchanged the tracker keeps its ownership and adopts the new slices, so the previous ring's can be freed. Ownership is computed with the operation the distributor writes with, which follows -distributor.extend-writes, passed to the ingester the way -distributor.shard-by-all-labels is. A series' ring token is computed only when its tracker entry is created, and outside the stripe lock, instead of for every series on every push. If it cannot be computed, as for a series with no metric name when sharding by metric name, the series is tracked as owned instead of failing the push. The per-tenant owned count is capped at the head. The head deletes a series before telling the tracker, so for a moment the tracker can hold more entries than the head holds series. The frequent recount compares each tenant's ownership with the one its tracker actually has, rather than with the one last applied to the tenant: a TSDB closed while idle and opened again has a new tracker with no ownership, which the recount used to skip until the next active series update. A tenant whose ownership cannot be computed is left alone and retried on the next check. When a tenant's shard size changes, the subring cached for the previous size is dropped. The ring built from the lifecycler's copy no longer computes detailed ring metrics, which nothing exports. -ingester.active-series-metrics-update-period must be positive when owned or active series are tracked. The flag help and v1-guarantees now say that new series count as owned on arrival, that ring changes are seen on this ingester's heartbeat, and what the ring copy costs in memory. Stale comments are corrected, and the unused clear() and HashAddUint and HashAddUint32 are removed. Test series names are scrambled, so that they spread over the token space rather than hashing into clusters, which made the scale-up assertions depend on luck. Signed-off-by: Anant Shanbhag <anantvas@amazon.com>
This ingester's view of the ring is the one its lifecycler reads on each heartbeat, which is also the view its local limits are computed from. Two problems remained with using it for owned series. The owned count lagged the limits. After a scale-up the local limit dropped as soon as the lifecycler read the new ring, while the owned count waited for the next recount, up to -ingester.owned-series-recompute-min-interval later, and the tenant could be throttled in between. The ingester now checks every second whether the lifecycler has read a changed ring, a pointer comparison, and recounts straight away, so the two change together. A stale view could un-count series routed here. Distributors refresh the ring more often than the lifecycler, so after this ingester gains token ranges new series arrive before its view shows them as its own. They counted as owned on arrival, but the next recount judged them by the stale view: the first ownership applied to a new or reopened tenant, or a shard size change. A tenant could then fill its limit again. The lifecycler now numbers its ring reads, each number taken before the read starts, and a series records the number current when it arrived. A recount only judges a series by a view read after it arrived; until then it stays owned, and the next read judges it, even if the ring did not change. A series older than the view is judged as before, so series which moved away still stop counting. A subring's ownership fingerprint now hashes its instances, zones and tokens instead of the ring's topology timestamp. An instance that restarts with a new address is a topology change to the ring, but changes no replica set, so it no longer recomputes the ownership of every sharded tenant. Also: - Rename -ingester.owned-series-recompute-min-interval to -ingester.owned-series-recompute-interval: it is a period, not a minimum. It now only governs shard size changes. - Require a positive lifecycler heartbeat period, the source of the ring view, when owned series are tracked. - Ring.UpdateDesc does nothing on a ring watching the KV store, where it would race with the watch. - Treat a ring with instances but no tokens yet as unknown ownership. - Remove the tracker's atomic ring state, which nothing reads concurrently any more, and the test of the race it guarded, which can no longer fail. - Keep HashNew32, HashAdd32 and HashAddByte32 in pkg/ingester/client as deprecated wrappers, so that code importing them from there still builds. - Document that ingesters need the distributors' sharding strategy, shard-by-all-labels and extend-writes settings, that a new series counts as owned on arrival, and the memory the ring copy takes. Signed-off-by: Anant Shanbhag <anantvas@amazon.com>
…ries Series restored from the WAL after a restart had no tracker entries, so they all counted as owned until head compaction dropped them. A rolling restart soon after a scale-up therefore brought back the throttling owned series is meant to prevent, on every restarted ingester, for up to a head compaction cycle. After the WAL replay and before the ingester joins the ring, every series in each tenant's head is now given a tracker entry: owned, not active, since it has had no sample since the restart, and with its kind set by its first sample. They predate every ring read of the process, so the first view of the ring judges them, even after a crash, when the instance is still ACTIVE in the ring and its next read is the first heartbeat. All owned-series work now runs on its own goroutine instead of the ingester's update loop. Every second it recounts the tenants whose ownership changed, because the lifecycler read a changed ring or a tenant's shuffle-shard size changed, which is cheap when nothing did; every active series update period it recounts every tracker. On the update loop, a ring walk of a few hundred milliseconds in a large ring delayed the instance ingestion rate tick, which assumes a steady interval and so over-reported the rate, and the recount could wait behind the once-a-minute update. A shard size change was only checked every -ingester.owned-series-recompute-interval, although it changes the tenant's local limit at once, so the tenant could be throttled until then. That flag is removed: the check runs every second. Ownership now records the lifecycler read it was computed from. A recount for which ownership could not be computed no longer judges waiting series by the tracker's previous bitmap, nor records the newer read as applied. The lifecycler keeps only the newest of overlapping ring reads, as its first heartbeat can overlap another read, and its copy of the ring is no longer replaced when only an instance's address changes. Active series counts match those without owned-series tracking in two more cases: - A retained entry which becomes active again takes the kind of the sample that reactivated it. Without retention it would have been released when idle and re-created with that sample's kind, so a series which switched between float and native histogram samples was counted as its old kind, and could trip the native histogram series limit, until head compaction. - The backstop purge after head compaction counts active series against the cutoff the last recount published, which the push and deletion paths also use. Counting against a newer one put the counters out of step with them until the next recount: a reactivation in between was missed, and a deletion in between could take the active count below zero. Benchmarking against the merge base found three costs, two of them paid with the feature off: - Updating a series the tracker already holds, which runs for every sample, was about 17% slower than without this change, and more than that under contention. Scanning a stripe copied each entry, the ring token was passed by value through calls that do not inline, and the active cutoff was loaded for every sample. The scan now takes the address of each entry, the token and the cutoff are passed by reference, and the cutoff is read only once the timestamp has moved, since a sample that merely advances a timestamp needs neither. - Every ring computed a topology fingerprint on every rebuild, including the rings in distributors, queriers and store-gateways, which never ask about ownership: building an uncached shuffle shard was 4-10% slower and applying a topology change 2-5% slower, fleet-wide. It is now computed by the first call to OwnershipFingerprint after a topology change, so a ring nobody asks does not compute it at all. - The per-tenant limit read the owned and tracked counts for every new series, each totalling 512 stripe counters: about 630ns, more than the rest of creating a series put together, and 14% of a benchmark that only creates series. Each counter is now also held as a cross-stripe total, which the accessors read, at 0.9ns. Every mutation applies its delta to the stripe and to the total while holding that stripe's lock, so a total stays equal to the sum of the stripes rather than being recomputed from a scan. An earlier revision of this file cached the totals and had the recount store an absolute sum computed from its scan, which lost any increment landing after the scan had read a stripe; deltas cannot lose one, because a stripe's own mutations are serialised by its lock and deltas from different stripes commute. With the feature off the result is now within noise of the merge base, and updating a tracked series is slightly faster than it. With enforcement on, creating a series costs what it does with the feature off. Also: - Build the ring test fixtures for subrings with tokens unique across instances: shared tokens made a subring test fail occasionally. - Test that a push records the ring read number, and the cutoff of the backstop purge after head compaction, now a function of its own. - Check the cross-stripe totals against the sum of the stripes after creations, deletions, native histograms, a backstop purge and a changing ring. This replaces a test which compared the per-stripe totals with themselves, and is what would catch a counter mutated without its delta. - Benchmark the owned-series work on a large ring of 1000 instances of 512 tokens across three zones: the ownership walk over the whole ring and over subrings, the per-second check, a state change, a topology change and the recount at 1 to 50 sharded tenants, the count the limit reads per new series, DeleteSeries, and updating an already tracked series serially and in parallel, which nothing covered although every sample takes that path. - Correct comments left from replaced designs, say in the OwnedTokenPositions doc that ownership applies no health filter, and document the memory the ring copy takes and that ownership is checked every second. Signed-off-by: Anant Shanbhag <anantvas@amazon.com>
3720c7b to
f678073
Compare
What this PR does
Adds owned series tracking to the ingester, so per-tenant and per-instance series limits stop throttling tenants for series that have already moved to other ingesters.
The problem. A tenant's local series limit is
global limit × replication factor ÷ ingesters(the tenant's shard size when shuffle sharding). When ingesters are added, or a tenant's shard grows, that limit drops at once. The series that now belong to other ingesters stay in the old ingesters' TSDB heads until head compaction, up to about two hours.PreCreationchecksHead().NumSeries(), so an old ingester compares everything it still holds against the new, lower limit and rejects new series the tenant is entitled to send.The fix. An ingester counts as owned only the series the ring currently routes to it, and with enforcement enabled the limits use that count.
How it works
Ring.Getuses (replicaSetAt, extracted fromRing.Get), so it can't drift from routing. It's precomputed per ring change as sorted tokens plus one bool per position: on push, ownership is a binary search and a bool read.ShuffleShardcall the distributor uses, from the ingester's own copy of the ring (built from the ring the lifecycler reads). A tenant's subring is only re-evaluated when its own members, states or tokens change.owned + series in the head without a tracker entry, capped at the head, so in-flight and just-restored series still count. The instancemax_seriescheck uses a snapshot taken by each recount, plus every series created since, capped at the series in memory.Flags (experimental, both default off)
With both off the behaviour is identical to master. Owned tracking is independent of
-ingester.active-series-metrics-enabled.Known properties
Testing
Unit and integration tests cover routing equivalence (whole ring and subrings), ring and shard-size changes, WAL restarts, concurrent creates during ring changes, the instance limit snapshot, and flag validation.
We also ran it in a multi-zone test cluster under steady write load, with a whole-ring tenant and a shuffle-sharded tenant:
per_user_series_limiton the old ingesters.Benchmarks
AMD EPYC 9R14. Master is3d636b064, the merge base when the benchmarks ran; the branch was later rebased onto two unrelated upstream commits with no change to this PR's diff.ActiveSeries_UpdateExistingSeriesis added by this PR, since nothing upstream covers the per-sample update of a known series; on master it ran as an equivalent temporary copy. Comparison benchmarks were interleaved master/PR in two rounds of 5 (count 10 per side); PR-only benchmarks count 6. Large-ring cases use 1,000 instances and shard sizes 10, 100, 250 and 500.Existing benchmarks (master vs PR): the push path is faster (
IngesterPush−7.8%,UpdateExistingSeries−6% serial / −16% parallel). The once-a-minute active series purge is slower (Purge_once+5.9%,Purge_twice+4.3%), from the extra per-entry state. Ring benchmarks are within ~1% overall (geomean +0.84%); the largest changes areUpdateRingStatewith token updates (up to +3.5%) andShuffleShardCached(up to +2.7%).Ingester: master vs PR (benchstat)
Ingester: owned-series benchmarks (PR only)
Ring: owned-series benchmarks (PR only)
Ring: master vs PR (benchstat)
Which issue(s) this PR fixes: Fixes #7509
Checklist
CHANGELOG.mdupdated