Skip to content

perf(ingester): cut postings cache memory with a single-object LRU entry - #7944

Open
alanprot wants to merge 5 commits into
cortexproject:masterfrom
alanprot:compact-postings-cache-lru
Open

alanprot wants to merge 5 commits into
cortexproject:masterfrom
alanprot:compact-postings-cache-lru

Conversation

@alanprot

@alanprot alanprot commented Nov 18, 2026 •

Copy link
Copy Markdown
Member

What this PR does:

Reduces the memory the ingester's expanded postings cache uses per entry, and fixes a byte-accounting drift in its LRU.

Each cached entry used to be four heap objects plus a sync.Map node: the cacheEntryPromise, its done channel, a list.Element, and the key boxed into that element. For small entries the real heap cost was about 3x the bytes the cache accounts for, so max_bytes understated memory badly.

Changes:

  • One object per entry. cacheEntryPromise now carries the key and intrusive prev/next pointers, so there is no list.Element and no boxed key.
  • Plain map under the existing mutex. Every hit already took cachedMtx.Lock() to move the entry to the back of the list, so sync.Map bought no concurrency. Map and list now change together under one lock, and an entry is in the map if and only if it is linked in the list.
  • Channel only while fetching. The done channel and the error move into a small inflight value. It is dropped when the fetch succeeds, so completed entries hold no channel. All concurrent callers for a key wait on the same inflight.
  • Waiters honor their context. A caller that finds a fetch already running no longer blocks inside getPromiseForKey. It waits in result() and returns ctx.Err() if its own context is cancelled, while the fetch continues for other callers.
  • Failed fetches are not cached. Previously an error was stored in the entry and returned to every caller until the TTL expired.
  • A panicking fetch does not leave the key stuck. The fetch runs under a deferred cleanup that removes the entry, wakes waiters with errFetchPanicked, and lets the panic continue, so the next caller fetches again.
  • Timestamps are stored as int64 nanoseconds instead of time.Time.

Per-entry memory, excluding the key bytes and the series IDs (both unchanged). Heap bytes per entry measured with 200k entries on amd64, Go 1.27:

Object kept per cached entry Before After
cacheEntryPromise (88 B struct before, 80 B after) 96 B 80 B
done channel (hchan) 112 B 0 B (only while the fetch runs)
list.Element plus the key boxed into any 64 B 0 B (intrusive prev/next)
Index: sync.Map node before, plain map slot after 125 B 35 B
Total overhead 397 B, 4 objects 115 B, 1 object

With a ~190-byte key and one series ID (192 B + 8 B allocated) this adds up to the ~600 B and ~320 B per entry in the benchmark below. The inflight value (24 B plus its channel) exists only while a fetch is running and is then garbage.

Accounting drift fixed. In the expired-refresh path, a goroutine swapped a new promise in for an expired one and started fetching. If expire() then popped the old promise's list element, LoadAndDelete(key) removed the new promise, whose size was still 0, so the old promise's bytes stayed in cachedBytes permanently. Orphan and duplicate list elements followed. Over time the cache counted bytes for entries it no longer held, which shrinks its effective capacity. -race also reported unsynchronized writes to ts and sizeBytes. Eviction now removes exactly the entry it unlinks, and a completed fetch only adds its bytes if its entry is still the one in the map.

Behaviour kept as before: TTL measured from creation time, full and expired eviction reasons and metrics, and Ttl <= 0 disabling retention.

Behaviour change: eviction now runs after every completed fetch, including the refresh of an expired key. Previously a refresh skipped eviction, so a larger refreshed value could push cachedBytes over max_bytes until the next fresh miss or the periodic purge, and expired neighbours stayed until then. TestFifoCacheExpire/TTL asserted the old behaviour and is updated: the first refresh now also sweeps the other expired entries.

Benchmarks (-tags netgo,slicelabels, 48 cores)

Heap per entry, 100k entries with ~190-byte keys (BenchmarkLruCache_BytesPerEntry, new):

IDs per entry accounted master heap this PR heap
1 187 B 607 B 323 B (-47%)
10 259 B 670 B 387 B (-42%)
100 979 B 1486 B 1203 B (-19%)

Existing benchmarks, benchstat over 5 runs:

master this PR
LruCacheHitUnderPressure sec/op 1092 ns 802 ns -26.6%
CacheGetPromise sec/op 572 ns 469 ns -18.0%
LruCacheHitUnderPressure allocs/op 8 5 -37.5%
CacheGetPromise allocs/op 4 1 -75.0%
CacheGetPromise B/op 222 14 -93.7%

Tests

  • TestLruCache_ConsistentUnderConcurrency: 64 goroutines, a clock advancing past the TTL while fetches are in flight, and a background expire(). Checks that cachedBytes equals the sum of entry sizes, that map and list sizes match, and that list links are consistent. The equivalent check against the current implementation fails in the expiring cases.
  • TestLruCache_DoesNotCacheErrors
  • TestPostingsCache_WaiterHonorsContextWhileFetchRuns
  • TestLruCache_FetchPanicDoesNotPoisonKey
  • Existing pkg/storage/tsdb tests pass, including under -race, and the ingester postings cache tests pass.

Not in this PR: the accounting still charges only len(key) + 8*len(ids). The remaining real cost is about 1.7x the accounted bytes for 1-ID entries. Charging a fixed per-entry overhead would make max_bytes reflect real heap; that changes how much a given max_bytes holds, so I'd rather do it separately.

Which issue(s) this PR fixes:
Fixes #

Checklist

  • Tests updated
  • Documentation added
  • CHANGELOG.md updated - the order of entries should be [CHANGE], [FEATURE], [ENHANCEMENT], [BUGFIX]
  • docs/configuration/v1-guarantees.md updated if this PR introduces experimental flags

Each cached entry used four heap objects: the promise, its done channel,
a list.Element and the key boxed into it, plus a sync.Map node. Entries
are now one object linked into an intrusive LRU list, kept in a plain map
under the cache mutex that every hit already takes. The done channel and
error live in an inflight call that is dropped once the fetch succeeds.

This also fixes byte accounting drift. Evicting the front of the list
called LoadAndDelete on the key, which could delete a newer promise that
had replaced an expired one mid-fetch, leaving the old bytes counted.
Eviction now removes exactly the entry it unlinks.

Failed fetches are no longer cached until the TTL expires, and a caller
waiting on another caller's fetch now returns when its own context is
cancelled.

Signed-off-by: Alan Protasio <alanprot@gmail.com>
Signed-off-by: Alan Protasio <alanprot@gmail.com>
A refresh used to skip eviction, so a larger refreshed value could leave
cachedBytes over max_bytes until the next fresh miss or periodic purge.
Run eviction after every completed fetch, and update the TTL test, whose
first refresh now also sweeps the other expired entries.

Signed-off-by: Alan Protasio <alanprot@gmail.com>
The lint target's faillint check blocks sync/atomic in favor of go.uber.org/atomic, which provides the same generic Pointer type.

Signed-off-by: Alan Protasio <alanprot@gmail.com>
@alanprot
alanprot marked this pull request as ready for review November 18, 2026 01:28
@alanprot
alanprot requested a review from a team as a code owner November 18, 2026 01:28
@alanprot
alanprot requested a review from yeya24 November 18, 2026 01:28
A panicking fetch left its entry in flight with its done channel never
closed. Callers joined to it blocked until their own deadline, and every
later caller for that key joined the dead fetch, so the key stayed
unusable until the entry was evicted for space. QueryStream recovers
runtime panics, so the ingester keeps running with the stuck key.

The fetch now runs under a deferred cleanup that removes the entry if it
is still current, gives waiters errFetchPanicked and closes done before
the panic continues. The next caller fetches again.

Signed-off-by: Alan Protasio <alanprot@gmail.com>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant