Skip to content

refactor: streamline user cache management by consolidating permission and role cache clearing functions - #7963

Merged
wxg0103 merged 1 commit into
v3from
pr@v3@refactor_user_cache
Nov 17, 2026
Merged

wxg0103 merged 1 commit into
v3from
pr@v3@refactor_user_cache

Conversation

@shaohuzhang1

Copy link
Copy Markdown
Contributor

refactor: streamline user cache management by consolidating permission and role cache clearing functions

def clear_user_permission_and_role_cache(user_ids):
"""同时清理指定用户的权限缓存与角色缓存。"""
clear_user_permission_cache(user_ids)
clear_user_role_cache(user_ids)

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Overall, the added helpers are reasonable, but there are several correctness and edge-case issues worth fixing.

Potential issues

  1. One-shot iterators/generators are consumed twice
    In clear_user_permission_and_role_cache, the same user_ids object is passed to two functions. If it is a generator/iterator/map/filter, the first call consumes it, so the second call receives an exhausted iterator and clears nothing.

    Example:

    clear_user_permission_and_role_cache(iter([1, 2, 3]))
    # permission cache may be cleared, role cache will not be

    Fix by normalizing once before calling both clear functions.

  2. bytes normalization is incorrect

    str(b"123")  # "b'123'"

    This will not match a cache key built from "123". If bytes are allowed, decode them explicitly.

  3. Empty values are not actually fully removed
    The docstring says “剔除空值”, but only None is skipped. Empty string "" is still converted to "" and included. If empty strings are invalid user IDs, skip them explicitly. Be careful not to skip valid 0 IDs.

  4. String conversion must match the cache key builder
    _normalize_user_ids converts every ID to str. This is only correct if get_key() expects a string. If the cache key builder expects an int, UUID, or another representation, the generated keys will not match the keys being cached.

  5. Possible double-versioning
    cache.delete_many(..., version=version) applies Django’s cache versioning. Make sure get_key(user_id) returns the base/unversioned key, not an already finalized key. If get_key already applies the version or calls cache.make_key, version=version will double-apply it.

  6. Default cache backend only
    _delete_user_cache uses the global cache object. If permission/role caches are stored in a named cache alias, e.g. caches["permission"], these clear functions will delete from the wrong backend.

  7. Invalidation order may cause stale permission cache repopulation
    If permission cache is derived from role cache, clear_user_permission_and_role_cache currently clears permission first, then role. A concurrent request could repopulate permission cache from the still-stale role cache between those two operations. Safer order:

    clear_user_role_cache(user_ids)
    clear_user_permission_cache(user_ids)
  8. Large input handling
    _normalize_user_ids materializes the whole input into a list and does not deduplicate. For large inputs, this can create a large delete_many call. Consider deduplicating and optionally chunking deletions.

  9. No return value / observability
    The clear functions return None. If useful, return or log the number of attempted keys, or log failures from the cache backend.

Suggested adjustment

def _normalize_user_ids(user_ids):
    if isinstance(user_ids, bytes):
        user_ids = [user_ids.decode("utf-8")]
    elif isinstance(user_ids, str):
        user_ids = [user_ids]
    else:
        try:
            iter(user_ids)
        except TypeError:
            user_ids = [user_ids]

    normalized = []
    seen = set()

    for user_id in user_ids:
        if user_id is None or user_id == "":
            continue
        if isinstance(user_id, bytes):
            user_id = user_id.decode("utf-8")

        key = str(user_id)
        if key not in seen:
            seen.add(key)
            normalized.append(key)

    return normalized


def clear_user_permission_and_role_cache(user_ids):
    user_ids = _normalize_user_ids(user_ids)

    # Clear role first if permission cache is derived from role cache.
    clear_user_role_cache(user_ids)
    clear_user_permission_cache(user_ids)

Also add tests for: None, empty list, single string, single integer, generator, duplicate IDs, empty string, bytes, and 0 if 0 is a valid ID.

key = Cache_Version.PERMISSION_LIST.get_key(user_id=user_id)
cache.delete(key, version=version)
clear_user_permission_cache(users_id)
return instance

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

Summary

The diff replaces inline cache-invalidation logic with a helper clear_user_permission_cache([user_id]). This is a reasonable refactor, but there are several issues worth verifying.

Potential Issues

  1. Behavioral change in auth_resource_batch (first hunk)

    • The removed code pasted the same user_id for every resource (a pre-existing bug), but the new helper call clear_user_permission_cache([user_id]) also only clears one user. Confirm that all affected users get their cache invalidated, not just one. If the batch operation affects multiple users, this will leave stale caches.
  2. Empty-list handling

    • In the last hunk, users_id may be empty or falsy. Verify that clear_user_permission_cache([]) is safe and does not raise / does not issue a pointless round-trip to the cache backend. If it unconditionally loops over an empty list, it's a no-op — acceptable — but consider guarding (if users_id:) if the helper has side effects.
  3. Consistency of import

    • Cache_Version and django.core.cache imports are removed. Ensure no other references to cache or Cache_Version remain in this module (would cause NameError at runtime). Grep before merging.
  4. Silent failure semantics

    • Previously the delete was explicit; now it depends on the helper's error handling. Confirm the helper doesn't swallow exceptions silently or that swallowing is intended (stale permissions = security risk).
  5. Duplicate detection in auth_resource_batch

    • Not part of this diff, but note: the batch method still iterates using a single user_id, which combined with the cache-clearing change suggests the method may implicitly assume one user. Worth double-checking scope.

Optimization Suggestions

  • Deduplicate users_id before passing to the helper: clear_user_permission_cache(set(users_id)) — avoids redundant cache deletes when the same user appears multiple times.
  • Use delete_many / pipeline inside clear_user_permission_cache for bulk invalidation rather than per-key deletes, if the cache backend supports it (e.g. Redis delete(*keys)).
  • Add a docstring/test for clear_user_permission_cache noting that it accepts a list of user IDs, since callers pass both [user_id] and users_id.

Minor Nit

  • clear_user_permission_cache([user_id]) — the list wrapper is fine, but if the helper clearly expects an iterable, keep it consistent across all call sites (it already is).

for user_id in member_user_ids:
cache.delete(Cache_Version.PERMISSION_LIST.get_key(user_id=user_id), version=version)
clear_user_permission_cache(member_user_ids)
return instance

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Here's a review of the diff:

Observations

1. clear_user_permission_cache likely expects a list, but receives a QuerySet or set

  • In the first hunk, member_user_ids is a QuerySet (.values_list(...) without list()).
  • In the second hunk, member_user_ids is now a set.
  • If the helper internally does something like ",".join(member_user_ids) or indexes into it, a set will fail (not subscriptable) or produce nondeterministic ordering. Verify the helper's signature; pass a list(...) explicitly for safety:
    clear_user_permission_cache(list(member_user_ids))

2. Behavioral change in caching semantics — cache.delete(..., version=...) vs helper

  • The original code deleted keys using an explicit version=Cache_Version.PERMISSION_LIST.get_version(). Ensure the new helper uses the exact same key derivation and version logic. If clear_user_permission_cache doesn't replicate the version handling, permission cache invalidation silently breaks (a serious correctness bug).

3. QuerySet evaluation order / DB trips

  • Previously, .values_list("user_id", flat=True) was iterated lazily but only once per group.
  • Now in the second hunk, member_user_ids.update(queryset) forces iteration — fine, but note that each SystemUserGroupRelation.filter(...) still triggers a separate query per user_group_id. Consider collapsing:
    member_user_ids = set(
        QuerySet(SystemUserGroupRelation)
        .filter(group_id__in=user_group_ids)
        .values_list("user_id", flat=True)
    )
    This reduces N queries to one — a real optimization.

4. Same optimization applies to the first hunk

member_user_ids = list(
    SystemUserGroupRelation.objects.filter(group_id=user_group_id)
    .values_list("user_id", flat=True)
)

and possibly guard on empty (if user_group_id is None: ...). At minimum, skip calling the helper when the sequence is empty to avoid needless work.

5. First hunk: QuerySet(...).bulk_create(...) if len(save_list) > 0 else None — pre-existing issue

  • len(save_list) > 0 → prefer if save_list:. Also QuerySet(...) wrapping looks unusual; typically Model.objects.bulk_create(...). Not part of the diff, but a latent irregularity worth flagging.

6. Unused/removed imports

  • Removing cache and Cache_Version from this module is fine only if they are not used elsewhere in the file (the diff shows they were removed at the top-level import site but used further down at line ~460 region and possibly other spots). Grep the file for cache. and Cache_Version. before merging — otherwise you'll get NameError.

7. Type consistency

  • First hunk passes a QuerySet to clear_user_permission_cache; second hunk passes a set. For readability and to guarantee the helper contract, normalize both to list (or whatever the helper documents).

8. No error handling around cache clearing

  • If clear_user_permission_cache raises (e.g., Redis down), the transaction may roll back the DB changes or surface an unexpected exception after DB writes are already done. Consider whether cache-clear failure should be non-fatal (log-and-continue) for permission updates.

9. with_valid parameter unused

  • In both edit methods, with_valid isn't referenced — pre-existing, but worth removing or documenting.

Suggested patch refinement

# Hunk 1
member_user_ids = list(
    SystemUserGroupRelation.objects
    .filter(group_id=user_group_id)
    .values_list("user_id", flat=True)
)
if member_user_ids:
    clear_user_permission_cache(member_user_ids)

# Hunk 2
member_user_ids = list(
    SystemUserGroupRelation.objects
    .filter(group_id__in=user_group_ids)
    .values_list("user_id", flat=True)
    .distinct()
)
if member_user_ids:
    clear_user_permission_cache(member_user_ids)

Summary

  • Correctness risk (highest): helper must replicate original version/key semantics; verify it handles a QuerySet/set gracefully.
  • Performance win available: collapse per-group queries into a single group_id__in=... query with distinct().
  • Style: pass list(...), guard against empty sequences, confirm removed imports aren't referenced elsewhere.

@wxg0103
wxg0103 force-pushed the pr@v3@refactor_user_cache branch from 0a7fc1b to d6e32dc Compare November 17, 2026 04:38
def clear_user_permission_and_role_cache(user_ids):
"""同时清理指定用户的权限缓存与角色缓存。"""
clear_user_permission_cache(user_ids)
clear_user_role_cache(user_ids)

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Here are my observations on the diff:

Potential Issues

  1. cache.delete_many(..., version=version) — version parameter support: Django's cache.delete_many() does not accept a version keyword argument (only get, set, add, get_or_set, incr, decr, and delete/set_many-style methods handle versions inconsistently; delete_many in the default backend ignores/has no version). This will likely raise TypeError: delete_many() got an unexpected keyword argument 'version'. If your custom cache backend supports it, fine — but this is fragile and worth verifying. The version is normally baked into the key via get_key, so passing version here may be redundant or incorrect.

  2. Assumption about cache_item.value shape: _delete_user_cache assumes Cache_Version.PERMISSION_LIST.value is a (version, get_key) tuple. This coupling is implicit. If .value returns something else (e.g., a dataclass/enum with named fields), this breaks silently or loudly. Consider unpacking defensively or documenting the contract.

  3. _normalize_user_ids semantics for bytes: Treating bytes like str wraps it as a single item, then str(b'abc') yields "b'abc'", which is almost certainly not the intended key. Clarify whether bytes are a legitimate input; if not, don't special-case them like strings.

  4. clear_user_permission_and_role_cache — partial failure: If clear_user_permission_cache succeeds but clear_user_role_cache raises, you get inconsistent cache state. Not necessarily a bug, but callers should be aware there's no atomicity.

  5. No keys= batching / large iterables: delete_many with a very large user_ids list can build a huge key list in memory. For bulk imports/migrations, consider chunking.

  6. Double normalization cost: Each public function calls _normalize_user_ids, and clear_user_permission_and_role_cache calls both helpers—so normalization happens twice on the same input. Minor, but could normalize once and pass lists down.

Optimization / Style Suggestions

  1. Simplify _normalize_user_ids iterator handling:

    def _normalize_user_ids(user_ids):
        if user_ids is None:
            return []
        if isinstance(user_ids, (str, bytes)):
            user_ids = [user_ids]
        try:
            iter(user_ids)
        except TypeError:
            user_ids = [user_ids]
        return [str(uid) for uid in user_ids if uid is not None]

    Note using a list comprehension also slightly reduces overhead vs. manual append.

  2. Deduplicate ids: set conversion before building keys avoids redundant work on delete_many: unique = {str(uid) for uid in ...}. Worth it for large inputs.

  3. Comment language consistency: The docstrings use Chinese (将用户ID...) while the module header/comments elsewhere are English. Align with project conventions for maintainability.

  4. Delete-on-empty short-circuit is good — keep it. Consider returning the count from delete_many for observability/tests.

  5. Naming: get_data parameter (in existing code above the diff) shadows nothing problematic, but _delete_user_cache might better be named _delete_user_caches since it handles multiple users/keys.

  6. Type hints: Adding Iterable[Union[str, int]] -> List[str] annotations to the new helpers would improve clarity, consistent with the existing key: str annotation style.

Summary of highest-priority items

  • Verify cache.delete_many() actually accepts version= in your backend (item 1).
  • Confirm Cache_Version.*.value == (version, get_key) contract (item 2).
  • Reconsider bytes handling in _normalize_user_ids (item 3).

@wxg0103 wxg0103 closed this Nov 17, 2026
@wxg0103 wxg0103 reopened this Nov 17, 2026
for user_id in member_user_ids:
cache.delete(Cache_Version.PERMISSION_LIST.get_key(user_id=user_id), version=version)
clear_user_permission_cache(member_user_ids)
return instance

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The changes are largely sound (moving cache invalidation into a helper, and collecting user IDs before clearing). A few observations:

Potential issues:

  1. member_user_ids type inconsistency in the first hunk. In edit() (around line 253), clear_user_permission_cache(member_user_ids) receives a lazy ValuesListQuerySet. If the helper iterates it once, that's fine; but if it needs to iterate multiple times or len(), a queryset is still re-evaluated per iteration. Consider wrapping in set(...) for consistency with the second hunk:

    member_user_ids = set(
        QuerySet(SystemUserGroupRelation).filter(group_id=user_group_id).values_list("user_id", flat=True)
    )
  2. Behavioral difference of dedup / ordering. The old loop iterated per-group and deleted per-user immediately, so duplicate user_ids across groups could cause redundant deletes but were harmless. The new set() approach is better (dedups, single batched call), but ensure clear_user_permission_cache accepts an iterable/set and not a queryset specifically.

  3. Transaction/consistency. If bulk_create and cache clearing aren't inside the same transaction, a failure between them can leave stale cache. Confirm the caller wraps this in transaction.atomic() (or that stale-cache-on-failure is acceptable).

  4. save_list/update_list unused variables? In the shown snippet, update_list is built but never used in the visible code after the if len(update_list) > 0 else None expression — verify the ternary isn't dead code (the result isn't assigned anywhere in either hunk shown).

  5. Batch-size for bulk_create. For large rule sets, pass batch_size= to bulk_create to avoid one giant INSERT and DB parameter limits.

  6. Empty-set safety. clear_user_permission_cache(set()) should be a no-op; confirm the helper early-returns on empty input to avoid unnecessary round-trips (e.g., a delete_many with empty keys).

Style/nits:

  • QuerySet(Model).filter(...) instead of Model.objects.filter(...) works but is unconventional; Model.objects.filter is more idiomatic.
  • The ... if len(x) > 0 else None expression statements are confusing; prefer explicit if save_list: bulk_create(...).
  • Confirm the import switch (Cache_Version/cache removed) doesn't break other usages in the file — grep before merging.

Optimization: skimming two hunks only; consider also invalidating when members are removed from a group (not shown), otherwise stale permission cache may persist.

clear_user_permission_cache([user_id])

return None

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Here's my review:

Potential Issues

  1. clear_user_permission_and_role_cache([user.id]) called after early return path is inconsistent — In the first hunk (line ~827), the cache clear is only invoked on the "create default mapping" branch, immediately before return. This means other branches in the same function that also mutate user-role mappings may not invalidate the cache. Verify the function has a single consistent invalidation point, otherwise stale permissions persist.

  2. Loss of transactional safety / cache-vs-DB ordering — The old code deleted cache keys after DB writes in update_user_role, and the new call preserves that order. However, if the surrounding code is inside a transaction (transaction.atomic) and later rolls back, the cache will already have been cleared (harmless but wasteful) — conversely if cache clearing raises, the DB write commits without invalidation. Consider clearing after commit (transaction.on_commit(...)).

  3. Single-element list argument — Passing [user.id] / [user_id] to batch-style helpers suggests the helper is designed for multi-user invalidation. Double-check the helper signature accepts List[str|UUID] and that user.id type matches (str vs. UUID). A mismatch could silently produce no-op deletes.

  4. Parameter shadowing in set_user_groups — The new code uses clear_user_permission_cache([user_id]), while the old code used Cache_Version.PERMISSION_LIST.get_key(user_id=user_id). Confirm the helper internally applies the same key-building logic (including version prefix); otherwise the cache won't actually be cleared.

  5. Behavior change: version bump removal — Old code did cache.delete(..., version=<current_version>). If the helper instead bumps the version or clears with a different version source, semantics differ subtly. Confirm the helper reads the same Cache_Version and doesn't skip versioned namespace handling.

  6. Local re-import inside function removed — The diff removes inline Cache_Version usage; ensure Cache_Version import is now unused elsewhere in this file (lint/dead-import), or still needed (if so, keep it).

  7. Error handling — The original inline deletes would surface any exceptions from cache.delete. Delegating to clear_user_permission_*_cache centralizes logic (good), but verify the helper doesn't swallow exceptions quietly — a silent failure leaves users with stale permissions.

  8. The return placement in the first hunk — Placing the cache clear immediately before return couples it specifically to that branch. If additional branches return early elsewhere in update_user_role, they must also invalidate. Prefer a try/finally or post-processing hook.

Optimization Suggestions

  • Consider replacing per-branch invalidation with a single invalidation at the end of update_user_role, or use a signal/on_commit pattern so all mutation paths are covered.
  • If clear_user_permission_and_role_cache and clear_user_permission_cache do simple deletes, ensure they use batch/pipeline operations when passed multiple IDs to avoid N round-trips.
  • Add unit tests asserting that cache keys are invalidated on all role-mutation branches (including early returns).
  • Verify import ordering (alphabetical): common.utils.cache_util should sit between common.utils.common and common.utils.rsa_util — currently it does, good.

Summary

The refactor improves consistency by consolidating cache-invalidation logic, which is a good direction. The main risks are branch coverage (not all early returns invalidate), type/signature assumptions on the helper, and transaction/cache-ordering semantics. Validate these before merging.

@wxg0103
wxg0103 merged commit 24801b9 into v3 Nov 17, 2026
4 of 5 checks passed
@wxg0103
wxg0103 deleted the pr@v3@refactor_user_cache branch November 17, 2026 04:39
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants