Repository navigation
fix(state-manager): stop redis lock handoffs from stalling events until the lock expires - #7534
Conversation
…ffs, keep a lease's lock while its state is in use and wake waiters a release missed
[Medium impact] The PR appears safe to merge; no new actionable issues were found. SummaryFixes Redis lock handoffs so cancelled events and missed notifications do not stall later events.
Reviews (11) · Last reviewed commit: "Condense the redis lock handoff fragment" · Reviewed by Greptile |
Merging this PR will degrade performance by 4.36%1 warning
Warning Please fix the performance issues or acknowledge them on CodSpeed. Performance Changes
Tip Investigate this regression by commenting Comparing |
…thin each use's lock expiration, and report a failed renewal
…tion interrupts it
…ed state changes hands
…of asyncio.wait_for, which can swallow its cancellation on Python 3.11
There was a problem hiding this comment.
All reported issues were addressed across 4 files
Reply with feedback, questions, or to request a fix.
View guided diff | Re-trigger cubic
…ts lock until the write lands
…stead of a try in the loop
…extract the lease A remote waiter now re-registers as a lock contender on each re-check, so a lease that outlasts lock_expiration no longer leases again ahead of it; the mock redis no longer reports an sadd for an existing member. A failed lease is reported once: the done callback only reports what flush() did not already report. The lease closures move into a module-level _Lease dataclass, with no change in behavior. The news fragment is split into short sentences.
There was a problem hiding this comment.
All reported issues were addressed across 4 files (changes from recent commits).
Reply with feedback, questions, or to request a fix.
View guided diff | Re-trigger cubic
Closing a second manager on the shared client disconnected the pool the main manager still used.
Under CPU load, our app's events stalled for 5 or 10 seconds (the oplock hold time and the lock expiration), and some updates made under a lease were lost.
StateManagerRedishad several ways to lose a lock or a wakeup:supersedesand a newon_load_internalcancel events at any await. This covers aSET NXwhose reply never arrived, the awaits in_try_modify_statebefore a lease or the single update owns the lock, and a lease task cancelled before its first step, which never enters the context that releases the lock.GETDEL, so a holder releasing after its lock expired deleted the next holder's lock.LockExpiredErrorand dropped every change made under the lease.This PR:
AsyncExitStack.pop_all()once the state is fetchedLockExpiredErrorEach case has a test in
tests/units/istate/manager/test_redis.pythat fails without the fix.test_oplock_immediate_cancelnow checks the outcome, because its lease can be cancelled before it starts.With a script that cancels a waiter around the handoff (2 s lock expiration, 50 ms redis latency), 19 of 30 waiters waited out the expiration before and none of 60 after. A fuzzer with up to 2 s holders lost up to 23 of 26 increments per round before and none after.
The unit tests pass (12512 without Redis, 12525 against Redis with and without oplock).