Page MenuHomePhabricator

When acquiring a GlobalLock, put good connections that just got unlucky back in the pool
ClosedPublic

Authored by epriestley on Jun 26 2020, 1:00 AM.
Tags
None
Referenced Files
F12840221: D21369.id50858.diff
Thu, Mar 28, 8:18 PM
Unknown Object (File)
Tue, Mar 19, 3:31 PM
Unknown Object (File)
Tue, Mar 5, 12:41 PM
Unknown Object (File)
Jan 11 2024, 11:57 AM
Unknown Object (File)
Dec 30 2023, 4:40 PM
Unknown Object (File)
Dec 30 2023, 3:52 AM
Unknown Object (File)
Dec 27 2023, 1:27 PM
Unknown Object (File)
Dec 25 2023, 8:28 PM
Subscribers
None

Details

Summary

See PHI1794, which describes a connection exhaustion issue with a large number of webhook tasks in queue.

The "GlobalLock" mechanism manages a separate connection pool from the main pool, and webhook workers immediately try to grab a webhook lock with a 0-second wait when they start. So far, this is fine.

Prior to this change, good connections which fail to acqiure a lock are discarded. This can lead to connection exhaustion as the worker rapidly cycles through lock attempts: the connections will remain open for at least 60 seconds (since D16389) in an effort to avoid outbound port exhaustion, but they're effectively orphaned because they aren't part of the main pool and aren't part of the lock pool. We're basically leaking a connection every time we fail to lock.

Failing to lock doesn't mean we need to discard the connection: it's a completely suitable connection for reuse. Instead of dropping it on the floor, put it into the lock pool.

Test Plan
  • Used "bin/webhook call ... --count 10000 --background" to queue a large number of webhook calls against a slow ("sleep(15);") webhook.
  • Used "bin/phd launch 32 taskmaster" to start taskmasters.
  • Observed MySQL connection behavior:
    • Before change: 2048 configured connections immediately exhausted.
    • After change: connections stable at ~160ish.
  • Ran queue for a while, saw expected single-threaded calls to webhook.

Diff Detail

Repository
rP Phabricator
Lint
Lint Not Applicable
Unit
Tests Not Applicable

Event Timeline