Page MenuHomePhabricator

Implement smart waits for rarely updated repositories
ClosedPublic

Authored by epriestley on Apr 16 2014, 1:37 AM.
Tags
None
Referenced Files
Unknown Object (File)
Fri, Sep 30, 3:39 PM
Unknown Object (File)
Mon, Sep 26, 8:34 AM
Unknown Object (File)
Thu, Sep 22, 11:26 PM
Unknown Object (File)
Tue, Sep 20, 11:22 PM
Unknown Object (File)
Mon, Sep 19, 3:04 AM
Unknown Object (File)
Sat, Sep 17, 12:36 AM
Unknown Object (File)
Wed, Sep 14, 9:12 PM
Unknown Object (File)
Mon, Sep 12, 10:46 PM
Subscribers
Tokens
"Pterodactyl" token, awarded by btrahan.

Details

Summary

Ref T4605. When figuring out how long to wait to update a repository, factor in when it was last pushed. For rarely updated repositories, wait longer between updates.

(A slightly funky thing about this is that empty repos update every 15 seconds, but that seems OK for the moment.)

Test Plan

Ran bin/phd debug pulllocal and saw sensible calculations and output:

...
<VERB> PhabricatorRepositoryPullLocalDaemon Last commit to repository "rPOEMS" was 1,239,608 seconds ago; considering a wait of 6,198 seconds before update.
>>> [79] <query> SELECT * FROM `repository` r   ORDER BY r.id DESC 
<<< [79] <query> 514 us
>>> [80] <query> SELECT * FROM `repository_statusmessage` WHERE statusType = 'needs-update' 
<<< [80] <query> 406 us
<VERB> PhabricatorRepositoryPullLocalDaemon Repository "rINIH" is not due for an update for 8,754 second(s).
<VERB> PhabricatorRepositoryPullLocalDaemon Repository "rDUCK" is not due for an update for 14 second(s).
<VERB> PhabricatorRepositoryPullLocalDaemon Repository "rMTESTX" is not due for an update for 21,598 second(s).
<VERB> PhabricatorRepositoryPullLocalDaemon Repository "rQWER" is not due for an update for 14 second(s).
<VERB> PhabricatorRepositoryPullLocalDaemon Repository "rBT" is not due for an update for 13 second(s).
<VERB> PhabricatorRepositoryPullLocalDaemon Repository "rSVNX" is not due for an update for 21,598 second(s).
<VERB> PhabricatorRepositoryPullLocalDaemon Repository "rINIG" is not due for an update for 13 second(s).
<VERB> PhabricatorRepositoryPullLocalDaemon Repository "rHGTEST" is not due for an update for 21,598 second(s).
<VERB> PhabricatorRepositoryPullLocalDaemon Repository "rBTX" is not due for an update for 14 second(s).
<VERB> PhabricatorRepositoryPullLocalDaemon Repository "rGX" is not due for an update for 13 second(s).
<VERB> PhabricatorRepositoryPullLocalDaemon Repository "rMTX" is currently updating.
<VERB> PhabricatorRepositoryPullLocalDaemon Repository "rPOEMS" is not due for an update for 6,198 second(s).
<VERB> PhabricatorRepositoryPullLocalDaemon Repository "rPHU" is currently updating.
<VERB> PhabricatorRepositoryPullLocalDaemon Repository "rSVN" is not due for an update for 21,598 second(s).
<VERB> PhabricatorRepositoryPullLocalDaemon Repository "rPHY" is currently updating.
<VERB> PhabricatorRepositoryPullLocalDaemon Repository "rGTEST" is not due for an update for 21,598 second(s).
<VERB> PhabricatorRepositoryPullLocalDaemon Repository "rINIS" is not due for an update for 6,894 second(s).
<VERB> PhabricatorRepositoryPullLocalDaemon Repository "rARCLINT" is not due for an update for 21,599 second(s).
<VERB> PhabricatorRepositoryPullLocalDaemon Repository "rLPHX" is not due for an update for 1,979 second(s).
<VERB> PhabricatorRepositoryPullLocalDaemon Repository "rARC" is not due for an update for 1,824 second(s).
<VERB> PhabricatorRepositoryPullLocalDaemon Repository "rINIHG" is not due for an update for 21,599 second(s).
...

Diff Detail

Repository
rP Phabricator
Lint
Lint Skipped
Unit
Tests Skipped

Event Timeline

epriestley retitled this revision from to Implement smart waits for rarely updated repositories.
epriestley updated this object.
epriestley edited the test plan for this revision. (Show Details)
epriestley added a reviewer: btrahan.
btrahan edited edge metadata.

I'm very down 'cuz of the arc land hook. :)

Otherwise, I'd expect the "smart" part of the smart wait to cause confusion. I think the "not updated all that much" *implies* that folks won't want to look at the repository that soon after a land, but its a weak implication to me.

Also, I don't know if "batching" (i.e. processing N commits instead of 1) ends up being actually less system-resource intensive, but if it does, I'd expect the best algorithm to be waiting a bit on frequently updated repositories to get that batch performance impact, waiting less for moderately updated repositories, and then waiting a long time again for infrequently updated repositories to exploit the "not updated all that much" implication.

This revision is now accepted and ready to land.Apr 16 2014, 6:26 PM

I think the "not updated all that much" *implies* that folks won't want to look at the repository that soon after a land, but its a weak implication to me.

Yeah, agreed. I'm hoping the "arc land" hook will fix a lot of this (we also get the hint for hosted repositories in all cases, so those are totally reliable). We can go add config or something if it's not good enough, but ideally this is close enough and it "just works" from users' points of view.

Also, I don't know if "batching" (i.e. processing N commits instead of 1) ends up being actually less system-resource intensive...

I think the break-even point on this is way above one commit per second, and we've never seen a repository which updates that fast.

epriestley updated this revision to Diff 20860.

Closed by commit rPc402d7d3076d (authored by @epriestley).