Page MenuHomePhabricator

Importing a repo with invalid credentials will cause a lot of retries in too short a period of time
Closed, ResolvedPublic

Description

If you add a repo with invalid credentials the daemons will keep trying over and over again in a loop:

# grep 'Cloning into bare repository' /var/phabricator/phd/log/daemons.log  | wc -l
8572694

graph.png (442×747 px, 74 KB)

(Drop when I fixed it)

I'm not sure if:

  • eh, this isn't that bad, look at your graphs fools!
  • Maybe add some kind of backoff
  • fail louder somehow

Is the reasonable response.

Event Timeline

cburroughs raised the priority of this task from to Needs Triage.
cburroughs updated the task description. (Show Details)
cburroughs added a subscriber: cburroughs.

The expectation is that they'll wait ~15 seconds between retires, so this looks like a bug. It's possible that D11296 is the fix, although that patch is missing a lot of context.

(If the 8.5M attempts were accumulated over 4 years of trying every 15 seconds, this is working as intended, but I assume this repository was added less than 4 years ago.)

epriestley renamed this task from Importing a repo with invalid credentials will cause a lot to Importing a repo with invalid credentials will cause a lot of retries in too short a period of time.Feb 2 2015, 4:20 PM

I can't immediately reproduce this -- I get proper 15-second waits between clone attempts.

Here's the excerpted output of phd debug pulllocal:

<VERB> PhabricatorRepositoryPullLocalDaemon Starting update for repository "rBADCREDENTIALS".
>>> [62] <exec> $ '/Users/epriestley/dev/core/lib/phabricator/bin/repository' update  -- 'BADCREDENTIALS'
2015-02-02 8:26:38 AM [STDE] <<< [62] <exec> 1,218,636 us
<VERB> PhabricatorRepositoryPullLocalDaemon Resolving update for "rBADCREDENTIALS".
[2015-02-02 08:26:37] EXCEPTION: (PhutilProxyException) Error while updating the "rBADCREDENTIALS" repository. {>} (CommandException) Command failed with error #255!
COMMAND
'/Users/epriestley/dev/core/lib/phabricator/bin/repository' update  -- 'BADCREDENTIALS'
STDOUT
(empty)
STDERR
[2015-02-02 08:26:37] EXCEPTION: (CommandException) Command failed with error #128!
COMMAND
git clone --bare -- 'xxxxx' '/Users/epriestley/dev/core//repo/local/BADCREDENTIALS'
STDOUT
(empty)
STDERR
Cloning into bare repository '/Users/epriestley/dev/core//repo/local/BADCREDENTIALS'...
Warning: Permanently added 'github.com,192.30.252.129' (RSA) to the list of known hosts.
Permission denied (publickey).
fatal: Could not read from remote repository.
Please make sure you have the correct access rights
and the repository exists.
 at [<phutil>/src/future/exec/ExecFuture.php:397]
  #0 ExecFuture::resolvex() called at [<phabricator>/src/applications/repository/storage/PhabricatorRepository.php:293]
  #1 PhabricatorRepository::execxRemoteCommand(string, PhutilOpaqueEnvelope, string) called at [<phabricator>/src/applications/repository/engine/PhabricatorRepositoryPullEngine.php:224]
  #2 PhabricatorRepositoryPullEngine::executeGitCreate() called at [<phabricator>/src/applications/repositor... (618 more bytes) ... at [<phutil>/src/future/exec/ExecFuture.php:397]
  #0 ExecFuture::resolvex() called at [<phabricator>/src/applications/repository/daemon/PhabricatorRepositoryPullLocalDaemon.php:328]
  #1 PhabricatorRepositoryPullLocalDaemon::resolveUpdateFuture(PhabricatorRepository, ExecFuture, integer) called at [<phabricator>/src/applications/repository/daemon/PhabricatorRepositoryPullLocalDaemon.php:198]
  #2 PhabricatorRepositoryPullLocalDaemon::run() called at [<phutil>/src/daemon/PhutilDaemon.php:91]
  #3 PhutilDaemon::execute() called at [<phutil>/scripts/daemon/exec/exec_daemon.php:111]
<VERB> PhabricatorRepositoryPullLocalDaemon Repository "rHGTEST" is not due for an update for 1,430 second(s).
<VERB> PhabricatorRepositoryPullLocalDaemon Repository "rTESTY" is not due for an update for 14 second(s).
<VERB> PhabricatorRepositoryPullLocalDaemon Repository "rBADCREDENTIALS" is not due for an update for 15 second(s).
<VERB> PhabricatorRepositoryPullLocalDaemon Repository "rPHU" is not due for an update for 1,721 second(s).
<VERB> PhabricatorRepositoryPullLocalDaemon Repository "rSVN" is not due for an update for 1,768 second(s).
<VERB> PhabricatorRepositoryPullLocalDaemon Repository "rGTEST" is not due for an update for 1,768 second(s).
<VERB> PhabricatorRepositoryPullLocalDaemon No repositories need updates right now, sleeping...
<VERB> PhabricatorRepositoryPullLocalDaemon Sleeping for 14 more second(s)...
2015-02-02 8:26:39 AM [STDE] >>> [65] <connect> local_repository
<<< [65] <connect> 966 us
<VERB> PhabricatorRepositoryPullLocalDaemon Sleeping for 13 more second(s)...
2015-02-02 8:26:40 AM [STDE] >>> [67] <connect> local_repository
<<< [67] <connect> 882 us
<VERB> PhabricatorRepositoryPullLocalDaemon Sleeping for 12 more second(s)...
2015-02-02 8:26:41 AM [STDE] >>> [69] <connect> local_repository
<<< [69] <connect> 933 us
<VERB> PhabricatorRepositoryPullLocalDaemon Sleeping for 11 more second(s)...
2015-02-02 8:26:42 AM [STDE] >>> [71] <connect> local_repository
<<< [71] <connect> 1,173 us
<VERB> PhabricatorRepositoryPullLocalDaemon Sleeping for 10 more second(s)...
2015-02-02 8:26:43 AM [STDE] >>> [73] <connect> local_repository
<<< [73] <connect> 1,060 us
<VERB> PhabricatorRepositoryPullLocalDaemon Sleeping for 9 more second(s)...
2015-02-02 8:26:44 AM [STDE] >>> [75] <connect> local_repository
<<< [75] <connect> 720 us
<VERB> PhabricatorRepositoryPullLocalDaemon Sleeping for 8 more second(s)...
2015-02-02 8:26:45 AM [STDE] >>> [77] <connect> local_repository
<<< [77] <connect> 705 us
<VERB> PhabricatorRepositoryPullLocalDaemon Sleeping for 7 more second(s)...
2015-02-02 8:26:46 AM [STDE] >>> [79] <connect> local_repository
<<< [79] <connect> 865 us
<VERB> PhabricatorRepositoryPullLocalDaemon Sleeping for 6 more second(s)...
2015-02-02 8:26:47 AM [STDE] >>> [81] <connect> local_repository
<<< [81] <connect> 991 us
<VERB> PhabricatorRepositoryPullLocalDaemon Sleeping for 5 more second(s)...
2015-02-02 8:26:48 AM [STDE] >>> [83] <connect> local_repository
<<< [83] <connect> 1,056 us
<VERB> PhabricatorRepositoryPullLocalDaemon Sleeping for 4 more second(s)...
2015-02-02 8:26:49 AM [STDE] >>> [85] <connect> local_repository
<<< [85] <connect> 892 us
<VERB> PhabricatorRepositoryPullLocalDaemon Sleeping for 3 more second(s)...
2015-02-02 8:26:50 AM [STDE] >>> [87] <connect> local_repository
<<< [87] <connect> 911 us
<VERB> PhabricatorRepositoryPullLocalDaemon Sleeping for 2 more second(s)...
2015-02-02 8:26:51 AM [STDE] >>> [89] <connect> local_repository
<<< [89] <connect> 1,087 us
<VERB> PhabricatorRepositoryPullLocalDaemon Sleeping for 1 more second(s)...
2015-02-02 8:26:52 AM [STDE] >>> [91] <connect> local_repository
<<< [91] <connect> 997 us
<VERB> PhabricatorRepositoryPullLocalDaemon Repository "rSVN" is not due for an update for 1,754 second(s).
<VERB> PhabricatorRepositoryPullLocalDaemon Repository "rHGTEST" is not due for an update for 1,416 second(s).
<VERB> PhabricatorRepositoryPullLocalDaemon Scheduling repository "rTESTY" for an update (0 seconds overdue).
<VERB> PhabricatorRepositoryPullLocalDaemon Repository "rGTEST" is not due for an update for 1,754 second(s).
<VERB> PhabricatorRepositoryPullLocalDaemon Repository "rBADCREDENTIALS" is not due for an update for 1 second(s).
<VERB> PhabricatorRepositoryPullLocalDaemon Repository "rPHU" is not due for an update for 1,707 second(s).
<VERB> PhabricatorRepositoryPullLocalDaemon Starting update for repository "rTESTY".
>>> [95] <exec> $ '/Users/epriestley/dev/core/lib/phabricator/bin/repository' update  -- 'TESTY'
<<< [95] <exec> 254,399 us
<VERB> PhabricatorRepositoryPullLocalDaemon Resolving update for "rTESTY".
        WHERE repositoryID = 4 AND epoch <= 1422894426
        ORDER BY epoch DESC LIMIT 1
<VERB> PhabricatorRepositoryPullLocalDaemon Based on activity in repository "rTESTY", considering a wait of 15 seconds before update.
<VERB> PhabricatorRepositoryPullLocalDaemon Repository "rHGTEST" is not due for an update for 1,416 second(s).
<VERB> PhabricatorRepositoryPullLocalDaemon Repository "rTESTY" is not due for an update for 15 second(s).
<VERB> PhabricatorRepositoryPullLocalDaemon Repository "rPHU" is not due for an update for 1,707 second(s).
<VERB> PhabricatorRepositoryPullLocalDaemon Repository "rBADCREDENTIALS" is not due for an update for 1 second(s).
<VERB> PhabricatorRepositoryPullLocalDaemon Repository "rSVN" is not due for an update for 1,754 second(s).
<VERB> PhabricatorRepositoryPullLocalDaemon Repository "rGTEST" is not due for an update for 1,754 second(s).
<VERB> PhabricatorRepositoryPullLocalDaemon No repositories need updates right now, sleeping...
<VERB> PhabricatorRepositoryPullLocalDaemon Sleeping for 1 more second(s)...
2015-02-02 8:26:53 AM [STDE] >>> [99] <connect> local_repository
<<< [99] <connect> 1,121 us
<VERB> PhabricatorRepositoryPullLocalDaemon Repository "rTESTY" is not due for an update for 14 second(s).
<VERB> PhabricatorRepositoryPullLocalDaemon Repository "rGTEST" is not due for an update for 1,753 second(s).
<VERB> PhabricatorRepositoryPullLocalDaemon Scheduling repository "rBADCREDENTIALS" for an update (0 seconds overdue).
<VERB> PhabricatorRepositoryPullLocalDaemon Repository "rPHU" is not due for an update for 1,706 second(s).
<VERB> PhabricatorRepositoryPullLocalDaemon Repository "rSVN" is not due for an update for 1,753 second(s).
<VERB> PhabricatorRepositoryPullLocalDaemon Repository "rHGTEST" is not due for an update for 1,415 second(s).
<VERB> PhabricatorRepositoryPullLocalDaemon Starting update for repository "rBADCREDENTIALS".
>>> [103] <exec> $ '/Users/epriestley/dev/core/lib/phabricator/bin/repository' update  -- 'BADCREDENTIALS'

If you have a chance and it's not a huge pain, can you show me the output for that locally?

So the daemon log is 17G so grep is a bit slow

[12-Jan-2015 13:43:36] [2015-01-12 13:43:36] EXCEPTION: (PhutilProxyException) Error while updating the "rSENSECACHE" repository. {>} (CommandException) Command failed with error #255!
[12-Jan-2015 13:43:52] [2015-01-12 13:43:52] EXCEPTION: (PhutilProxyException) Error while updating the "rSENSECACHE" repository. {>} (CommandException) Command failed with error #255!
[12-Jan-2015 13:44:07] [2015-01-12 13:44:07] EXCEPTION: (PhutilProxyException) Error while updating the "rSENSECACHE" repository. {>} (CommandException) Command failed with error #255!
[12-Jan-2015 13:44:22] [2015-01-12 13:44:22] EXCEPTION: (PhutilProxyException) Error while updating the "rSENSECACHE" repository. {>} (CommandException) Command failed with error #255!
[12-Jan-2015 13:44:31] [2015-01-12 13:44:31] EXCEPTION: (PhutilProxyException) Error while updating the "rSENSECACHE" repository. {>} (CommandException) Command failed with error #255!
[12-Jan-2015 13:44:32] [2015-01-12 13:44:32] EXCEPTION: (PhutilProxyException) Error while updating the "rSENSECACHE" repository. {>} (CommandException) Command failed with error #255!
[12-Jan-2015 13:44:34] [2015-01-12 13:44:34] EXCEPTION: (PhutilProxyException) Error while updating the "rSENSECACHE" repository. {>} (CommandException) Command failed with error #255!
[12-Jan-2015 13:44:34] [2015-01-12 13:44:34] EXCEPTION: (PhutilProxyException) Error while updating the "rSENSECACHE" repository. {>} (CommandException) Command failed with error #255!
[12-Jan-2015 13:44:34] [2015-01-12 13:44:34] EXCEPTION: (PhutilProxyException) Error while updating the "rSENSECACHE" repository. {>} (CommandException) Command failed with error #255!
[12-Jan-2015 13:44:35] [2015-01-12 13:44:35] EXCEPTION: (PhutilProxyException) Error while updating the "rSENSECACHE" repository. {>} (CommandException) Command failed with error #255!
[12-Jan-2015 13:44:35] [2015-01-12 13:44:35] EXCEPTION: (PhutilProxyException) Error while updating the "rSENSECACHE" repository. {>} (CommandException) Command failed with error #255!
[12-Jan-2015 13:44:35] [2015-01-12 13:44:35] EXCEPTION: (PhutilProxyException) Error while updating the "rSENSECACHE" repository. {>} (CommandException) Command failed with error #255!
[12-Jan-2015 13:44:36] [2015-01-12 13:44:36] EXCEPTION: (PhutilProxyException) Error while updating the "rSENSECACHE" repository. {>} (CommandException) Command failed with error #255!
[12-Jan-2015 13:44:36] [2015-01-12 13:44:36] EXCEPTION: (PhutilProxyException) Error while updating the "rSENSECACHE" repository. {>} (CommandException) Command failed with error #255!
[12-Jan-2015 13:44:36] [2015-01-12 13:44:36] EXCEPTION: (PhutilProxyException) Error while updating the "rSENSECACHE" repository. {>} (CommandException) Command failed with error #255!
[12-Jan-2015 13:44:37] [2015-01-12 13:44:37] EXCEPTION: (PhutilProxyException) Error while updating the "rSENSECACHE" repository. {>} (CommandException) Command failed with error #255!
[12-Jan-2015 13:44:37] [2015-01-12 13:44:37] EXCEPTION: (PhutilProxyException) Error while updating the "rSENSECACHE" repository. {>} (CommandException) Command failed with error #255!
[12-Jan-2015 13:44:37] [2015-01-12 13:44:37] EXCEPTION: (PhutilProxyException) Error while updating the "rSENSECACHE" repository. {>} (CommandException) Command failed with error #255!
[12-Jan-2015 13:44:38] [2015-01-12 13:44:38] EXCEPTION: (PhutilProxyException) Error while updating the "rSENSECACHE" repository. {>} (CommandException) Command failed with error #255!
[12-Jan-2015 13:44:38] [2015-01-12 13:44:38] EXCEPTION: (PhutilProxyException) Error while updating the "rSENSECACHE" repository. {>} (CommandException) Command failed with error #255!
[12-Jan-2015 13:44:38] [2015-01-12 13:44:38] EXCEPTION: (PhutilProxyException) Error while updating the "rSENSECACHE" repository. {>} (CommandException) Command failed with error #255!
[12-Jan-2015 13:44:39] [2015-01-12 13:44:39] EXCEPTION: (PhutilProxyException) Error while updating the "rSENSECACHE" repository. {>} (CommandException) Command failed with error #255!
[12-Jan-2015 13:44:39] [2015-01-12 13:44:39] EXCEPTION: (PhutilProxyException) Error while updating the "rSENSECACHE" repository. {>} (CommandException) Command failed with error #255!
[12-Jan-2015 13:44:40] [2015-01-12 13:44:40] EXCEPTION: (PhutilProxyException) Error while updating the "rSENSECACHE" repository. {>} (CommandException) Command failed with error #255!
[12-Jan-2015 13:44:40] [2015-01-12 13:44:40] EXCEPTION: (PhutilProxyException) Error while updating the "rSENSECACHE" repository. {>} (CommandException) Command failed with error #255!
[12-Jan-2015 13:44:40] [2015-01-12 13:44:40] EXCEPTION: (PhutilProxyException) Error while updating the "rSENSECACHE" repository. {>} (CommandException) Command failed with error #255!
[12-Jan-2015 13:44:41] [2015-01-12 13:44:41] EXCEPTION: (PhutilProxyException) Error while updating the "rSENSECACHE" repository. {>} (CommandException) Command failed with error #255!
[12-Jan-2015 13:44:41] [2015-01-12 13:44:41] EXCEPTION: (PhutilProxyException) Error while updating the "rSENSECACHE" repository. {>} (CommandException) Command failed with error #255!
[12-Jan-2015 13:44:41] [2015-01-12 13:44:41] EXCEPTION: (PhutilProxyException) Error while updating the "rSENSECACHE" repository. {>} (CommandException) Command failed with error #255!
[12-Jan-2015 13:44:41] [2015-01-12 13:44:41] EXCEPTION: (PhutilProxyException) Error while updating the "rSENSECACHE" repository. {>} (CommandException) Command failed with error #255!
[12-Jan-2015 13:44:42] [2015-01-12 13:44:42] EXCEPTION: (PhutilProxyException) Error while updating the "rSENSECACHE" repository. {>} (CommandException) Command failed with error #255!
[12-Jan-2015 13:44:42] [2015-01-12 13:44:42] EXCEPTION: (PhutilProxyException) Error while updating the "rSENSECACHE" repository. {>} (CommandException) Command failed with error #255!
[12-Jan-2015 13:44:43] [2015-01-12 13:44:43] EXCEPTION: (PhutilProxyException) Error while updating the "rSENSECACHE" repository. {>} (CommandException) Command failed with error #255!
[12-Jan-2015 13:44:43] [2015-01-12 13:44:43] EXCEPTION: (PhutilProxyException) Error while updating the "rSENSECACHE" repository. {>} (CommandException) Command failed with error #255!
[12-Jan-2015 13:44:43] [2015-01-12 13:44:43] EXCEPTION: (PhutilProxyException) Error while updating the "rSENSECACHE" repository. {>} (CommandException) Command failed with error #255!
[12-Jan-2015 13:44:44] [2015-01-12 13:44:44] EXCEPTION: (PhutilProxyException) Error while updating the "rSENSECACHE" repository. {>} (CommandException) Command failed with error #255!
[12-Jan-2015 13:44:44] [2015-01-12 13:44:44] EXCEPTION: (PhutilProxyException) Error while updating the "rSENSECACHE" repository. {>} (CommandException) Command failed with error #255!
[12-Jan-2015 13:44:45] [2015-01-12 13:44:45] EXCEPTION: (PhutilProxyException) Error while updating the "rSENSECACHE" repository. {>} (CommandException) Command failed with error #255!
[12-Jan-2015 13:44:45] [2015-01-12 13:44:45] EXCEPTION: (PhutilProxyException) Error while updating the "rSENSECACHE" repository. {>} (CommandException) Command failed with error #255!
[12-Jan-2015 13:44:45] [2015-01-12 13:44:45] EXCEPTION: (PhutilProxyException) Error while updating the "rSENSECACHE" repository. {>} (CommandException) Command failed with error #255!
[12-Jan-2015 13:44:46] [2015-01-12 13:44:46] EXCEPTION: (PhutilProxyException) Error while updating the "rSENSECACHE" repository. {>} (CommandException) Command failed with error #255!
[12-Jan-2015 13:44:46] [2015-01-12 13:44:46] EXCEPTION: (PhutilProxyException) Error while updating the "rSENSECACHE" repository. {>} (CommandException) Command failed with error #255!
[12-Jan-2015 13:44:46] [2015-01-12 13:44:46] EXCEPTION: (PhutilProxyException) Error while updating the "rSENSECACHE" repository. {>} (CommandException) Command failed with error #255!

So some of those are closer together than 15 seconds:

Individual failures looked something like:

[02-Feb-2015 09:35:42] [2015-02-02 09:35:42] EXCEPTION: (PhutilProxyException) Error while updating the "rPOSTMERGENPMINSTALL" repository. {>} (CommandException) Command failed with error #255!
COMMAND
'/home/web/sites/phabricator/phabricator/bin/repository' update  -- 'POSTMERGENPMINSTALL'

STDOUT
(empty)

STDERR
[2015-02-02 09:35:41] EXCEPTION: (CommandException) Command failed with error #128!
COMMAND
git clone --bare -- 'xxxxx' '/var/phabricator/repo/POSTMERGENPMINSTALL'

STDOUT
(empty)

STDERR
Cloning into bare repository '/var/phabricator/repo/POSTMERGENPMINSTALL'...
fatal: could not read Username for 'http://git.clearspring.local': No such device or address
 at [<phutil>/src/future/exec/ExecFuture.php:397]
  #0 ExecFuture::resolvex() called at [<phabricator>/src/applications/repository/storage/PhabricatorRepository.php:294]
  #1 PhabricatorRepository::execxRemoteCommand(string, PhutilOpaqueEnvelope, string) called at [<phabricator>/src/applications/repository/engine/PhabricatorRepositoryPullEngine.php:213]
  #2 PhabricatorRepositoryPullEngine::executeGitCreate() called at [<phabricator>/src/applications/repository/engine/PhabricatorRepositoryPullEngine.php:79]
  #3 PhabricatorRepositoryPullEngine::pullRepository() called at [<phabricator>/src/applications/repository/management/Phabricator... (439 more bytes) ... at [<phutil>/src/future/exec/ExecFuture.php:397]
[02-Feb-2015 09:35:42]   #0 ExecFuture::resolvex() called at [<phabricator>/src/applications/repository/daemon/PhabricatorRepositoryPullLocalDaemon.php:328]
[02-Feb-2015 09:35:42]   #1 PhabricatorRepositoryPullLocalDaemon::resolveUpdateFuture(PhabricatorRepository, ExecFuture, integer) called at [<phabricator>/src/applications/repository/daemon/PhabricatorRepositoryPullLocalDaemo
n.php:198]
[02-Feb-2015 09:35:42]   #2 PhabricatorRepositoryPullLocalDaemon::run() called at [<phutil>/src/daemon/PhutilDaemon.php:91]
[02-Feb-2015 09:35:42]   #3 PhutilDaemon::execute() called at [<phutil>/scripts/daemon/exec/exec_daemon.php:111]
[02-Feb-2015 09:35:42] [2015-02-02 09:35:42] EXCEPTION: (PhutilProxyException) Error while updating the "rSENSECACHE" repository. {>} (CommandException) Command failed with error #255!
COMMAND
'/home/web/sites/phabricator/phabricator/bin/repository' update  -- 'SENSECACHE'

STDOUT
(empty)

STDERR
[2015-02-02 09:35:42] EXCEPTION: (CommandException) Command failed with error #128!
COMMAND
git clone --bare -- 'xxxxx' '/var/phabricator/repo/SENSECACHE'

STDOUT
(empty)

STDERR
Cloning into bare repository '/var/phabricator/repo/SENSECACHE'...
fatal: could not read Username for 'http://git.clearspring.local': No such device or address
 at [<phutil>/src/future/exec/ExecFuture.php:397]
  #0 ExecFuture::resolvex() called at [<phabricator>/src/applications/repository/storage/PhabricatorRepository.php:294]
  #1 PhabricatorRepository::execxRemoteCommand(string, PhutilOpaqueEnvelope, string) called at [<phabricator>/src/applications/repository/engine/PhabricatorRepositoryPullEngine.php:213]
  #2 PhabricatorRepositoryPullEngine::executeGitCreate() called at [<phabricator>/src/applications/repository/engine/PhabricatorRepositoryPullEngine.php:79]
  #3 PhabricatorRepositoryPullEngine::pullRepository() called at [<phabricator>/src/applications/repository/management/PhabricatorRepositoryManageme... (421 more bytes) ... at [<phutil>/src/future/exec/ExecFuture.php:397]
[02-Feb-2015 09:35:42]   #0 ExecFuture::resolvex() called at [<phabricator>/src/applications/repository/daemon/PhabricatorRepositoryPullLocalDaemon.php:328]
[02-Feb-2015 09:35:42]   #1 PhabricatorRepositoryPullLocalDaemon::resolveUpdateFuture(PhabricatorRepository, ExecFuture, integer) called at [<phabricator>/src/applications/repository/daemon/PhabricatorRepositoryPullLocalDaemon.php:198]
[02-Feb-2015 09:35:42]   #2 PhabricatorRepositoryPullLocalDaemon::run() called at [<phutil>/src/daemon/PhutilDaemon.php:91]
[02-Feb-2015 09:35:42]   #3 PhutilDaemon::execute() called at [<phutil>/scripts/daemon/exec/exec_daemon.php:111]
:

Possible relevant: we had 3 messed up repos, and they were lacking credential instead of using 'wrong' credentials.

(Oh and we last rebased on upstream on 2015-01-07)

Found this, missing piece was clicking "Update Now" to queue an update message.