Page MenuHomePhabricator
Create Task
Maniphest T10982

Harbormaster builds fail to allocate resource
Closed, InvalidPublic
Actions

Description

Current versions:

phabricator
    3aed39b8b072c8bfa89b3f45183dfa126600ff1d (Wed, May 18) 
arcanist
    2234c8cacc21ce61c9c10e8e5918b6a63cc38fc8 (Mon, May 16) 
phutil
    b25e0477b280ca3e8345bb97cd55e95bcb5023ec (Wed, May 11)

Following the instructions at https://secure.phabricator.com/T10246#163438 to run a simple git show manually for a recent commit, build attempts to activate a lease which fails, continuously, until aborted.

The relevant error seems to be "fatal: could not read Username for 'http://phabricator.<HOSTNAME_MASKED>': No such device or address"

I can log in to the build server using the private key, in fact the builder creates a workingcopy, but fails to clone into it.

builder@builder:~$ ls -la /var/drydock/workingcopy-92/repo/
total 8
drwxrwxr-x 2 builder builder 4096 May 18 16:02 .
drwxrwxr-x 3 builder builder 4096 May 18 16:02 ..
builder@builder:~$

Build Log:

exception 'PhabricatorWorkerPermanentFailureException' with message 'Lease "PHID-DRYL-2pvhwujlbsollyzaeipd" never activated.' in /usr/local/phabricator/phabricator/src/applications/harbormaster/step/HarbormasterLeaseWorkingCopyBuildStepImplementation.php:91
Stack trace:
#0 /usr/local/phabricator/phabricator/src/applications/harbormaster/worker/HarbormasterTargetWorker.php(64): HarbormasterLeaseWorkingCopyBuildStepImplementation->execute(Object(HarbormasterBuild), Object(HarbormasterBuildTarget))
#1 /usr/local/phabricator/phabricator/src/infrastructure/daemon/workers/PhabricatorWorker.php(122): HarbormasterTargetWorker->doWork()
#2 /usr/local/phabricator/phabricator/src/infrastructure/daemon/workers/storage/PhabricatorWorkerActiveTask.php(171): PhabricatorWorker->executeTask()
#3 /usr/local/phabricator/phabricator/src/infrastructure/daemon/workers/PhabricatorTaskmasterDaemon.php(22): PhabricatorWorkerActiveTask->executeTask()
#4 /usr/local/phabricator/libphutil/src/daemon/PhutilDaemon.php(183): PhabricatorTaskmasterDaemon->run()
#5 /usr/local/phabricator/libphutil/scripts/daemon/exec/exec_daemon.php(125): PhutilDaemon->execute()
#6 {main}

Blueprint log:

Resource activation failed: [CommandException] Command failed with error #128! COMMAND ssh '-o' 'LogLevel=quiet' '-o' 'StrictHostKeyChecking=no' '-o' 'UserKnownHostsFile=/dev/null' '-o' 'BatchMode=yes' -l 'xxxxx' -p '22' -i 'xxxxx' '<BUILDER_IP_MASKED>' -- 'git clone -- '\''http://phabricator.<HOSTNAME_MASKED>/diffusion/48/testrepo.git'\'' '\''/var/drydock/workingcopy-92/repo/testrepo/'\''' STDOUT (empty) STDERR Cloning into '/var/drydock/workingcopy-92/repo/testrepo'... fatal: could not read Username for 'http://phabricator.<HOSTNAME_MASKED>': No such device or address

Revisions and Commits

rPHU libphutil
D16181rPHU5796beffff16 Censor credentials with "********" instead of "xxxxx"

Related Objects

Event Timeline

fanis created this task.May 18 2016, 1:05 PM
chad added a project: Harbormaster.May 19 2016, 4:49 PM
hach-que added subscribers: chad, epriestley, hach-que.May 21 2016, 6:16 AM

This sounds like an issue with either git clone or the SSH client installed on the build machine itself, and not Phabricator. I don't believe upstream offers support for configuring build agents (they don't offer support for envionment issues on machines that host Phabricator, let alone environment issues on machines that Phabricator uses for builds).

You can potentially get more information by performing the command that Phabricator uses to connect to the machine, but manually and providing -vvv to get more information about why the command is failing.

I'll leave it to @chad or @epriestley to actually close this task.

ilendir added a subscriber: ilendir.Jun 24 2016, 12:19 PM

I have a somewhat similar problem when I try to start a working copy job:

Resource activation failed: [CommandException] Command failed with error #255! COMMAND ssh '-o' 'LogLevel=quiet' '-o' 'StrictHostKeyChecking=no' '-o' 'UserKnownHostsFile=/dev/null' '-o' 'BatchMode=yes' -l 'xxxxx' -p '22' -i 'xxxxx' '10.10.30.120' -- 'git clone -- '\''ssh://xxxxx@phabricator-eng.{hostname-masked}:2222/diffusion/BAR/b.git'\'' '\''/var/drydock/workingcopy-532/repo/b/'\''' STDOUT (empty) STDERR (empty)

I double checked the documentation that I did everything according to what is written. Manually cloning works fine, ssh on the machine works fine.
not sure how i can debug it to find out if my configuration is off, or if phabricator is not fully implemented on that regards.
I tried manually running the command, but 'xxxxx' is unknown (who would have guessed? :P) and I'm not sure where this information comes from or if it's just the Phabricator masking some information.

Any help or pointers are welcome.

ilendir mentioned this in Z1336: General Chat.Jun 27 2016, 8:31 AM
asherkin added a subscriber: asherkin.Jun 27 2016, 11:29 AM

the -l xxxxx is your configured SSH user, the -i xxxxx is the path to your configured SSH private key (it's written out to a temporary file).

ilendir added a comment.Jun 27 2016, 12:44 PM

So, the information is just masked but really present in the real command call? The ssh man-page states what the options before the xxxxxs mean, I'm just not sure if the call is right when the information is masked. E.g. the clone call...does phabricator call with git@phabricator... or with the username provided for the ssh connection. anyway. Is there a log where the real command is written? Why is it even masked here?

epriestley added a comment.Jun 27 2016, 1:51 PM

If we masked with ***** instead of xxxxx, do you think that would have been more intuitive?

ilendir added a comment.Jun 27 2016, 2:48 PM

I would say, definitively. 'xxxxx' looks more like a placeholder for me, but that might be a personal preference.

epriestley added a revision: D16181: Censor credentials with "********" instead of "xxxxx".Jun 27 2016, 2:55 PM
ilendir added a comment.Jun 27 2016, 3:10 PM

Nevertheless, I'm still not sure how to debug this error message / my configuration.
Is there a way to test/run the Drydock Host Blueprint and get more information than the command failed?

A a second note: It would be nice to give drydock a timeout for acquiring a resource (e.g. try connecting to a host/service for 15 min and then fail) and not run for a few days. Or at least add a way in a drydock blueprint to reliably cancel the acquisition attempts for resources.
Not sure if I missed something, but releasing leases seems to be unreliable for canceling the execution of a blueprint.

epriestley added a commit: rPHU5796beffff16: Censor credentials with "********" instead of "xxxxx".Jun 27 2016, 3:22 PM
epriestley closed this task as Invalid.Jun 27 2016, 3:32 PM

This isn't an actionable bug report because it doesn't include reproduction instructions. Reports MUST include reproduction instructions; see Contributing Bug Reports and Providing Reproduction Steps.

We don't provide general setup and configuration support because issues in this vein often require a tremendous amount of time and energy to resolve and boil down to environmental issues that help exactly one user. See Support Resources for what we do provide, and for other ways to get help.

You can likely file the timeout issue as a legitimate, actionable report since it should be reasonable to provide reproduction steps. It probably falls under T8153, but having it filed as a specific test case will help make sure it gets resolved as part of that task (that task is prioritized and should happen soon).

There should be no distinction between Drydock running git clone and you running git clone (while logged in as the same user), but it sounds like one works and the other doesn't. There is no secret hidden magic, we're just SSH'ing to the host and running git clone. git clone works fine in the upstream environments and other user environments, so this strongly suggests there is some sort of environmental problem in your configuration. You need to identify it and reduce it to repeatable reproduction steps before we can move forward.

ilendir added a comment.Jul 25 2016, 12:52 PM

@epriestley I was trying to recreate my problem I had a month ago and write a task (bug report) for it and wasn't able to reproduce the issue of Drydock trying to allocate resources indefinitely. Have you already added a timeout to Drydock leases?

Phabricator · Built by Phacility