Page MenuHomePhabricator

ssh-auth hangs on FreeBSD
Closed, ResolvedPublic

Description

Phabricator version: 20161003
PHP Version: 5.6
SSH Version: 7.2p2

ssh-auth sometimes seems to hang. The solution is to print the keys one by one instead of flushing the whole key list at once.

Related freebsd bug report: http://lists.mindrot.org/pipermail/openssh-bugs/2015-November/015714.html

Patch:

--- ssh-auth.old	2016-11-07 13:58:40.003632883 +0530
+++ ssh-auth	2016-11-07 13:59:03.217630698 +0530
@@ -70,8 +70,10 @@
   );
   $options = implode(',', $options);
 
-  $lines[] = $options.' '.$type.' '.$key."\n";
+  //$lines[] = $options.' '.$type.' '.$key."\n";
+  $line = $options.' '.$type.' '.$key."\n";
+  echo $line;
 }
 
-echo implode('', $lines);
+//echo implode('', $lines);
 exit(0);

Event Timeline

I'd prefer not to produce any output until we've generated everything, so we don't end up in a situation where we output half of the file and then fail if we can help it. Additionally, this script changed in D16744, a few weeks ahead of your version.

Does this alternate patch resolve the issue for you?

diff --git a/scripts/ssh/ssh-auth.php b/scripts/ssh/ssh-auth.php
index b259056..03a3512 100755
--- a/scripts/ssh/ssh-auth.php
+++ b/scripts/ssh/ssh-auth.php
@@ -83,5 +83,12 @@ if ($authfile === null) {
   $cache->setKey($authfile_key, $authfile, $ttl);
 }
 
-echo $authfile;
+$parts = str_split($authfile, 512);
+foreach ($parts as $part) {
+  echo $part;
+
+  // If just str_split() on its own isn't good enough, try adding flush()?
+  // flush();
+}
+
 exit(0);

(mindrot.org is currently inaccessible for me so I wasn't able to read the details you linked, although I imagine they aren't too relevant to how we handle things.)

$ curl -v http://lists.mindrot.org/pipermail/openssh-bugs/2015-November/015714.html
*   Trying 130.102.96.4...
* connect to 130.102.96.4 port 80 failed: Operation timed out
* Failed to connect to lists.mindrot.org port 80: Operation timed out
* Closing connection 0
curl: (7) Failed to connect to lists.mindrot.org port 80: Operation timed out

I'll have to update my instance first to test your patch, since it seems to be for newer version as you said earlier. I installed this from FreeBSD ports system, have to first convert it to normal git based install.

Here's Google webacache of the mailing list: http://webcache.googleusercontent.com/search?q=cache%3Ahttp%3A%2F%2Flists.mindrot.org%2Fpipermail%2Fopenssh-bugs%2F2015-November%2F015714.html

I upgraded my installation to

commit ad65d933fa75a2c417ba23f3684f0109de0f6ace
Merge: cfcc3b8 f0bf041
Author: epriestley <git@epriestley.com>
Date:   Sat Dec 3 02:39:57 2016 -0800

    (stable) Promote 2016 Week 49

And even now it doesn't work by default. The str_split thing works though.

diff --git a/scripts/ssh/ssh-auth.php b/scripts/ssh/ssh-auth.php
index b259056..ba614c1 100755
--- a/scripts/ssh/ssh-auth.php
+++ b/scripts/ssh/ssh-auth.php
@@ -83,5 +83,9 @@ if ($authfile === null) {
   $cache->setKey($authfile_key, $authfile, $ttl);
 }
 
-echo $authfile;
+//echo $authfile;
+$parts = str_split($authfile, 512);
+foreach($parts as $part) {
+       echo $part;
+}
 exit(0);
epriestley claimed this task.

From https://bugzilla.mindrot.org/show_bug.cgi?id=2496, this is apparently fixed in the OpenSSH upstream (https://github.com/openssh/openssh-portable/commit/ddd3d34e5c7979ca6f4a3a98a7d219a4ed3d98c2). Since this is a bug in OpenSSH, an upstream fix is available, and the impact is fairly narrow, I don't plan to work around it in Phabricator.

The workaround does not work for me on Debian 9.5 with OpenSSH 7.4p1. Actually I am surprised this worked for anyone, since the issue appears to be that the buffer of the pipe between ssh-auth's stdout and the parent's stdin is being filled, hence ssh-auth blocking at some point. It should not matter how fast these buffers are being filled...

Some more information on the OpenSSH upstream issue and the status in the Linux distros:

Building a patched OpenSSH using the patch for the upstream bug report solves the issue for us: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=905226#17