Accounts with old ssh keys added to them work perfectly work with diffusion repos.
Recently, been more than 2 weeks now, new users, when they added a ssh key to their account, it wouldn't work.
Went back and started troubleshooting with the methods listed in the Diffusion User Guide.
Starting the SSHD in debug mode and firing the test command ssh -T -p 2222 git@xyz.net results in the following:
debug2: load_server_config: filename /etc/ssh/sshd_config.phabricator debug2: load_server_config: done config len = 315 debug2: parse_server_config: config /etc/ssh/sshd_config.phabricator len 315 debug3: /etc/ssh/sshd_config.phabricator:6 setting AuthorizedKeysCommand /usr/lib/phabricator-ssh-hook.sh debug3: /etc/ssh/sshd_config.phabricator:7 setting AuthorizedKeysCommandUser git debug3: /etc/ssh/sshd_config.phabricator:8 setting AllowUsers git debug3: /etc/ssh/sshd_config.phabricator:13 setting Port 2222 debug3: /etc/ssh/sshd_config.phabricator:14 setting Protocol 2 debug3: /etc/ssh/sshd_config.phabricator:15 setting PermitRootLogin no debug3: /etc/ssh/sshd_config.phabricator:16 setting AllowAgentForwarding no debug3: /etc/ssh/sshd_config.phabricator:17 setting AllowTcpForwarding no debug3: /etc/ssh/sshd_config.phabricator:18 setting PrintMotd no debug3: /etc/ssh/sshd_config.phabricator:19 setting PrintLastLog no debug3: /etc/ssh/sshd_config.phabricator:20 setting PasswordAuthentication no debug3: /etc/ssh/sshd_config.phabricator:21 setting AuthorizedKeysFile none debug3: /etc/ssh/sshd_config.phabricator:23 setting PidFile /var/run/sshd-phabricator.pid debug1: sshd version OpenSSH_7.2, OpenSSL 1.0.2g 1 Mar 2016 debug1: private host key #0: ssh-rsa SHA256:i8qzUa0IflrvsBgc458IBuvBDm/I5tY9rKvvAjJTBvs debug1: private host key #1: ssh-dss SHA256:SbHHo9yRCtkCxnmDwndd1dB/bEZhSXS2ZiiM52vn3Tg debug1: private host key #2: ecdsa-sha2-nistp256 SHA256:PZFMO22a3/wJmzL5mdZqOVA16Cbuvk+R4hzHcIikYBU debug1: private host key #3: ssh-ed25519 SHA256:Wl8zNfrPeKBTgeGonXBhH3NW4+3VMIlrwFnx8QC8poM debug1: rexec_argv[0]='/usr/sbin/sshd' debug1: rexec_argv[1]='-d' debug1: rexec_argv[2]='-d' debug1: rexec_argv[3]='-d' debug1: rexec_argv[4]='-f' debug1: rexec_argv[5]='/etc/ssh/sshd_config.phabricator' debug3: oom_adjust_setup debug1: Set /proc/self/oom_score_adj from 0 to -1000 debug2: fd 3 setting O_NONBLOCK debug1: Bind to port 2222 on 0.0.0.0. Server listening on 0.0.0.0 port 2222. debug2: fd 4 setting O_NONBLOCK debug3: sock_set_v6only: set socket 4 IPV6_V6ONLY debug1: Bind to port 2222 on ::. Server listening on :: port 2222. debug3: fd 5 is not O_NONBLOCK debug1: Server will not fork when running in debugging mode. debug3: send_rexec_state: entering fd = 8 config len 315 debug3: ssh_msg_send: type 0 debug3: send_rexec_state: done debug1: rexec start in 5 out 5 newsock 5 pipe -1 sock 8 debug1: inetd sockets after dupping: 3, 3 Connection from 114.143.206.68 port 35130 on 172.30.0.234 port 2222 debug1: Client protocol version 2.0; client software version OpenSSH_7.2p2 Ubuntu-4ubuntu2.2 debug1: match: OpenSSH_7.2p2 Ubuntu-4ubuntu2.2 pat OpenSSH* compat 0x04000000 debug1: Enabling compatibility mode for protocol 2.0 debug1: Local version string SSH-2.0-OpenSSH_7.2p2 Ubuntu-4ubuntu2.2 debug2: fd 3 setting O_NONBLOCK debug3: ssh_sandbox_init: preparing seccomp filter sandbox debug2: Network child is on pid 9762 debug3: preauth child monitor started debug3: privsep user:group 110:65534 [preauth] debug1: permanently_set_uid: 110/65534 [preauth] debug3: ssh_sandbox_child: setting PR_SET_NO_NEW_PRIVS [preauth] debug3: ssh_sandbox_child: attaching seccomp filter program [preauth] debug3: list_hostkey_types: ssh-dss key not permitted by HostkeyAlgorithms [preauth] debug1: list_hostkey_types: ssh-rsa,rsa-sha2-512,rsa-sha2-256,ecdsa-sha2-nistp256,ssh-ed25519 [preauth] debug3: send packet: type 20 [preauth] debug1: SSH2_MSG_KEXINIT sent [preauth] debug3: receive packet: type 20 [preauth] debug1: SSH2_MSG_KEXINIT received [preauth] debug2: local server KEXINIT proposal [preauth] debug2: KEX algorithms: curve25519-sha256@libssh.org,ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group14-sha1 [preauth] debug2: host key algorithms: ssh-rsa,rsa-sha2-512,rsa-sha2-256,ecdsa-sha2-nistp256,ssh-ed25519 [preauth] debug2: ciphers ctos: chacha20-poly1305@openssh.com,aes128-ctr,aes192-ctr,aes256-ctr,aes128-gcm@openssh.com,aes256-gcm@openssh.com [preauth] debug2: ciphers stoc: chacha20-poly1305@openssh.com,aes128-ctr,aes192-ctr,aes256-ctr,aes128-gcm@openssh.com,aes256-gcm@openssh.com [preauth] debug2: MACs ctos: umac-64-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-sha1-etm@openssh.com,umac-64@openssh.com,umac-128@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-sha1 [preauth] debug2: MACs stoc: umac-64-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-sha1-etm@openssh.com,umac-64@openssh.com,umac-128@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-sha1 [preauth] debug2: compression ctos: none,zlib@openssh.com [preauth] debug2: compression stoc: none,zlib@openssh.com [preauth] debug2: languages ctos: [preauth] debug2: languages stoc: [preauth] debug2: first_kex_follows 0 [preauth] debug2: reserved 0 [preauth] debug2: peer client KEXINIT proposal [preauth] debug2: KEX algorithms: curve25519-sha256@libssh.org,ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group-exchange-sha1,diffie-hellman-group14-sha1,ext-info-c [preauth] debug2: host key algorithms: ecdsa-sha2-nistp256-cert-v01@openssh.com,ecdsa-sha2-nistp384-cert-v01@openssh.com,ecdsa-sha2-nistp521-cert-v01@openssh.com,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521,ssh-ed25519-cert-v01@openssh.com,ssh-rsa-cert-v01@openssh.com,ssh-ed25519,rsa-sha2-512,rsa-sha2-256,ssh-rsa [preauth] debug2: ciphers ctos: chacha20-poly1305@openssh.com,aes128-ctr,aes192-ctr,aes256-ctr,aes128-gcm@openssh.com,aes256-gcm@openssh.com,aes128-cbc,aes192-cbc,aes256-cbc,3des-cbc [preauth] debug2: ciphers stoc: chacha20-poly1305@openssh.com,aes128-ctr,aes192-ctr,aes256-ctr,aes128-gcm@openssh.com,aes256-gcm@openssh.com,aes128-cbc,aes192-cbc,aes256-cbc,3des-cbc [preauth] debug2: MACs ctos: umac-64-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-sha1-etm@openssh.com,umac-64@openssh.com,umac-128@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-sha1 [preauth] debug2: MACs stoc: umac-64-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-sha1-etm@openssh.com,umac-64@openssh.com,umac-128@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-sha1 [preauth] debug2: compression ctos: none,zlib@openssh.com,zlib [preauth] debug2: compression stoc: none,zlib@openssh.com,zlib [preauth] debug2: languages ctos: [preauth] debug2: languages stoc: [preauth] debug2: first_kex_follows 0 [preauth] debug2: reserved 0 [preauth] debug1: kex: algorithm: curve25519-sha256@libssh.org [preauth] debug1: kex: host key algorithm: ecdsa-sha2-nistp256 [preauth] debug1: kex: client->server cipher: chacha20-poly1305@openssh.com MAC: <implicit> compression: none [preauth] debug1: kex: server->client cipher: chacha20-poly1305@openssh.com MAC: <implicit> compression: none [preauth] debug1: expecting SSH2_MSG_KEX_ECDH_INIT [preauth] debug3: receive packet: type 30 [preauth] debug3: mm_key_sign entering [preauth] debug3: mm_request_send entering: type 6 [preauth] debug3: mm_key_sign: waiting for MONITOR_ANS_SIGN [preauth] debug3: mm_request_receive_expect entering: type 7 [preauth] debug3: mm_request_receive entering [preauth] debug3: mm_request_receive entering debug3: monitor_read: checking request 6 debug3: mm_answer_sign debug3: mm_answer_sign: hostkey proof signature 0x55a12bb6daa0(100) debug3: mm_request_send entering: type 7 debug2: monitor_read: 6 used once, disabling now debug3: send packet: type 31 [preauth] debug3: send packet: type 21 [preauth] debug2: set_newkeys: mode 1 [preauth] debug1: rekey after 134217728 blocks [preauth] debug1: SSH2_MSG_NEWKEYS sent [preauth] debug1: expecting SSH2_MSG_NEWKEYS [preauth] debug3: send packet: type 7 [preauth] debug3: receive packet: type 21 [preauth] debug2: set_newkeys: mode 0 [preauth] debug1: rekey after 134217728 blocks [preauth] debug1: SSH2_MSG_NEWKEYS received [preauth] debug1: KEX done [preauth] debug3: receive packet: type 5 [preauth] debug3: send packet: type 6 [preauth] debug3: receive packet: type 50 [preauth] debug1: userauth-request for user git service ssh-connection method none [preauth] debug1: attempt 0 failures 0 [preauth] debug3: mm_getpwnamallow entering [preauth] debug3: mm_request_send entering: type 8 [preauth] debug3: mm_getpwnamallow: waiting for MONITOR_ANS_PWNAM [preauth] debug3: mm_request_receive_expect entering: type 9 [preauth] debug3: mm_request_receive entering [preauth] debug3: mm_request_receive entering debug3: monitor_read: checking request 8 debug3: mm_answer_pwnamallow debug2: parse_server_config: config reprocess config len 315 debug3: auth_shadow_acctexpired: today 17312 sp_expire -1 days left -17313 debug3: account expiration disabled debug3: mm_answer_pwnamallow: sending MONITOR_ANS_PWNAM: 1 debug3: mm_request_send entering: type 9 debug2: monitor_read: 8 used once, disabling now debug2: input_userauth_request: setting up authctxt for git [preauth] debug3: mm_inform_authserv entering [preauth] debug3: mm_request_send entering: type 4 [preauth] debug2: input_userauth_request: try method none [preauth] debug3: userauth_finish: failure partial=0 next methods="publickey,keyboard-interactive" [preauth] debug3: send packet: type 51 [preauth] debug3: mm_request_receive entering debug3: monitor_read: checking request 4 debug3: mm_answer_authserv: service=ssh-connection, style=, role= debug2: monitor_read: 4 used once, disabling now debug3: receive packet: type 50 [preauth] debug1: userauth-request for user git service ssh-connection method publickey [preauth] debug1: attempt 1 failures 0 [preauth] debug2: input_userauth_request: try method publickey [preauth] debug1: userauth_pubkey: test whether pkalg/pkblob are acceptable for RSA SHA256:ZY1wvU7Ig7L/BG+GcSr0zXtsejfODnJASHaQUuoKPhY [preauth] debug3: mm_key_allowed entering [preauth] debug3: mm_request_send entering: type 22 [preauth] debug3: mm_key_allowed: waiting for MONITOR_ANS_KEYALLOWED [preauth] debug3: mm_request_receive_expect entering: type 23 [preauth] debug3: mm_request_receive entering [preauth] debug3: mm_request_receive entering debug3: monitor_read: checking request 22 debug3: mm_answer_keyallowed entering debug3: mm_answer_keyallowed: key_from_blob: 0x55a12bb6d9c0 debug3: subprocess: AuthorizedKeysCommand command "/usr/lib/phabricator-ssh-hook.sh git" running as git debug1: temporarily_use_uid: 1002/1002 (e=0/0) debug1: restore_uid: 0/0 debug3: subprocess: AuthorizedKeysCommand pid 9763 debug1: temporarily_use_uid: 1002/1002 (e=0/0) debug2: user_key_allowed: check options: 'command="'/opt/phabricator/phabricator/bin/ssh-exec' '--phabricator-ssh-user' 'exp10r3r' '--phabricator-ssh-key' '196'",no-port-forwarding,no-X11-forwarding,no-agent-forwarding,no-pty ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDEUI7zWdi2VvsGXCM4gVGIKY3NcDG8hm5egklTYEuXzZjaBfM/R6W+gZrGqpKZO+z2JtRShK59w+kCvwH/X1XbJiPfjdN4DZrmq1xwaUWzf6Zh6Gf8k9UJzUF+nGQHUmoOsBrWlVSVtmqzhnj+NZtC5wYIWUmnfdFFeRcrWtLqO706FvOTEF1iI5FTQNUhjwIL8fpW1mlCl77IhZziZuHvI/UA/yhfouU/r46VDsAYzgItUS5Im+0H1jDr3Y4MAVeUHy0agqzYiawj1dtMSRrxYwLJvxto2hL8cntHg3OJzZQJ/PHUX1kZp6DFT2eejqCHnnXn/0xGekdMKEMXcDaMCjT1nFoGsqP67gJFfiBKKY1D/02yMFejMSNCIB86+2u77KPHhwAmxlvVpDW2edWvrl1/r8OmxkSrTBz8kVIQmVvvqZbsA51BvLEcDlvmHkxYUojHoMqBRnpaxBqXbz8/1cqA1+oVrEAD6P3K5XYIkMRc7uexf5GXd51ShutGdCV7h9dnZNAe4BYPcd0oQzc7VfcAa52rlLVUxTKGXerjL0aj5Tf1frmQSY2rN1ugVYibuF8rMU7/xbV+0ueeH3a+odxp4dqNoTVfJTtVMLBwOweGjYHy1uvn7PncbyG8AOTMCr9KkR8CdZDjrgIQSSW+CK53Sj/AZUsDI0ZU4jsLdw== ' debug1: matching key found: file /usr/lib/phabricator-ssh-hook.sh, line 1 RSA SHA256:ZY1wvU7Ig7L/BG+GcSr0zXtsejfODnJASHaQUuoKPhY
And its just stuck there.
When I run that same command with an older ssh key from the same account, it goes ahead and works perfectly fine:
Environment: USER=git LOGNAME=git HOME=/home/git PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games MAIL=/var/mail/git SHELL=/bin/bash SSH_CLIENT=114.143.206.68 35124 2222 SSH_CONNECTION=114.143.206.68 35124 172.30.0.234 2222 phabricator-ssh-exec: Welcome to Phabricator. You are logged in as chachaji. You haven't specified a command to run. This means you're requesting an interactive shell, but Phabricator does not provide an interactive shell over SSH. Usually, you should run a command like `git clone` or `hg push` rather than connecting directly with SSH. Supported commands are: conduit, git-lfs-authenticate, git-receive-pack, git-upload-pack, hg, svnserve.
Environment Info:
Ubuntu 16.04 on AWS
MySQL 5.7 RDS
Php7.1 from ondrej PPA
phabricator
4d2c7e4d3d5d61e34b7e2af3df0e901d89d29433 (Sat, May 20) (branched from c6a7bcfe89287c2be2baa290def7c2f4e365396d on origin)
arcanist
21fe07925b07cab0f73ec824fe8ce02f53b24fe0 (Sat, May 20) (branched from 129d51fa0936c9bae48fadf3a3f39e26d69d24da on origin)
phutil
d02cc05931b02c684d4c729510090591ca45f951 (Sat, Apr 29) (branched from a900d7b63e954e221efe140f0f33d3d701524aae on origin)
Reproduction Steps
Generate a new SSH key from terminal via ssh-keygen -t rsa -b 4096 -c test
Upload this new key to my account
Test with git clone
No, I haven't taken a new image and tested this out, since this has been happening on production server