Accounts with old ssh keys added to them work perfectly work with diffusion repos.
Recently, been more than 2 weeks now, new users, when they added a ssh key to their account, it wouldn't work.
Went back and started troubleshooting with the methods listed in the Diffusion User Guide.
Starting the SSHD in debug mode and firing the test command `ssh -T -p 2222 git@xyz.net` results in the following:
```
debug2: load_server_config: filename /etc/ssh/sshd_config.phabricator
debug2: load_server_config: done config len = 315
debug2: parse_server_config: config /etc/ssh/sshd_config.phabricator len 315
debug3: /etc/ssh/sshd_config.phabricator:6 setting AuthorizedKeysCommand /usr/lib/phabricator-ssh-hook.sh
debug3: /etc/ssh/sshd_config.phabricator:7 setting AuthorizedKeysCommandUser git
debug3: /etc/ssh/sshd_config.phabricator:8 setting AllowUsers git
debug3: /etc/ssh/sshd_config.phabricator:13 setting Port 2222
debug3: /etc/ssh/sshd_config.phabricator:14 setting Protocol 2
debug3: /etc/ssh/sshd_config.phabricator:15 setting PermitRootLogin no
debug3: /etc/ssh/sshd_config.phabricator:16 setting AllowAgentForwarding no
debug3: /etc/ssh/sshd_config.phabricator:17 setting AllowTcpForwarding no
debug3: /etc/ssh/sshd_config.phabricator:18 setting PrintMotd no
debug3: /etc/ssh/sshd_config.phabricator:19 setting PrintLastLog no
debug3: /etc/ssh/sshd_config.phabricator:20 setting PasswordAuthentication no
debug3: /etc/ssh/sshd_config.phabricator:21 setting AuthorizedKeysFile none
debug3: /etc/ssh/sshd_config.phabricator:23 setting PidFile /var/run/sshd-phabricator.pid
debug1: sshd version OpenSSH_7.2, OpenSSL 1.0.2g 1 Mar 2016
debug1: private host key #0: ssh-rsa SHA256:i8qzUa0IflrvsBgc458IBuvBDm/I5tY9rKvvAjJTBvs
debug1: private host key #1: ssh-dss SHA256:SbHHo9yRCtkCxnmDwndd1dB/bEZhSXS2ZiiM52vn3Tg
debug1: private host key #2: ecdsa-sha2-nistp256 SHA256:PZFMO22a3/wJmzL5mdZqOVA16Cbuvk+R4hzHcIikYBU
debug1: private host key #3: ssh-ed25519 SHA256:Wl8zNfrPeKBTgeGonXBhH3NW4+3VMIlrwFnx8QC8poM
debug1: rexec_argv[0]='/usr/sbin/sshd'
debug1: rexec_argv[1]='-d'
debug1: rexec_argv[2]='-d'
debug1: rexec_argv[3]='-d'
debug1: rexec_argv[4]='-f'
debug1: rexec_argv[5]='/etc/ssh/sshd_config.phabricator'
debug3: oom_adjust_setup
debug1: Set /proc/self/oom_score_adj from 0 to -1000
debug2: fd 3 setting O_NONBLOCK
debug1: Bind to port 2222 on 0.0.0.0.
Server listening on 0.0.0.0 port 2222.
debug2: fd 4 setting O_NONBLOCK
debug3: sock_set_v6only: set socket 4 IPV6_V6ONLY
debug1: Bind to port 2222 on ::.
Server listening on :: port 2222.
debug3: fd 5 is not O_NONBLOCK
debug1: Server will not fork when running in debugging mode.
debug3: send_rexec_state: entering fd = 8 config len 315
debug3: ssh_msg_send: type 0
debug3: send_rexec_state: done
debug1: rexec start in 5 out 5 newsock 5 pipe -1 sock 8
debug1: inetd sockets after dupping: 3, 3
Connection from 114.143.206.68 port 35130 on 172.30.0.234 port 2222
debug1: Client protocol version 2.0; client software version OpenSSH_7.2p2 Ubuntu-4ubuntu2.2
debug1: match: OpenSSH_7.2p2 Ubuntu-4ubuntu2.2 pat OpenSSH* compat 0x04000000
debug1: Enabling compatibility mode for protocol 2.0
debug1: Local version string SSH-2.0-OpenSSH_7.2p2 Ubuntu-4ubuntu2.2
debug2: fd 3 setting O_NONBLOCK
debug3: ssh_sandbox_init: preparing seccomp filter sandbox
debug2: Network child is on pid 9762
debug3: preauth child monitor started
debug3: privsep user:group 110:65534 [preauth]
debug1: permanently_set_uid: 110/65534 [preauth]
debug3: ssh_sandbox_child: setting PR_SET_NO_NEW_PRIVS [preauth]
debug3: ssh_sandbox_child: attaching seccomp filter program [preauth]
debug3: list_hostkey_types: ssh-dss key not permitted by HostkeyAlgorithms [preauth]
debug1: list_hostkey_types: ssh-rsa,rsa-sha2-512,rsa-sha2-256,ecdsa-sha2-nistp256,ssh-ed25519 [preauth]
debug3: send packet: type 20 [preauth]
debug1: SSH2_MSG_KEXINIT sent [preauth]
debug3: receive packet: type 20 [preauth]
debug1: SSH2_MSG_KEXINIT received [preauth]
debug2: local server KEXINIT proposal [preauth]
debug2: KEX algorithms: curve25519-sha256@libssh.org,ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group14-sha1 [preauth]
debug2: host key algorithms: ssh-rsa,rsa-sha2-512,rsa-sha2-256,ecdsa-sha2-nistp256,ssh-ed25519 [preauth]
debug2: ciphers ctos: chacha20-poly1305@openssh.com,aes128-ctr,aes192-ctr,aes256-ctr,aes128-gcm@openssh.com,aes256-gcm@openssh.com [preauth]
debug2: ciphers stoc: chacha20-poly1305@openssh.com,aes128-ctr,aes192-ctr,aes256-ctr,aes128-gcm@openssh.com,aes256-gcm@openssh.com [preauth]
debug2: MACs ctos: umac-64-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-sha1-etm@openssh.com,umac-64@openssh.com,umac-128@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-sha1 [preauth]
debug2: MACs stoc: umac-64-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-sha1-etm@openssh.com,umac-64@openssh.com,umac-128@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-sha1 [preauth]
debug2: compression ctos: none,zlib@openssh.com [preauth]
debug2: compression stoc: none,zlib@openssh.com [preauth]
debug2: languages ctos: [preauth]
debug2: languages stoc: [preauth]
debug2: first_kex_follows 0 [preauth]
debug2: reserved 0 [preauth]
debug2: peer client KEXINIT proposal [preauth]
debug2: KEX algorithms: curve25519-sha256@libssh.org,ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group-exchange-sha1,diffie-hellman-group14-sha1,ext-info-c [preauth]
debug2: host key algorithms: ecdsa-sha2-nistp256-cert-v01@openssh.com,ecdsa-sha2-nistp384-cert-v01@openssh.com,ecdsa-sha2-nistp521-cert-v01@openssh.com,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521,ssh-ed25519-cert-v01@openssh.com,ssh-rsa-cert-v01@openssh.com,ssh-ed25519,rsa-sha2-512,rsa-sha2-256,ssh-rsa [preauth]
debug2: ciphers ctos: chacha20-poly1305@openssh.com,aes128-ctr,aes192-ctr,aes256-ctr,aes128-gcm@openssh.com,aes256-gcm@openssh.com,aes128-cbc,aes192-cbc,aes256-cbc,3des-cbc [preauth]
debug2: ciphers stoc: chacha20-poly1305@openssh.com,aes128-ctr,aes192-ctr,aes256-ctr,aes128-gcm@openssh.com,aes256-gcm@openssh.com,aes128-cbc,aes192-cbc,aes256-cbc,3des-cbc [preauth]
debug2: MACs ctos: umac-64-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-sha1-etm@openssh.com,umac-64@openssh.com,umac-128@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-sha1 [preauth]
debug2: MACs stoc: umac-64-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-sha1-etm@openssh.com,umac-64@openssh.com,umac-128@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-sha1 [preauth]
debug2: compression ctos: none,zlib@openssh.com,zlib [preauth]
debug2: compression stoc: none,zlib@openssh.com,zlib [preauth]
debug2: languages ctos: [preauth]
debug2: languages stoc: [preauth]
debug2: first_kex_follows 0 [preauth]
debug2: reserved 0 [preauth]
debug1: kex: algorithm: curve25519-sha256@libssh.org [preauth]
debug1: kex: host key algorithm: ecdsa-sha2-nistp256 [preauth]
debug1: kex: client->server cipher: chacha20-poly1305@openssh.com MAC: <implicit> compression: none [preauth]
debug1: kex: server->client cipher: chacha20-poly1305@openssh.com MAC: <implicit> compression: none [preauth]
debug1: expecting SSH2_MSG_KEX_ECDH_INIT [preauth]
debug3: receive packet: type 30 [preauth]
debug3: mm_key_sign entering [preauth]
debug3: mm_request_send entering: type 6 [preauth]
debug3: mm_key_sign: waiting for MONITOR_ANS_SIGN [preauth]
debug3: mm_request_receive_expect entering: type 7 [preauth]
debug3: mm_request_receive entering [preauth]
debug3: mm_request_receive entering
debug3: monitor_read: checking request 6
debug3: mm_answer_sign
debug3: mm_answer_sign: hostkey proof signature 0x55a12bb6daa0(100)
debug3: mm_request_send entering: type 7
debug2: monitor_read: 6 used once, disabling now
debug3: send packet: type 31 [preauth]
debug3: send packet: type 21 [preauth]
debug2: set_newkeys: mode 1 [preauth]
debug1: rekey after 134217728 blocks [preauth]
debug1: SSH2_MSG_NEWKEYS sent [preauth]
debug1: expecting SSH2_MSG_NEWKEYS [preauth]
debug3: send packet: type 7 [preauth]
debug3: receive packet: type 21 [preauth]
debug2: set_newkeys: mode 0 [preauth]
debug1: rekey after 134217728 blocks [preauth]
debug1: SSH2_MSG_NEWKEYS received [preauth]
debug1: KEX done [preauth]
debug3: receive packet: type 5 [preauth]
debug3: send packet: type 6 [preauth]
debug3: receive packet: type 50 [preauth]
debug1: userauth-request for user git service ssh-connection method none [preauth]
debug1: attempt 0 failures 0 [preauth]
debug3: mm_getpwnamallow entering [preauth]
debug3: mm_request_send entering: type 8 [preauth]
debug3: mm_getpwnamallow: waiting for MONITOR_ANS_PWNAM [preauth]
debug3: mm_request_receive_expect entering: type 9 [preauth]
debug3: mm_request_receive entering [preauth]
debug3: mm_request_receive entering
debug3: monitor_read: checking request 8
debug3: mm_answer_pwnamallow
debug2: parse_server_config: config reprocess config len 315
debug3: auth_shadow_acctexpired: today 17312 sp_expire -1 days left -17313
debug3: account expiration disabled
debug3: mm_answer_pwnamallow: sending MONITOR_ANS_PWNAM: 1
debug3: mm_request_send entering: type 9
debug2: monitor_read: 8 used once, disabling now
debug2: input_userauth_request: setting up authctxt for git [preauth]
debug3: mm_inform_authserv entering [preauth]
debug3: mm_request_send entering: type 4 [preauth]
debug2: input_userauth_request: try method none [preauth]
debug3: userauth_finish: failure partial=0 next methods="publickey,keyboard-interactive" [preauth]
debug3: send packet: type 51 [preauth]
debug3: mm_request_receive entering
debug3: monitor_read: checking request 4
debug3: mm_answer_authserv: service=ssh-connection, style=, role=
debug2: monitor_read: 4 used once, disabling now
debug3: receive packet: type 50 [preauth]
debug1: userauth-request for user git service ssh-connection method publickey [preauth]
debug1: attempt 1 failures 0 [preauth]
debug2: input_userauth_request: try method publickey [preauth]
debug1: userauth_pubkey: test whether pkalg/pkblob are acceptable for RSA SHA256:ZY1wvU7Ig7L/BG+GcSr0zXtsejfODnJASHaQUuoKPhY [preauth]
debug3: mm_key_allowed entering [preauth]
debug3: mm_request_send entering: type 22 [preauth]
debug3: mm_key_allowed: waiting for MONITOR_ANS_KEYALLOWED [preauth]
debug3: mm_request_receive_expect entering: type 23 [preauth]
debug3: mm_request_receive entering [preauth]
debug3: mm_request_receive entering
debug3: monitor_read: checking request 22
debug3: mm_answer_keyallowed entering
debug3: mm_answer_keyallowed: key_from_blob: 0x55a12bb6d9c0
debug3: subprocess: AuthorizedKeysCommand command "/usr/lib/phabricator-ssh-hook.sh git" running as git
debug1: temporarily_use_uid: 1002/1002 (e=0/0)
debug1: restore_uid: 0/0
debug3: subprocess: AuthorizedKeysCommand pid 9763
debug1: temporarily_use_uid: 1002/1002 (e=0/0)
debug2: user_key_allowed: check options: 'command="'/opt/phabricator/phabricator/bin/ssh-exec' '--phabricator-ssh-user' 'exp10r3r' '--phabricator-ssh-key' '196'",no-port-forwarding,no-X11-forwarding,no-agent-forwarding,no-pty ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDEUI7zWdi2VvsGXCM4gVGIKY3NcDG8hm5egklTYEuXzZjaBfM/R6W+gZrGqpKZO+z2JtRShK59w+kCvwH/X1XbJiPfjdN4DZrmq1xwaUWzf6Zh6Gf8k9UJzUF+nGQHUmoOsBrWlVSVtmqzhnj+NZtC5wYIWUmnfdFFeRcrWtLqO706FvOTEF1iI5FTQNUhjwIL8fpW1mlCl77IhZziZuHvI/UA/yhfouU/r46VDsAYzgItUS5Im+0H1jDr3Y4MAVeUHy0agqzYiawj1dtMSRrxYwLJvxto2hL8cntHg3OJzZQJ/PHUX1kZp6DFT2eejqCHnnXn/0xGekdMKEMXcDaMCjT1nFoGsqP67gJFfiBKKY1D/02yMFejMSNCIB86+2u77KPHhwAmxlvVpDW2edWvrl1/r8OmxkSrTBz8kVIQmVvvqZbsA51BvLEcDlvmHkxYUojHoMqBRnpaxBqXbz8/1cqA1+oVrEAD6P3K5XYIkMRc7uexf5GXd51ShutGdCV7h9dnZNAe4BYPcd0oQzc7VfcAa52rlLVUxTKGXerjL0aj5Tf1frmQSY2rN1ugVYibuF8rMU7/xbV+0ueeH3a+odxp4dqNoTVfJTtVMLBwOweGjYHy1uvn7PncbyG8AOTMCr9KkR8CdZDjrgIQSSW+CK53Sj/AZUsDI0ZU4jsLdw==
'
debug1: matching key found: file /usr/lib/phabricator-ssh-hook.sh, line 1 RSA SHA256:ZY1wvU7Ig7L/BG+GcSr0zXtsejfODnJASHaQUuoKPhY
```
And its just stuck there.
When I run that same command with an older ssh key from the same account, it goes ahead and works perfectly fine:
```
Environment:
USER=git
LOGNAME=git
HOME=/home/git
PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games
MAIL=/var/mail/git
SHELL=/bin/bash
SSH_CLIENT=114.143.206.68 35124 2222
SSH_CONNECTION=114.143.206.68 35124 172.30.0.234 2222
phabricator-ssh-exec: Welcome to Phabricator.
You are logged in as chachaji.
You haven't specified a command to run. This means you're requesting an interactive shell, but Phabricator does not provide an interactive shell over SSH.
Usually, you should run a command like `git clone` or `hg push` rather than connecting directly with SSH.
Supported commands are: conduit, git-lfs-authenticate, git-receive-pack, git-upload-pack, hg, svnserve.
```
Environment Info:
Ubuntu 16.04 on AWS
MySQL 5.7 RDS
Php7.1 from ondrej PPA
phabricator
4d2c7e4d3d5d61e34b7e2af3df0e901d89d29433 (Sat, May 20) (branched from c6a7bcfe89287c2be2baa290def7c2f4e365396d on origin)
arcanist
21fe07925b07cab0f73ec824fe8ce02f53b24fe0 (Sat, May 20) (branched from 129d51fa0936c9bae48fadf3a3f39e26d69d24da on origin)
phutil
d02cc05931b02c684d4c729510090591ca45f951 (Sat, Apr 29) (branched from a900d7b63e954e221efe140f0f33d3d701524aae on origin)
**Reproduction Steps**
Generate a new SSH key from terminal via `ssh-keygen -t rsa -b 4096 -c test`
Upload this new key to my account
Test with git clone
(NOTE) No, I haven't taken a new image and tested this out, since this has been happening on production server