sshd connection reset somewhere during KEX phase

[prof]

Hi, sshd does not accept connections, even from the local system. Once the computer was powered up last time months ago, it worked fine. Unfortunately, I cannot remember for sure if there ran package update or it happened out of blue. sshd_config was not touched since the last time sshd worked, I started playing with config only after sshd already broke apart.

arch: x86-32bit subarch: atom_32

OpenSSH_8.4p1, OpenSSL 1.1.1l  24 Aug 2021

Any ssh attempt drops connection immediately. 

Server command and log:

/usr/sbin/sshd -dddDe -o PidFile=/run/sshd.pid

Quote

debug2: load_server_config: filename /etc/ssh/sshd_config
debug2: load_server_config: done config len = 553
debug2: parse_server_config_depth: config /etc/ssh/sshd_config len 553
debug3: /etc/ssh/sshd_config:15 setting AddressFamily inet
debug3: /etc/ssh/sshd_config:21 setting HostKey /etc/ssh/ssh_host_ed25519_key
debug3: /etc/ssh/sshd_config:26 setting RekeyLimit default none
debug3: /etc/ssh/sshd_config:40 setting PubkeyAuthentication no
debug3: /etc/ssh/sshd_config:52 setting HostbasedAuthentication no
debug3: /etc/ssh/sshd_config:60 setting PasswordAuthentication yes
debug3: /etc/ssh/sshd_config:65 setting ChallengeResponseAuthentication no
debug3: /etc/ssh/sshd_config:86 setting UsePAM no
debug3: /etc/ssh/sshd_config:95 setting PrintMotd no
debug3: /etc/ssh/sshd_config:96 setting PrintLastLog no
debug3: /etc/ssh/sshd_config:123 setting AcceptEnv LANG LC_ALL LC_COLLATE LC_CTYPE LC_MESSAGES LC_MONETARY LC_NUMERIC LC_TIME LANGUAGE LC_ADDRESS LC_IDENTIFICATION LC_MEASUREMENT LC_NAME LC_PAPER LC_TELEPHONE
debug3: /etc/ssh/sshd_config:125 setting AcceptEnv COLORTERM
debug1: sshd version OpenSSH_8.4, OpenSSL 1.1.1l  24 Aug 2021
debug1: private host key #0: ssh-ed25519 SHA256:uSzM72qmxWdv7Wl3L4bGj5NMBpd0x9VxAwNibIXdP6k
debug1: rexec_argv[0]='/usr/sbin/sshd'
debug1: rexec_argv[1]='-dddDe'
debug1: rexec_argv[2]='-o'
debug1: rexec_argv[3]='PidFile=/run/sshd.pid'
debug3: oom_adjust_setup
debug1: Set /proc/self/oom_score_adj from 0 to -1000
debug2: fd 3 setting O_NONBLOCK
debug1: Bind to port 22 on 0.0.0.0.
Server listening on 0.0.0.0 port 22.
debug3: fd 4 is not O_NONBLOCK
debug1: Server will not fork when running in debugging mode.
debug3: send_rexec_state: entering fd = 7 config len 553
debug3: ssh_msg_send: type 0
debug3: send_rexec_state: done
debug1: rexec start in 4 out 4 newsock 4 pipe -1 sock 7
debug3: recv_rexec_state: entering fd = 5
debug3: ssh_msg_recv entering
debug3: recv_rexec_state: done
debug2: parse_server_config_depth: config rexec len 553
debug3: rexec:15 setting AddressFamily inet
debug3: rexec:21 setting HostKey /etc/ssh/ssh_host_ed25519_key
debug3: rexec:26 setting RekeyLimit default none
debug3: rexec:40 setting PubkeyAuthentication no
debug3: rexec:52 setting HostbasedAuthentication no
debug3: rexec:60 setting PasswordAuthentication yes
debug3: rexec:65 setting ChallengeResponseAuthentication no
debug3: rexec:86 setting UsePAM no
debug3: rexec:95 setting PrintMotd no
debug3: rexec:96 setting PrintLastLog no
debug3: rexec:123 setting AcceptEnv LANG LC_ALL LC_COLLATE LC_CTYPE LC_MESSAGES LC_MONETARY LC_NUMERIC LC_TIME LANGUAGE LC_ADDRESS LC_IDENTIFICATION LC_MEASUREMENT LC_NAME LC_PAPER LC_TELEPHONE
debug3: rexec:125 setting AcceptEnv COLORTERM
debug1: sshd version OpenSSH_8.4, OpenSSL 1.1.1l  24 Aug 2021
debug1: private host key #0: ssh-ed25519 SHA256:uSzM72qmxWdv7Wl3L4bGj5NMBpd0x9VxAwNibIXdP6k
debug1: inetd sockets after dupping: 3, 3
Connection from 127.0.0.1 port 59658 on 127.0.0.1 port 22 rdomain ""
ebug1: Local version string SSH-2.0-OpenSSH_8.4
debug1: Remote protocol version 2.0, remote software version OpenSSH_8.4
debug1: match: OpenSSH_8.4 pat OpenSSH* compat 0x04000000
debug2: fd 3 setting O_NONBLOCK
debug3: ssh_sandbox_init: preparing seccomp filter sandbox
debug2: Network child is on pid 8608
debug3: preauth child monitor started
debug3: privsep user:group 22:22 [preauth]
debug1: permanently_set_uid: 22/22 [preauth]
debug3: ssh_sandbox_child: setting PR_SET_NO_NEW_PRIVS [preauth]
debug3: ssh_sandbox_child: attaching seccomp filter program [preauth]
debug1: list_hostkey_types: ssh-ed25519 [preauth]
debug3: send packet: type 20 [preauth]
debug1: SSH2_MSG_KEXINIT sent [preauth]
debug1: monitor_read_log: child log fd closed
debug3: mm_request_receive entering
debug1: do_cleanup
debug1: Killing privsep child 8608

Client command and log:

ssh -vvv localhost

Quote

OpenSSH_8.4p1, OpenSSL 1.1.1l  24 Aug 2021
debug1: Reading configuration data /home/prof/.ssh/config
debug1: Reading configuration data /etc/ssh/ssh_config
debug3: expanded UserKnownHostsFile '~/.ssh/known_hosts' -> '/home/prof/.ssh/known_hosts'
debug3: expanded UserKnownHostsFile '~/.ssh/known_hosts2' -> '/home/prof/.ssh/known_hosts2'
debug1: Authenticator provider $SSH_SK_PROVIDER did not resolve; disabling
debug2: resolving "localhost" port 22
debug2: ssh_connect_direct
debug1: Connecting to localhost [::1] port 22.
debug1: connect to address ::1 port 22: Connection refused
debug1: Connecting to localhost [127.0.0.1] port 22.
debug1: Connection established.
debug1: identity file /home/prof/.ssh/id_ed25519 type 3
debug1: identity file /home/prof/.ssh/id_ed25519-cert type 7
debug1: Local version string SSH-2.0-OpenSSH_8.4
debug1: Remote protocol version 2.0, remote software version OpenSSH_8.4
debug1: match: OpenSSH_8.4 pat OpenSSH* compat 0x04000000
debug2: fd 3 setting O_NONBLOCK
debug1: Authenticating to localhost:22 as 'prof'
debug3: hostkeys_foreach: reading file "/home/prof/.ssh/known_hosts"
debug3: hostkeys_foreach: reading file "/etc/ssh/ssh_known_hosts"
debug3: record_hostkey: found ca key type ED25519 in file /etc/ssh/ssh_known_hosts:1
debug3: load_hostkeys: loaded 1 keys from localhost
debug3: order_hostkeyalgs: prefer hostkeyalgs: ecdsa-sha2-nistp256-cert-v01@openssh.com,ecdsa-sha2-nistp384-cert-v01@openssh.com,ecdsa-sha2-nistp521-cert-v01@openssh.com,sk-ecdsa-sha2-nistp256-cert-v01@openssh.com,ssh-ed25519-cert-v01@openssh.com,sk-ssh-ed25519-cert-v01@openssh.com,rsa-sha2-512-cert-v01@openssh.com,rsa-sha2-256-cert-v01@openssh.com,ssh-rsa-cert-v01@openssh.com
debug3: send packet: type 20
debug1: SSH2_MSG_KEXINIT sent
debug3: receive packet: type 20
debug1: SSH2_MSG_KEXINIT received
debug2: local client KEXINIT proposal
debug2: KEX algorithms: curve25519-sha256,curve25519-sha256@libssh.org,ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group16-sha512,diffie-hellman-group18-sha512,diffie-hellman-group14-sha256,ext-info-c
debug2: host key algorithms: ecdsa-sha2-nistp256-cert-v01@openssh.com,ecdsa-sha2-nistp384-cert-v01@openssh.com,ecdsa-sha2-nistp521-cert-v01@openssh.com,sk-ecdsa-sha2-nistp256-cert-v01@openssh.com,ssh-ed25519-cert-v01@openssh.com,sk-ssh-ed25519-cert-v01@openssh.com,rsa-sha2-512-cert-v01@openssh.com,rsa-sha2-256-cert-v01@openssh.com,ssh-rsa-cert-v01@openssh.com,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521,sk-ecdsa-sha2-nistp256@openssh.com,ssh-ed25519,sk-ssh-ed25519@openssh.com,rsa-sha2-512,rsa-sha2-256,ssh-rsa
debug2: ciphers ctos: chacha20-poly1305@openssh.com,aes128-ctr,aes192-ctr,aes256-ctr,aes128-gcm@openssh.com,aes256-gcm@openssh.com
debug2: ciphers stoc: chacha20-poly1305@openssh.com,aes128-ctr,aes192-ctr,aes256-ctr,aes128-gcm@openssh.com,aes256-gcm@openssh.com
debug2: MACs ctos: umac-64-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-sha1-etm@openssh.com,umac-64@openssh.com,umac-128@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-sha1
debug2: MACs stoc: umac-64-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-sha1-etm@openssh.com,umac-64@openssh.com,umac-128@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-sha1
debug2: compression ctos: none,zlib@openssh.com,zlib
debug2: compression stoc: none,zlib@openssh.com,zlib
debug2: languages ctos:
debug2: languages stoc:
debug2: first_kex_follows 0
debug2: reserved 0
debug2: peer server KEXINIT proposal
debug2: KEX algorithms: curve25519-sha256,curve25519-sha256@libssh.org,ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group16-sha512,diffie-hellman-group18-sha512,diffie-hellman-group14-sha256
debug2: host key algorithms: ssh-ed25519
debug2: ciphers ctos: chacha20-poly1305@openssh.com,aes128-ctr,aes192-ctr,aes256-ctr,aes128-gcm@openssh.com,aes256-gcm@openssh.com
debug2: ciphers stoc: chacha20-poly1305@openssh.com,aes128-ctr,aes192-ctr,aes256-ctr,aes128-gcm@openssh.com,aes256-gcm@openssh.com
debug2: MACs ctos: umac-64-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-sha1-etm@openssh.com,umac-64@openssh.com,umac-128@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-sha1
debug2: MACs stoc: umac-64-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-sha1-etm@openssh.com,umac-64@openssh.com,umac-128@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-sha1
debug2: compression ctos: none,zlib@openssh.com
debug2: compression stoc: none,zlib@openssh.com
debug2: languages ctos:
debug2: languages stoc:
debug2: first_kex_follows 0
debug2: reserved 0
debug1: kex: algorithm: curve25519-sha256
debug1: kex: host key algorithm: ssh-ed25519
debug1: kex: server->client cipher: chacha20-poly1305@openssh.com MAC: <implicit> compression: none
debug1: kex: client->server cipher: chacha20-poly1305@openssh.com MAC: <implicit> compression: none
debug3: send packet: type 30
debug1: expecting SSH2_MSG_KEX_ECDH_REPLY
Connection reset by 127.0.0.1 port 22

Tried different things found on google, such as network MTU, various ciphers/MACs/KexAlgorithms settings, nothing sheds any light. Getting kind of desperate now.

Thank you for all your help!

[cardinal]

Vote for Bug Report:  i686 openssh requires >=sys-kernel/linux-headers-5.1

The problem below could be an issue with my computer, try updating the headers first by copy/renaming to linux-headers-5.7-r1.ebuild.

Unmask with keyword="*" install linux-headers-5.7 and rebuild openssh-8.4_p1-r3

Yesterday I had the same problem again in spite of linux-headers-5.7 still being installed when the upgrade of openssl forced an openssh rebuild.

To get a working version again I had to update openssh by adding the openssh folder from gentoo portage to my local overlay, remove  acct-group/sshd acct-user/sshd from openssh-8.8_p1.ebuild, add user-info.eclass from gentoo to funtoo.

 

 

Edited October 7, 2021 by cardinal
grammer

[prof]

Oh Gee, I would never come up to this on my own!

Thank you @cardinal for the workaround which actually works. I did not even expect anyone to be able to answer this at all, not to mention the same day answer!

Just for the record, on top of doing everything you described, I also had to remove a reference to virtual/libcrypto from the ebuild as this package does not exist in Funtoo, and manually download the openssh tarball from distfiles.gentoo.org into /var/cache/portage/distfiles directory.

Got my SSH daemon back! Thank you again!

P.S. We both understand that it is just a dirty hack, a real solution is to be delivered from Funtoo... Upvoted a bug.