Adding SSH Key Login and Troubleshooting

1. Steps to add a key

1.1 Client-side generation of ssh key

There are two formats of keys:

The old format, the private key starts with -----BEGIN RSA PRIVATE KEY-----.
1

ssh-keygen -m PEM -t rsa -b 4096 -C "mail@chenshaowen.com"
New format, private key starts with -----BEGIN OPENSSH PRIVATE KEY-----.
1

ssh-keygen -t rsa -b 4096 -C "mail@chenshaowen.com"

Since some older systems do not support the new format of the Key, it is recommended to generate the old format of the Key. If you are already using a new format Key, you can use the puttygen tool to convert the new format Key to the old format.

Edit the /etc/ssh/sshd_config file and make sure the following configuration is turned on.

PubkeyAuthentication yes
AuthorizedKeysFile .ssh/authorized_keys
PermitRootLogin yes
PasswordAuthentication yes

It is recommended to keep the PasswordAuthentication login method until Key login is successfully configured. Whether or not to allow root login depends on your needs and is usually disabled.

After modifying the sshd configuration, you can test if the configuration file is correct.

sshd -t

Make sure there are no error messages, and then restart sshd.

`1`	`systemctl restart sshd`

1.3 Add the client’s key to `authorized_keys` on the server side

Edit the authorized_keys file of the logged-in user and add the public key. The public key is stored in the client ~/.ssh/id_rsa.pub file.

Get the public key on the client side
1 2 3

cat ~/.ssh/id_rsa.pub xxxx
Add the public key on the server side
1

vim ~/.ssh/authorized_keys

At this point, you can log in with the key under normal circumstances. However, I always encounter problems and cannot configure it successfully at once. I had a case where I couldn’t configure the key for root on CentOS 7.6. After about half a year of tolerating password login, I finally found out when I looked at the sshd logs that it was a problem with home directory permissions. Here are some common troubleshooting methods.

2. View ssh access logs

Use the -v parameter to view detailed logs.

ssh root@1.1.1.1 -v

OpenSSH_8.9p1, OpenSSL 1.1.1m  14 Dec 2021
debug1: Reading configuration data /Users/shaowenchen/.ssh/config
debug1: Reading configuration data /usr/local/etc/ssh/ssh_config
debug1: Connecting to 1.1.1.1 port 22.
debug1: Connection established.
debug1: identity file /Users/shaowenchen/.ssh/id_rsa type 0
debug1: identity file /Users/shaowenchen/.ssh/id_rsa-cert type -1
debug1: identity file /Users/shaowenchen/.ssh/id_ecdsa type -1
debug1: identity file /Users/shaowenchen/.ssh/id_ecdsa-cert type -1
debug1: identity file /Users/shaowenchen/.ssh/id_ecdsa_sk type -1
debug1: identity file /Users/shaowenchen/.ssh/id_ecdsa_sk-cert type -1
debug1: identity file /Users/shaowenchen/.ssh/id_ed25519 type -1
debug1: identity file /Users/shaowenchen/.ssh/id_ed25519-cert type -1
debug1: identity file /Users/shaowenchen/.ssh/id_ed25519_sk type -1
debug1: identity file /Users/shaowenchen/.ssh/id_ed25519_sk-cert type -1
debug1: identity file /Users/shaowenchen/.ssh/id_xmss type -1
debug1: identity file /Users/shaowenchen/.ssh/id_xmss-cert type -1
debug1: identity file /Users/shaowenchen/.ssh/id_dsa type -1
debug1: identity file /Users/shaowenchen/.ssh/id_dsa-cert type -1
debug1: Local version string SSH-2.0-OpenSSH_8.9
debug1: Remote protocol version 2.0, remote software version OpenSSH_7.4
debug1: compat_banner: match: OpenSSH_7.4 pat OpenSSH_7.4* compat 0x04000006
debug1: Authenticating to 1.1.1.1:22 as 'root'
debug1: load_hostkeys: fopen /Users/shaowenchen/.ssh/known_hosts2: No such file or directory
debug1: load_hostkeys: fopen /usr/local/etc/ssh/ssh_known_hosts: No such file or directory
debug1: load_hostkeys: fopen /usr/local/etc/ssh/ssh_known_hosts2: No such file or directory
debug1: SSH2_MSG_KEXINIT sent
debug1: SSH2_MSG_KEXINIT received
debug1: kex: algorithm: curve25519-sha256
debug1: kex: host key algorithm: ecdsa-sha2-nistp256
debug1: kex: server->client cipher: chacha20-poly1305@openssh.com MAC: <implicit> compression: none
debug1: kex: client->server cipher: chacha20-poly1305@openssh.com MAC: <implicit> compression: none
debug1: expecting SSH2_MSG_KEX_ECDH_REPLY
debug1: SSH2_MSG_KEX_ECDH_REPLY received
debug1: Server host key: ecdsa-sha2-nistp256 SHA256:dIN1fBcDGeQ07m3An2G+p5sNC0Sx9TEAg95qXSs01s8
debug1: load_hostkeys: fopen /Users/shaowenchen/.ssh/known_hosts2: No such file or directory
debug1: load_hostkeys: fopen /usr/local/etc/ssh/ssh_known_hosts: No such file or directory
debug1: load_hostkeys: fopen /usr/local/etc/ssh/ssh_known_hosts2: No such file or directory
debug1: Host '1.1.1.1' is known and matches the ECDSA host key.
debug1: Found key in /Users/shaowenchen/.ssh/known_hosts:2
debug1: rekey out after 134217728 blocks
debug1: SSH2_MSG_NEWKEYS sent
debug1: expecting SSH2_MSG_NEWKEYS
debug1: SSH2_MSG_NEWKEYS received
debug1: rekey in after 134217728 blocks
debug1: get_agent_identities: ssh_fetch_identitylist: agent contains no identities
debug1: Will attempt key: /Users/shaowenchen/.ssh/id_rsa RSA SHA256:xxx/bYCe3TYhQ68gJA
debug1: Will attempt key: /Users/shaowenchen/.ssh/id_ecdsa
debug1: Will attempt key: /Users/shaowenchen/.ssh/id_ecdsa_sk
debug1: Will attempt key: /Users/shaowenchen/.ssh/id_ed25519
debug1: Will attempt key: /Users/shaowenchen/.ssh/id_ed25519_sk
debug1: Will attempt key: /Users/shaowenchen/.ssh/id_xmss
debug1: Will attempt key: /Users/shaowenchen/.ssh/id_dsa
debug1: SSH2_MSG_EXT_INFO received
debug1: kex_input_ext_info: server-sig-algs=<rsa-sha2-256,rsa-sha2-512>
debug1: SSH2_MSG_SERVICE_ACCEPT received
debug1: Authentications that can continue: publickey,gssapi-keyex,gssapi-with-mic
debug1: Next authentication method: publickey
debug1: Offering public key: /Users/shaowenchen/.ssh/id_rsa RSA SHA256:xxx/xxx
debug1: Server accepts key: /Users/shaowenchen/.ssh/id_rsa RSA SHA256:xxx/xxx
Authenticated to 1.1.1.1 using "publickey".
debug1: channel 0: new [client-session]
debug1: Requesting no-more-sessions@openssh.com
debug1: Entering interactive session.
debug1: pledge: filesystem
debug1: client_input_global_request: rtype hostkeys-00@openssh.com want_reply 0
debug1: client_input_hostkeys: searching /Users/shaowenchen/.ssh/known_hosts for 1.1.1.1 / (none)
debug1: client_input_hostkeys: searching /Users/shaowenchen/.ssh/known_hosts2 for 1.1.1.1 / (none)
debug1: client_input_hostkeys: hostkeys file /Users/shaowenchen/.ssh/known_hosts2 does not exist
debug1: client_global_hostkeys_private_confirm: server used untrusted RSA signature algorithm ssh-rsa for key 0, disregarding
Learned new hostkey: ED25519 SHA256:xxx/xxx
Adding new key for 1.1.1.1 to /Users/shaowenchen/.ssh/known_hosts: ssh-ed25519 SHA256:xxx/xxx
debug1: update_known_hosts: known hosts file /Users/shaowenchen/.ssh/known_hosts2 does not exist

You can see that the ssh client will try various login methods in the runtime environment until it succeeds; if it tries all of them and still does not succeed, an error will be reported.

3. Viewing the sshd access log

The sshd logs are an easy place to overlook, but can provide very useful information.

journalctl -u sshd -f

-- Logs begin at Wed 2021-12-15 10:44:46 CST. --
Mar 09 19:56:22 node1 sshd[171565]: Accepted publickey for root from 1.1.1.1 port 61832 ssh2: RSA SHA256:xxx/xxx
Mar 10 08:17:15 node1 sshd[2028880]: Accepted publickey for root from 1.1.1.1 port 63966 ssh2: RSA SHA256:xxx/xxx
Mar 10 08:23:49 node1 sshd[2045429]: Accepted publickey for root from 1.1.1.1 port 64047 ssh2: RSA SHA256:xxx/xxx
Mar 10 08:32:11 node1 systemd[1]: Stopping OpenSSH server daemon...
Mar 10 08:32:11 node1 sshd[171382]: Received signal 15; terminating.
Mar 10 08:32:11 node1 systemd[1]: Stopped OpenSSH server daemon.
Mar 10 08:32:11 node1 systemd[1]: Starting OpenSSH server daemon...
Mar 10 08:32:11 node1 sshd[2066365]: Server listening on 0.0.0.0 port 22.
Mar 10 08:32:11 node1 sshd[2066365]: Server listening on :: port 22.

The problem I ran into was that I saw this line in the log: sshd[7302]: Authentication refused: bad ownership or modes for directory /root and realized that the user home directory permissions were incorrect.

4. Check server-side file permissions

You need to check the permissions of the relevant directory or file.

~
~/.ssh
``~/.ssh/authorized_keys`''

If it is not as expected, you need to fix the permissions.

1
2
3

chmod -R 750 ~
chmod 700 ~/.ssh
chmod 600 ~/.ssh/authorized_keys

Table of Contents