After upgrading to MacOS Ventura, if you try to log in to the server with ssh, you will probably find that you can’t log in anymore.

1
2
3

$ ssh 192.1681.86
Unable to negotiate with 192.168.1.86 port 22: no matching host key type found.
Their offer: ssh-rsa

Troubleshooting

From the error message, it is easy to deduce that the local ssh does not accept the public key type provided by the server sshd.

So the guess is that the MacOS ssh client has been upgraded with the system and some of the default parameters have been changed. Checking the ssh version status of the old and new systems, I did find that the version has been upgraded, from 8.x to 9.x.

1
2
3
4
5

$ ssh -V
OpenSSH_8.6p1, LibreSSL 3.3.6

$ ssh -V
OpenSSH_9.0, LibreSSL 3.3.6

MacOS uses OpenSSH, an open source ssh package. So I went to the OpenSSH official website to check the version records, and found that OpenSSH has deprecated the rsa/sha1 algorithm combination since version 8.8. The original article is as follows.

Potentially-incompatible changes

This release disables RSA signatures using the SHA-1 hash algorithm by default. This change has been made as the SHA-1 hash algorithm is cryptographically broken, and it is possible to create chosen-prefix hash collisions for <USD$50K [1]

For most users, this change should be invisible and there is no need to replace ssh-rsa keys. OpenSSH has supported RFC8332 RSA/SHA-256/512 signatures since release 7.2 and existing ssh-rsa keys will automatically use the stronger algorithm where possible.

Incompatibility is more likely when connecting to older SSH implementations that have not been upgraded or have not closely tracked improvements in the SSH protocol. For these cases, it may be necessary to selectively re-enable RSA/SHA1 to allow connection and/or user authentication via the HostkeyAlgorithms and PubkeyAcceptedAlgorithms options. For example, the following stanza in ~/.ssh/config will enable RSA/SHA1 for host and user authentication for a single destination host:

1
2
3

Host old-host
   HostkeyAlgorithms +ssh-rsa
   PubkeyAcceptedAlgorithms +ssh-rsa

We recommend enabling RSA/SHA1 only as a stopgap measure until legacy implementations can be upgraded or reconfigured with another key type (such as ECDSA or Ed25519).

[1] “SHA-1 is a Shambles: First Chosen-Prefix Collision on SHA-1 and Application to the PGP Web of Trust” Leurent, G and Peyrin, T (2020) https://eprint.iacr.org/2020/014.pdf

Simply put, OpenSSH no longer considers the combination of RSA signature + SHA1 hash algorithm to meet cryptographic security requirements, so it is deprecated by default. As for how it is insecure, you can refer to a paper cited later in the original article: SHA-1 is a Shambles….

The original paper also mentions how to temporarily circumvent this problem, see the solution below.

At first glance, OpenSSH version 8.8 was released on 2021-09-26. MacOS Ventura uses version 9.0, which was released on 2022-04-08, and MacOS itself is publicly released on 2022-10-24. I have to say, this is pretty radical of Apple. One of the “consequences” of radicalization is that it will break a lot of tool chains that are not radical. Just like our company, almost all of our employees’ ssh-related tools were affected after upgrading to MacOS Ventura.

Of course, this is in line with Apple’s usual style, and I can accept it. Radical is good to keep up with the latest security standards, otherwise, I don’t know when the company will be motivated to upgrade it.

Simple solution

Open the ~/.ssh/config file, add the following configuration at the beginning of the file and save it.

1
2
3

Host *
    HostkeyAlgorithms +ssh-rsa
    PubkeyAcceptedAlgorithms +ssh-rsa

The above is the simplest and most brutal solution. The reason for this is that the above configuration takes effect for all Hosts, without granular control. This requires familiarity with ssh and ssh_config, which is not covered in detail in this article.

If you need a temporary solution, you can modify it by passing parameters directly on the command line (command line parameters have a higher priority than ssh_config).

1

$ ssh -o HostkeyAlgorithms=+ssh-rsa -o PubkeyAcceptedAlgorithms=+ssh-rsa ...

Of course, not all ssh behavior is hand-entered; many tools also execute ssh within the code, for example, by implementing ProxyCommand, which also requires modifying the parameters passed to ssh by these tools.

Accidental discovery

One of the big problems encountered by programmers, memory leaks (or related), was not expected to be fixed many times in the latest version of the OpenSSH version record:

This release contains fixes for three minor memory safety problems. None are believed to be exploitable, but we report most memory safety problems as potential security vulnerabilities out of caution.

  • ssh-keyscan(1): fix a one-byte overflow in SSH- banner processing. Reported by Qualys
  • ssh-keygen(1): double free() in error path of file hashing step in signing/verify code; GHPR333
  • ssh-keysign(8): double-free in error path introduced in openssh-8.9

Who would have thought that an open source software running on almost every MacOS/Linux/Windows in the world would have so many memory problems! Isn’t it nice to think that this was the case when you first learned C? 🤪

I was actually a little curious, and then I dug a little deeper to find out why the author wrote this code, and found the location of the code that wrote the bug (with deletions) (openssh/openssh-portable@d637c4a).

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12

@@ -484,7 +508,8 @@ hash_file(int fd, int hashalg, u_char *hash, size_t hashlen)
                        error("%s: read: %s", __func__, strerror(errno));
                        ssh_digest_free(ctx);
                        errno = oerrno;
-                       return SSH_ERR_SYSTEM_ERROR;
+                       r = SSH_ERR_SYSTEM_ERROR;
+                       goto out;
                }
        }
+ out:
+       sshbuf_free(b);
        ssh_digest_free(ctx);

As you can see, it originally freed (ssh_digest_free) first and then returned the error code directly (return SSH_ERR_SYSTEM_ERROR). When modified later, the error code is saved to r and goto to out, where ssh_digest_free is done again at out. This creates the double-free problem.

So: easy to write code, hard to maintain? 😅

A better solution

The simple solution mentioned above is not suitable for general use because it is not secure.

The new version of OpenSSH does not disable the entire RSA and SHA algorithms (the name ssh-rsa is quite misleading), but only the combination with SHA1. Other combinations, such as RSA/SHA-256/512, can still be used. You can upgrade the SSHD on the server side to solve this problem.

Changing the server and local public key types is also possible. There are many public key encryption types supported by OpenSSH, which can be obtained by the following command.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21

$ ssh -Q HostkeyAlgorithms
ssh-ed25519
ssh-ed25519-cert-v01@openssh.com
sk-ssh-ed25519@openssh.com
sk-ssh-ed25519-cert-v01@openssh.com
ssh-rsa
rsa-sha2-256
rsa-sha2-512
ssh-dss
ecdsa-sha2-nistp256
ecdsa-sha2-nistp384
ecdsa-sha2-nistp521
sk-ecdsa-sha2-nistp256@openssh.com
ssh-rsa-cert-v01@openssh.com
rsa-sha2-256-cert-v01@openssh.com
rsa-sha2-512-cert-v01@openssh.com
ssh-dss-cert-v01@openssh.com
ecdsa-sha2-nistp256-cert-v01@openssh.com
ecdsa-sha2-nistp384-cert-v01@openssh.com
ecdsa-sha2-nistp521-cert-v01@openssh.com
sk-ecdsa-sha2-nistp256-cert-v01@openssh.com

I personally like ed25519 very much because it is very short and provides higher security than RSA with the same key length. Honestly, it’s long overdue for promotion. It’s not that it hasn’t been promoted, it’s just that people’s habits are too hard to change.

Generating ed25519 is very simple, just specify the parameter -t ed25519.

1
2
3
4

$ ssh-keygen -t ed25519
Generating public/private ed25519 key pair.
Enter file in which to save the key (/home/tao/.ssh/id_ed25519): key
...

Then check the generated Keys.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11

$ cat key.pub
ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIKVbraJiv9DqLw2nJZij5re7b3cA9vOfDfVsEXaIykmS tao@nuc

$ cat key
-----BEGIN OPENSSH PRIVATE KEY-----
b3BlbnNzaC1rZXktdjEAAAAABG5vbmUAAAAEbm9uZQAAAAAAAAABAAAAMwAAAAtzc2gtZW
QyNTUxOQAAACClW62iYr/Q6i8NpyWYo+a3u293APbznw31bBF2iMpJkgAAAJD1B/Hy9Qfx
8gAAAAtzc2gtZWQyNTUxOQAAACClW62iYr/Q6i8NpyWYo+a3u293APbznw31bBF2iMpJkg
AAAEDlMj0HE1tbHzKl/4UAp3uixJ4R0e8hT8AQDKLPae/alqVbraJiv9DqLw2nJZij5re7
b3cA9vOfDfVsEXaIykmSAAAAB3Rhb0BudWMBAgMEBQY=
-----END OPENSSH PRIVATE KEY-----

It’s much shorter than RSA, right?

Ref