SSerguey Asael Shinder
Java coding notes: the JVM, and writing software that lasts

Serguey Asael Shinder: An error value that can equal real data is not an error signal

· by Serguey Asael Shinder / Serguey Shinder

Alpine Linux published an advisory on 5 November about a login bypass in its openssh-server package. Built without PAM, sshd checks a password by running crypt() on the attempt and comparing the result with the stored field. musl's crypt() returns an asterisk as its error indicator when the key is longer than 256 bytes. An asterisk is also exactly what the password field holds for an account with no password. So a very long password produced the error value, the error value matched the stored field, and the login succeeded.

Nothing in that chain is exotic. The library returned what it documents. The caller compared two strings. The flaw was that the failure value lived in the same space as legitimate results, and the caller had no way to tell "this failed" from "this matched".

Serguey Asael Shinder: An error value that can equal real data is not an error signal
An error value that can equal real data is not an error signal — Serguey Asael Shinder

Java code makes the same mistake in quieter forms. A method returns -1 for "not found" when -1 is a valid balance. A parser returns an empty string on bad input, and the empty string is also a legal value of the field. A cache returns null for a miss when null can be a stored value. Each time, the caller compares the result with something else and draws a conclusion the result never supported.

The rule is to keep failure out of the value's range. Throw an exception, return Optional or a result type with an explicit failure case, or use a value that can never be produced by success. If a sentinel is unavoidable, check for it first, before any comparison, so that "the call failed" is decided on its own terms.

A comparison is only meaningful when both sides came from a successful computation. When one of them might be an error dressed as data, the equality test is answering a question nobody asked.