Technically at the machine language levels FPU do not generate NaN as a value in the results as the result itself is incapable of conveying that information. It's set in the flags register instead.
For example on the 8087 divide by zero actually throws an interrupt error, meaning program execution should stop. you have to manually mask the interrupt (via FLDCW if memory serves) , something no compiled programming language ever really bothered with other than to customize the error message and/or provide debugging output / IDE integration. Gah, been decades since I dealt with that at the ASM level.
It's not something that would/should be reported at the execution level on a live program in a way that the software keeps plodding on blindly. That was a Fortran mistake and it's why few languages do anything more than exit on error with it. It might be in the spec, but much like assembler flags it's the type of thing high level languages were created to hide from the programmer.
That said, so many times I'd kill for access to ZF, PF, and SF after an assignment/operation.
You’re right that their mistake was using it for error handling, but the bigger mistake is returning it as the value via the loose typecasting, instead of throwing an error that halts execution dead in its tracks. MORE confusing when they have the perfectly good — though somewhat hobbled — “throw new Error()” that one could try/catch.
Though that treads into the Fortran vs. Algol argument of if “divide by zero” should be infinity or “no such thing”. I’m a little surprised Eich sided with the Fortran crowd on that one given the language does have an “infinity” theoretical as well.
And I feel you supporting Fortran code in the early '90's. I was doing the same thing with DiBol and Forth, and later was tasked with porting much of it to Ada as those Dec mainframes they were running on were giving up the ghost.