f*::min/max do not behave as documented for signaling NaN on aarch64, or with optimizations

[RalfJung]

View all comments

Our documentation says that max(SNaN, 0.0) should return 0.0. On x86_64, this is indeed what happens. However, on aarch64, we get a QNaN as a result instead. On Android, the reported result for this test is:

F32::from(F32::nan(Sign::Pos, NaNKind::Signaling, 1).as_f32().max(0.0)) = 0x7fc00001 (NaN: Pos, Quiet, payload = 0x1)

It seems that LLVM never actually correctly implemented the promise that "if exactly one argument is NaN, the other argument is returned". Recently things got a lot more messy (see llvm/llvm-project#170082 and also this discussion), but apparently things were already broken before this recent chaos.

Also, since Rust 1.91 optimizations seem to fold max(SNaN, x) into NaN.

It is also worth noting that glibc fmin/fmax were deliberately patched so that SNaN inputs produce NaN results. (However, not all libc do that; the C standard generally says very little about SNaN.)

So it might just be worth dropping that promise and allowing NaN outputs when there is an SNaN input (but also continue to allow returning the other input).

Cc @tgross35 @valadaptive @nikic @workingjubilee @rust-lang/libs-api

[workingjubilee]

should we just stop trying to make these poorly-conceived intrinsics work and just use minimum and maximum instead?

[tgross35]

should we just stop trying to make these poorly-conceived intrinsics work and just use minimum and maximum instead?

Do you mean {min,max}imumnum? I think that's the ideal case, but buggy LLVM optimizations and potential lost performance are holding it back. (I expect that the result of the LLVM discourse discussion will be that we will know exactly which intrinsic to use for our desired behavior, then anything not matching it will be an LLVM bug.)

[RalfJung]

{min,max}imumnum will never be very good on x86 because they require ordering -0.0 below +0.0 and x86 does not have an instruction for that.

My understanding is that we can get the NaN behavior of {min,max}imumnum (that's what we already document), but not the signed-zero behavior.

[Amanieu]

I personally like the idea of having min/max do whatever is fastest on the current target, which essentially means that:

• if one input is NaN then we non-deterministically return either an arbitrary NaN or the other input value.
• if the inputs are two zeros of different signs then we non-deterministically return either 0.0 or -0.0.

This works well for the vast majority of cases where people don't care about NaN or signed zeros. We could then have specialized functions which guarantee one of the 3 IEEE 754 behaviors:

• minimum/maximum
  • if any input is NaN, return NaN
• minimum_number/maximum_number (2019)
  • if any input is NaN, return the other, or an arbitrary NaN if both are NaN
• min_num/max_num (2008)
  • if any input is sNaN, return qNaN
  • otherwise: if any input is NaN, return the other, or an arbitrary NaN if both are NaN

The downsides are that this would be a breaking change and that it would be hard to explain the differences between these similarly-named functions to users.

[valadaptive]

{min,max}imumnum will never be very good on x86 because they require ordering -0.0 below +0.0 and x86 does not have an instruction for that.

The current lowering in LLVM is pretty bad, especially on the default x86-64(-v1) microarchitecture level, since it uses "select" operations and LLVM really wants to lower them to branches. llvm/llvm-project#170069 improves this a fair bit, and makes the operations easily vectorizable. I believe WebAssembly engines implement a similar lowering, although last I checked, it's not identical.

I surveyed the range of behaviors and available hardware operations when working on a SIMD library (linebender/fearless_simd#133), although keep in mind that this was before I discovered all the LLVM weirdness and the sNaN behavior divergence. There are a bajillion different sets of semantics here, and none of them are optimal in all cases.

[RalfJung]

if any input is sNaN, return qNaN

Note that LLVM generally reserves the right to treat an sNaN like a qNaN, so for any function that distinguishes between sNaN and qNaN, an sNaN input can always non-deterministically also generate the qNaN behavior. Currently, we have only very few such functions -- specifically, powf and powi. I'd rather avoid adding more, so I don't think we should have any operation that has the 2008 NaN behavior. (It's also non-associative to make matters worse.)

[RalfJung]

Seems like with optimizations, we also get the wrong behavior on x86: Playground

[src/main.rs:9:5] max(f32::NAN, 0.0) = 0.0
[src/main.rs:10:5] max(0.0, f32::NAN) = 0.0
[src/main.rs:11:5] max(F32_SNAN, 0.0) = NaN
[src/main.rs:12:5] max(0.0, F32_SNAN) = NaN

If I disable inlining for max, then the NaN results turn into 0.0.