Float/double to int/long conversion produces wrong results for NaN and infinity (FISTP lacks JLS special casing)

[LSantha]

Bug: Primitive float/double→int/long conversion produces wrong results for NaN and infinity

The JVM Specification (§5.1.3) requires:

• (int) NaN / (long) NaN → 0
• (int) +Infinity → Integer.MAX_VALUE, (long) +Infinity → Long.MAX_VALUE
• (int) -Infinity → Integer.MIN_VALUE, (long) -Infinity → Long.MIN_VALUE

JNode produces Integer.MIN_VALUE / Long.MIN_VALUE for all of NaN, +Infinity, and -Infinity — only correct for -Infinity.

Impact (mauve MathTest failures caused by this bug)

Test	Expression	Expected	JNode Actual	In scope?
test_round - 3	Math.round(Double.NaN)	0L	Long.MIN_VALUE	✅ Yes
test_round - 5	Math.round(+Inf float)	Integer.MAX_VALUE	Integer.MIN_VALUE	✅ Yes
test_round - 7	Math.round(+Inf double)	Long.MAX_VALUE	Long.MIN_VALUE	✅ Yes
test_round - 10	Math.round(Float.NaN)	0	Integer.MIN_VALUE	✅ Yes

Tests that happen to pass (correct by accident despite the bug):

• test_round - 4: expects Integer.MIN_VALUE → JNode's FISTP returns the indefinite integer Integer.MIN_VALUE → matches
• test_round - 6: expects Long.MIN_VALUE → JNode's FISTP returns the indefinite integer Long.MIN_VALUE → matches

Tests NOT in scope (separate x87 precision bug): test_rint - 1,2,3 — caused by x87 FPU running in extended precision (64-bit mantissa) instead of double precision (53-bit). See core/src/native/x86/cpu.asm.

Steps to Reproduce

1. Boot JNode in "tests" mode (select JNode tests (all plugins + tests) from GRUB)
2. Run the following Java code inside JNode:

public class Test {
    public static void main(String[] a) {
        // NaN cases - should be 0
        System.out.println("(long)NaN=" + (long)Double.NaN);
        System.out.println("(int)NaNf=" + (int)Float.NaN);
        // +Inf cases - should be MAX_VALUE
        System.out.println("(long)+Inf=" + (long)Double.POSITIVE_INFINITY);
        System.out.println("(int)+INff=" + (int)Float.POSITIVE_INFINITY);
        // -Inf cases - should be MIN_VALUE
        System.out.println("(long)-Inf=" + (long)Double.NEGATIVE_INFINITY);
        System.out.println("(int)-INff=" + (int)Float.NEGATIVE_INFINITY);
    }
}

3. Compile with javac Test.java and run java Test

Expected output:

(long)NaN=0
(int)NaNf=0
(long)+Inf=9223372036854775807
(int)+INff=2147483647
(long)-Inf=-9223372036854775808
(int)-INff=-2147483648

Actual output:

(long)NaN=-9223372036854775808
(int)NaNf=-2147483648
(long)+Inf=-9223372036854775808
(int)+INff=-2147483648
(long)-Inf=-9223372036854775808
(int)-INff=-2147483648

3. Also verify via mauve: mauve-plugin -v java.lang.Math | grep FAIL should show the 4 failing round tests above.

Root Cause

JNode's x87 FPU-to-integer conversion uses the FISTP x87 instruction directly without pre-checking for NaN or infinity. The x87 FISTP instruction produces the "indefinite integer value" (0x80000000 for int, 0x8000000000000000 for long) for any value that is NaN, infinity, or otherwise cannot be represented as an integer. This indefinite value equals Integer.MIN_VALUE / Long.MIN_VALUE.

The Java Language Specification §5.1.3 requires special-case handling:

If the floating-point number is NaN, the result of an integer conversion is 0.
Otherwise, if the floating-point number is not an infinity, the floating-point value is rounded to an integer value V, rounding toward zero using IEEE 754 round-toward-zero mode. Then there are two cases:

• If T is long and this integer value can be represented as a long, then the result is the long value V.
• If T is int and this integer value can be represented as an int, then the result is the int value V.
  Otherwise, one of the following two cases must be true:
• The value must be too small (a negative value of large magnitude or negative infinity), and the result is the smallest representable value of type int or long.
• The value must be too large (a positive value of large magnitude or positive infinity), and the result is the largest representable value of type int or long.

JNode must check for NaN, infinity, and overflow before executing FISTP.

Affected Source Files

All f2i/f2l/d2i/d2l conversion paths:

• core/src/core/org/jnode/vm/x86/compiler/l1a/FPCompilerFPU.java (lines 166-183)
• core/src/core/org/jnode/vm/x86/compiler/l1a/IntItem.java (lines 93-95)
• core/src/core/org/jnode/vm/x86/compiler/l1a/LongItem.java (lines 109-111)
• core/src/core/org/jnode/vm/x86/compiler/l1b/FPCompilerFPU.java (lines 172-189)
• core/src/core/org/jnode/vm/x86/compiler/l1b/IntItem.java (lines 94-95)
• core/src/core/org/jnode/vm/x86/compiler/l1b/LongItem.java (lines 104-105)
• core/src/core/org/jnode/vm/x86/compiler/l2/GenericX86CodeGenerator.java (lines 430-695, multiple overloads, only F2I implemented)

x87 assembler emission:

• core/src/core/org/jnode/assembler/x86/X86BinaryAssembler.java (lines 1777-1789)
• core/src/core/org/jnode/assembler/x86/X86TextAssembler.java (lines 843-847)

Suggested Fix

Option A (emit guards before FISTP): Before each FISTP instruction, emit conditional code that:

1. Checks if the FPU value is NaN (using FXAM + FSTSW to check the C0/C2/C3 flags)
2. Checks if the FPU value exceeds the integer range (compare against max/min representable values)
3. Branches to special-case handlers that return the correct JLS values
4. Falls through to FISTP for normal values

Option B (saturated conversion helper): Write a small assembly helper routine that:

1. Checks the FPU value with FXAM
2. For NaN: returns 0
3. For +Infinity: returns MAX_VALUE
4. For -Infinity: returns MIN_VALUE
5. For overflow: returns MIN_VALUE or MAX_VALUE depending on sign
6. For normal values: uses FISTP with truncation
   Then replace all FISTP sites with calls to this helper.

Option C (fix at a higher level): Add a check in FPCompilerFPU.convert() (and the L2 equivalents) to emit guard code that pops the value from the FPU stack, converts to memory, loads back, compares, and branches accordingly.

Option D (workaround in NativeStrictMath.round): Fix only Math.round by special-casing NaN and infinity in Java before the (int)floor(f + 0.5f) cast. Simple but only fixes Math.round, not raw f2i/d2i JVM bytecodes.

Verification of Fix

Primary test (ConversionTest.java — 18 assertions):

public class ConversionTest {
    static int failures = 0;
    static void check(long actual, long expected, String name) {
        if (actual != expected) {
            System.out.println("FAIL: " + name + " got " + actual + " expected " + expected);
            failures++;
        }
    }
    static void check(int actual, int expected, String name) {
        if (actual != expected) {
            System.out.println("FAIL: " + name + " got " + actual + " expected " + expected);
            failures++;
        }
    }
    public static void main(String[] a) {
        // NaN conversion
        check((long)Double.NaN, 0L, "(long)NaN");
        check((int)Float.NaN, 0, "(int)NaNf");
        // Positive infinity
        check((long)Double.POSITIVE_INFINITY, Long.MAX_VALUE, "(long)+Inf");
        check((int)Float.POSITIVE_INFINITY, Integer.MAX_VALUE, "(int)+INff");
        // Negative infinity
        check((long)Double.NEGATIVE_INFINITY, Long.MIN_VALUE, "(long)-Inf");
        check((int)Float.NEGATIVE_INFINITY, Integer.MIN_VALUE, "(int)-INff");
        // Normal values still work
        check((long)1.5, 1L, "(long)1.5");
        check((int)1.5f, 1, "(int)1.5f");
        check((long)-1.5, -1L, "(long)-1.5");
        check((int)-1.5f, -1, "(int)-1.5f");
        // Zero
        check((long)0.0, 0L, "(long)0.0");
        check((long)-0.0, 0L, "(long)-0.0");
        // Overflow
        check((long)1e20, Long.MAX_VALUE, "(long)1e20 overflow");
        check((long)-1e20, Long.MIN_VALUE, "(long)-1e20 underflow");
        check((int)1e20f, Integer.MAX_VALUE, "(int)1e20f overflow");
        check((int)-1e20f, Integer.MIN_VALUE, "(int)-1e20f underflow");
        // MAX/MIN_VALUE boundaries
        check((long)Long.MAX_VALUE, Long.MAX_VALUE, "(long)Long.MAX_VALUE");
        check((long)Long.MIN_VALUE, Long.MIN_VALUE, "(long)Long.MIN_VALUE");

        if (failures == 0) System.out.println("All tests PASS");
        else System.out.println("FAILURES: " + failures);
    }
}

Mauve in-scope verification (the 4 round tests this bug fixes):

mauve-plugin -v java.lang.Math | grep "test_round - \\(3\\|5\\|7\\|10\\)"
# All 4 should show PASS

Test_round - 4 and -6 will continue to PASS (they were correct by accident even before the fix — verify they do not regress).

The 3 rint tests (test_rint - 1,2,3) will stay FAIL — they are a separate bug (x87 precision mode) and are NOT expected to change.

Broader regression check:

mauve-plugin -c java.lang.Integer
mauve-plugin -c java.lang.Long
# Should show no regressions

Proposed Solution Constraints

The solution must be submitted as a Pull Request that includes:

1. The source code fix (in IntItem.java, LongItem.java, and/or GenericX86CodeGenerator.java conversion paths)
2. ConversionTest.java added to the test suite
3. Verification that mauve-plugin -v java.lang.Math | grep "test_round - [35710]" shows all PASS
4. Verification that mauve-plugin -c java.lang.Integer and java.lang.Long do not regress

[LSantha]

/oc fix this , the jnode-interact skill may be useful for testing your fix

[opencode-agent]

Created PR #462

opencode session  |  github run

[LSantha]

/oc - the pr fails to build, try to test the code by building and running jnode , see the jnode-interact skill, current the boot is failing early

[LSantha]

/oc - the pr fails to build, try to test the code by building and running jnode , see the jnode-interact skill, current the boot is failing early

[LSantha]

/oc - the pr fails to build, try to test the code by building and running jnode , see the jnode-interact skill, current the boot is failing early