Wednesday, March 16, 2011

Decimal rounding problems

Given that Decimal.MaxValue = 79228162514264337593543950335m

Why does the next line give me 7922816251426433759354395034M in the Local window instead of 7922816251426433759354395033.5m as expected?

Decimal target = Decimal.MaxValue / 10m;

TIA

From stackoverflow
  • I suspect this is a compiler error, actually.

    Here's a short but complete program to show why I think that:

    using System;
    
    class Test
    {
        static void Main()
        {
            decimal constant = decimal.MaxValue / 10m;
            decimal calculated = decimal.MaxValue;
            calculated /= 10m;
    
            Console.WriteLine (constant);
            Console.WriteLine (calculated);        
        }
    }
    

    Output:

    7922816251426433759354395034
    7922816251426433759354395033.5
    

    I'll dig into the spec to see what guarantees are given.

    EDIT: In the spec, section 7.18 claims:

    The compile-time evaluation of constant expressions uses the same rules as run-time evaluation of non-constant expressions, except that where run-time evaluation would have thrown an exception, compile-time evaluation causes a compile-time error to occur.

    That's clearly not the case here. Hmm.

    EDIT: I've submitted a bug to Microsoft Connect. We'll see what happens to it.

    izokurew : This Assert fail: Assert.IsTrue(Decimal.MaxValue / 10m == 7922816251426433759354395033.5m);
    Jon Skeet : Yup, that's doing the constant check. Basically the compiler constant arithmetic is at fault here, I believe.
  • I'd guess because you are overstepping the max precision value when you are dividing by 10. http://en.wikipedia.org/wiki/Arithmetic_precision

    Jon Skeet : No, the precision is the same, as we're using a decimal type. It should easily be able to notice that the division is by zero, keep the mantissa the same and just change the exponent. It's perfectly possible to represent the exact result as a decimal.

0 comments:

Post a Comment