Subject: Re: [xsl] floor() and idiv From: "Andrew Welch" <andrew.j.welch@xxxxxxxxx> Date: Thu, 6 Nov 2008 20:30:06 +0000 |
2008/11/6 Colin Paul Adams <colin@xxxxxxxxxxxxxxxxxx>: >>>>>> "Andrew" == Andrew Welch <andrew.j.welch@xxxxxxxxx> writes: > > >> >> Gestalt gives 10, 10, 10. > >> > Andrew> As far as I understand, which is not very far, it's a > Andrew> fundamental problem with binary arithmetic. The standard > Andrew> division technique is fast but incurs this sort of > Andrew> problem, while the BigDecimal approach is exact but much > Andrew> slower. > > You've lost me - we were talking about integer division, I thought. 1000000000000000000000000000000001 is just too big for an 32 bit integer... (I think that's why DC chose it) > If you are talking about xs:double, compared with xs:decimal, then > yes. But I don't see how that comes into the problem. I think it's the problem of representing a decimal: http://speleotrove.com/decimal/decifaq1.html#inexact -- Andrew Welch http://andrewjwelch.com Kernow: http://kernowforsaxon.sf.net/
Current Thread |
---|
|
<- Previous | Index | Next -> |
---|---|---|
Re: [xsl] floor() and idiv, Colin Paul Adams | Thread | Re: [xsl] floor() and idiv, Colin Paul Adams |
Re: [xsl] floor() and idiv, Colin Paul Adams | Date | Re: [xsl] floor() and idiv, Colin Paul Adams |
Month |