Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

round() ignores the innate precision of a real number #392

Closed
jonhanke opened this issue Jun 28, 2007 · 1 comment
Closed

round() ignores the innate precision of a real number #392

jonhanke opened this issue Jun 28, 2007 · 1 comment

Comments

@jonhanke
Copy link

The function round() seems to ignore precision information slightly beyond the default 53 bits for a real number. This leads to some incorrect rounding results for close calls.

sage: a = 5.499999999999999 
sage: a.prec()
56
sage: round(a)  ## This is ok
5.0

sage: b = 5.4999999999999999
sage: b.prec()
59
sage: round(b)  ## This isn't ok 
6.0

Component: algebraic geometry

Keywords: round, real, precision, arithmetic

Issue created by migration from https://trac.sagemath.org/ticket/392

@jonhanke
Copy link
Author

Changed keywords from round, real, arithmetic to round, real, precision, arithmetic

@williamstein williamstein added this to the sage-2.8.2 milestone Aug 18, 2007
tobiasdiez pushed a commit to tobiasdiez/sage that referenced this issue Feb 22, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

2 participants