-
Notifications
You must be signed in to change notification settings - Fork 1.2k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Number of significant figures displayed when converting floats to base 5 with (float).toString(5) differs from other engines #2512
Comments
Looks like our results are half-way between v8 and jsc/sm. I haven't consulted the spec yet. |
Dat precision tho |
In base 5, |
It looks like d8 has updated their rounding behavior:
|
It does look like our rounding behavior differs from other engines:
For powers of two where there is no rounding necessary we do the same thing as the others, but for other bases we often differ. I initially thought that we were rounding down rather than up in these cases, but that's not always true, for example using base 11 above. In fact, it somewhat looks like for bases up to 10 we are rounding down when others round up, and for bases greater than 10 we are displaying more digits / not rounding when others do. |
I do see discrepancies in string representation between Chrome and SpiderMonkey (though not as big ones as between Chrome and Chakra). Tuned our algorithm a little bit so the result matches Chrome's more closely. |
Tested with the following:
(0.88).toString(5)
(1.88).toString(5)
(10.88).toString(5)
and the results were:
0.420000000000000000000004
1.41444444444444444444443
20.4200000000000000000001
The result for (1.88).toString(5) seems to be incorrect. Was expecting something closer to 1.42
The text was updated successfully, but these errors were encountered: