Skip to content

Commit

Permalink
apd: don't strip trailing zeros in Quo and Sqrt
Browse files Browse the repository at this point in the history
Before this commit, `Quo` and `Sqrt` were stripping trailing zeros from
their result. This was different from the behavior in PostgreSQL, where
these operations retain all trailing zeros based on the precision of
the computation:

In PostgreSQL:
```sql
nathan=# select d, v from dec;
    d    |  v
---------+-----
      10 |   4
 10.0000 | 4.0
(2 rows)

nathan=# select d/v, scale(d/v), trim_scale(d/v) from dec;
      ?column?      | scale | trim_scale
--------------------+-------+------------
 2.5000000000000000 |    16 |        2.5
 2.5000000000000000 |    16 |        2.5
(2 rows)

nathan=# select sqrt(v), scale(sqrt(v)), trim_scale(sqrt(v)) from dec;
       sqrt        | scale | trim_scale
-------------------+-------+------------
 2.000000000000000 |    15 |          2
 2.000000000000000 |    15 |          2
(2 rows)
```

Making this change will improve PG compatibility, especially when we
decide to implement the `scale` and `trim_scale` builtin functions.

Conveniently, removing this also improves performance, as these operations
now perform less work.

```
name                 old time/op    new time/op    delta
GDA/exp-10             20.6ms ± 0%    16.2ms ± 1%  -21.21%  (p=0.000 n=10+10)
GDA/log10-10           25.6ms ± 0%    22.7ms ± 0%  -11.57%  (p=0.000 n=10+9)
GDA/ln-10              19.9ms ± 0%    17.6ms ± 0%  -11.35%  (p=0.000 n=10+10)
GDA/divide-10          65.3µs ± 0%    59.4µs ± 0%   -9.03%  (p=0.000 n=9+9)
GDA/squareroot-10      5.31ms ± 0%    5.01ms ± 0%   -5.57%  (p=0.000 n=10+8)
GDA/powersqrt-10       73.9ms ± 0%    71.3ms ± 0%   -3.48%  (p=0.000 n=9+9)
GDA/cuberoot-apd-10     397µs ± 0%     385µs ± 0%   -3.05%  (p=0.000 n=10+10)
GDA/divideint-10       13.4µs ± 0%    13.4µs ± 0%     ~     (p=0.859 n=9+10)

name                 old alloc/op   new alloc/op   delta
GDA/divide-10          10.6kB ± 0%    10.6kB ± 0%     ~     (all equal)
GDA/divideint-10        96.0B ± 0%     96.0B ± 0%     ~     (all equal)
GDA/cuberoot-apd-10     122kB ± 0%     124kB ± 0%   +1.81%  (p=0.000 n=10+10)
GDA/powersqrt-10       6.27MB ± 0%    6.66MB ± 0%   +6.22%  (p=0.000 n=10+10)
GDA/log10-10           10.4MB ± 0%    11.4MB ± 0%  +10.03%  (p=0.000 n=10+10)
GDA/ln-10              7.93MB ± 0%    8.74MB ± 0%  +10.22%  (p=0.000 n=10+10)
GDA/squareroot-10       147kB ± 0%     170kB ± 0%  +15.82%  (p=0.000 n=10+10)
GDA/exp-10             9.56MB ± 0%   11.26MB ± 0%  +17.71%  (p=0.000 n=10+10)

name                 old allocs/op  new allocs/op  delta
GDA/divide-10             213 ± 0%       213 ± 0%     ~     (all equal)
GDA/divideint-10         2.00 ± 0%      2.00 ± 0%     ~     (all equal)
GDA/cuberoot-apd-10     2.43k ± 0%     2.48k ± 0%   +1.81%  (p=0.000 n=10+10)
GDA/powersqrt-10         210k ± 0%      217k ± 0%   +3.53%  (p=0.000 n=10+10)
GDA/squareroot-10       3.93k ± 0%     4.18k ± 0%   +6.36%  (p=0.000 n=10+10)
GDA/log10-10             205k ± 0%      220k ± 0%   +7.46%  (p=0.000 n=8+10)
GDA/ln-10                157k ± 0%      169k ± 0%   +7.67%  (p=0.000 n=10+10)
GDA/exp-10               127k ± 0%      144k ± 0%  +13.27%  (p=0.000 n=10+10)
```
  • Loading branch information
nvanbenschoten committed Jan 31, 2022
1 parent 94925d2 commit 04d1483
Show file tree
Hide file tree
Showing 2 changed files with 4 additions and 5 deletions.
2 changes: 0 additions & 2 deletions context.go
Original file line number Diff line number Diff line change
Expand Up @@ -362,7 +362,6 @@ func (c *Context) Quo(d, x, y *Decimal) (Condition, error) {
}

res |= d.setExponent(c, nd, res, shift, -adjCoeffs, -adjExp10)
d.Reduce(d) // remove trailing zeros
return c.goError(res)
}

Expand Down Expand Up @@ -552,7 +551,6 @@ func (c *Context) Sqrt(d, x *Decimal) (Condition, error) {
d.Exponent += int32(e / 2)
nc.Precision = c.Precision
nc.Rounding = RoundHalfEven
d.Reduce(d) // remove trailing zeros
res := nc.round(d, d)
return nc.goError(res)
}
Expand Down
7 changes: 4 additions & 3 deletions example_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -56,9 +56,10 @@ func ExampleContext_inexact() {
return
}
}
// Output: d: 9, inexact: false, err: <nil>
// d: 3, inexact: false, err: <nil>
// d: 1, inexact: false, err: <nil>
// Output:
// d: 9.0000, inexact: false, err: <nil>
// d: 3.0000, inexact: false, err: <nil>
// d: 1.0000, inexact: false, err: <nil>
// d: 0.33333, inexact: true, err: <nil>
}

Expand Down

0 comments on commit 04d1483

Please sign in to comment.