DecimalBigNum values are not handled correctly.

[whscullin]

What happens?

Numeric values from duckdb-wasm can be returned in several formats, including Arrow's DecimalBigNum. When a value with a decimal point is returned via DecimalBigNum, it loses its decimal point.

To Reproduce

With this snippet, where connection is a DuckDB wasm connection:

  const sql = `SELECT
  1.234 as "n1",
  1234.0*1.0/1000 as "n2"
  `;
  const results = await connection.query(sql);
  console.log(results.toArray());

the values returned become:

[ {"n1": 1234, "n2": 1.234} ]

n1, as a constant, has lost its decimal point. It is returned as a DecimalBigNum. n2 has the correct decimal placement, and it is returned as a Javascript number. If you do any computation with n1, such as 1.234/1, it gets returned as a number and is correct.

This may be an Arrow issue with a mismatch between the IPC serialization of of DecimalBigNum between Rust and Javascript, but I don't have an easy way to test that, yet.

Browser/Environment:

node v18.16.0

Device:

MacBook Pro

DuckDB-Wasm Version:

1.28.1-dev179.0

DuckDB-Wasm Deployment:

Malloy VS Code Extension

Full Name:

Will Scullin

Affiliation:

Google

[domoritz]

There are issues with decimals in arrow in the current version. We started fixing it in the current main branch and I hope to make some time to fix the remaining issues before the next release.

Please use float for now.

[whscullin]

That's not something we can really control. We may have some ability to automatically cast some numbers to float, but we don't control all the input SQL, and there's no way to programmatically force floats in the output that I've found.

[domoritz]

Let's fix how Arrow handles decimal for sure. If you can help out, that would be great or you can follow the respective issues over on the arrow repo. I just described the plan in apache/arrow#37920 (comment).