Support for uint256/int256

[elyase]

Description

The most commonly used data types for smart contracts and token math on the blockchain are uint256 and int256. Currently, people are resorting to inefficient methods like converting to float or string because there is no native support for these data types. For example, paradigmxyz/cryo exports duplicate data columns in various formats (float, binary, string) due to the absence of native support.

[banteg]

willing to add a bounty of $500 to the implementer of this feature. could be more when the scope of work is more clear. i guess we'll need support in arrow first.

found this library that implements arbitrary sized uint: https://github.com/recmo/uint

[stinodego]

I think we want to support a Decimal backed by an Int256 in the future- this is also available in Arrow.

Not sure about a true 256-bit integer. We should probably first debate adding Int128/UInt128 with the 256-bit version as a potential next step. I don't think this will happen any time soon. Just my 2 cents though.

[l1t1]

two many things will be effect if adding new tyoes. ref duckdb/duckdb#8635 (comment)

[banteg]

unfortunately decimal256 in arrow doesn't support the full range of uint256 values, it cuts off at 76 digits, which is approximately 252.4653 bits.

import pyarrow as pa

pa.array([2**256-1], type=pa.decimal256(76, 0))
# ArrowInvalid: Decimal type with precision 78 does not fit into precision inferred from first array element: 76

pa.array([2**256-1], type=pa.decimal256(78, 0))
# ValueError: precision should be between 1 and 76

[0xvanbeethoven]

Willing to add $500 to the bounty @banteg proposed.

[gakonst]

@paradigmxyz (maintainers of Cryo) to chip in an additional $1K, incl. the Arrow integration in scope.

[orlp]

What sort of operations would one expect to be able to do on this pl.UInt256/pl.Int256, and for what purposes (preferably with concrete examples)?

For the record, this issue is not yet accepted, we still need to discuss if we want this in Polars. Adding types is a lot of work, slows down future feature development, and increases the binary size of Polars by quite a bit. So implementers beware: even though third parties are offering PR bounties, that does not mean we will merge said PR's and you might not get paid if we end up deciding this is not in scope for Polars, or the drawbacks outweigh the positives.

[sslivkoff]

What sort of operations would one expect to be able to do on this pl.UInt256/pl.Int256, and for what purposes (preferably with concrete examples)?

For the record, this issue is not yet accepted, we still need to discuss if we want this in Polars. Adding types is a lot of work, slows down future feature development, and increases the binary size of Polars by quite a bit. So implementers beware: even though third parties are offering PR bounties, that does not mean we will merge said PR's and you might not get paid if we end up deciding this is not in scope for Polars, or the drawbacks outweigh the positives.

hi

in terms of raw operations I think the most important functionality would be

• arithmetic ops + - * / % ^
• comparison == >= > < <=
• min/max/mean/value_counts
• cum_sum / diff

these operations would be most commonly used in these contexts

• pl.col() expressions in select() and with_columns()
• inside .group_by(non_int_column).agg(int_column)
• bare Series

concrete example: lets say you have a large dataframe of transactions (>100M rows). each row has a u256 column of tx price and a string/binary column of account id. you want to aggregate the total spend per account. right now the most common approach is to convert to f64 for tx price so that group_by(id).agg(pl.sum(price)) can be used. using f64 sacrifices precision because the data is natively u256 and often utilizes the full precision

[gakonst]

Totally @orlp definitely no expectation from our side and appreciate the clear communication. Makes sense and understand the nuances w.r.t maintenance, as we also follow a similar philosophy. Thank you for the swift reply.

[yenicelik]

happy to chip in another $100 to this bounty

[IlluvatarEru]

This is a blocker for a lot of people and orgs, would be great to see it priorised. What can we do see it done faster?