Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

BigInteger in Neo.Json #3037

Open
wants to merge 20 commits into
base: HF_Echidna
Choose a base branch
from
Open

BigInteger in Neo.Json #3037

wants to merge 20 commits into from

Conversation

shargon
Copy link
Member

@shargon shargon commented Dec 27, 2023

Close #3036

@shargon shargon added the Bug Used to tag confirmed bugs label Dec 27, 2023
@shargon
Copy link
Member Author

shargon commented Dec 27, 2023

During this PR an error was found, a number greater than MAX_SAFE_INTEGER or less than MIN_SAFE_INTEGER could be parsed before.

Open question: We want this restriction only for serialize?

@Jim8y
Copy link
Contributor

Jim8y commented Dec 27, 2023

This will change the behavior of existing contracts as well.

@cschuchardt88
Copy link
Member

cschuchardt88 commented Dec 27, 2023

I was expecting something other than the double type; for example all integer and floating point types. Why are we using double only anyways?

@shargon
Copy link
Member Author

shargon commented Dec 27, 2023

This will change the behavior of existing contracts as well.

Yes, we need a fork, but currently we allow integers out of the range

@Jim8y
Copy link
Contributor

Jim8y commented Dec 27, 2023

Yes, we need a fork, but currently we allow integers out of the range

Its not about a fork, it is that you have changed the behavior of existing contract. Which means the logic in existing contract may become invalid or different, and cause unexpected execution result.

@cschuchardt88
Copy link
Member

cschuchardt88 commented Dec 27, 2023

This PR isnt going to work seeing how big integers can be 256-bits. double isn't that big.

Comment on lines 36 to 41
public void TestBigInteger()
{
((JNumber)BigInteger.One).AsNumber().Should().Be(1);
((JNumber)BigInteger.Zero).AsNumber().Should().Be(0);
((JNumber)BigInteger.MinusOne).AsNumber().Should().Be(-1);
}
Copy link
Member

@cschuchardt88 cschuchardt88 Dec 27, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

When doing testing, use max and min values please. You will see that using the max or min will fail.... double is to small to hold the data. BigInteger is 32 BYTES and double is 8 BYTES you get overflows. Its like doing

var failed = checked((int)long.MaxValue);

image

image

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ping @shargon

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Added ut with min and max

@shargon
Copy link
Member Author

shargon commented Dec 27, 2023

Yes, we need a fork, but currently we allow integers out of the range

Its not about a fork, it is that you have changed the behavior of existing contract. Which means the logic in existing contract may become invalid or different, and cause unexpected execution result.

Any idea? currently we allow bigger values than expected

@Jim8y
Copy link
Contributor

Jim8y commented Dec 27, 2023

Any idea? currently we allow bigger values than expected

We need to track the version or the deploy or latest update block of contracts, if it is before your fix, it should run as before, otherwise, it can adopt new logic. Otherwise, a bug should be a bug,,,, we should not fix it.

@cschuchardt88
Copy link
Member

cschuchardt88 commented Dec 27, 2023

we should not fix it

What do you mean not fix it? The bug?

@Jim8y
Copy link
Contributor

Jim8y commented Dec 27, 2023

we should not fix it

What do you mean not fix it? The bug?

We should not fix a bug that is already there in the chain, at least not in a way that can change the behavior of existing contract. Cause our update may cause unexpected behavior of those contracts. Take this one for example, it changed the behavior of the deserialize interface, if a contract uses it to deserialize a very large number, it works in the past, so he just write the contract that way and core logics rely on the deserialize operation, what will happen when this pr is patched? his contract may fail, may also have who knows behavior cause he might have wrapped it in a try block.

@cschuchardt88
Copy link
Member

cschuchardt88 commented Dec 27, 2023

But take this in to count
image

Im sure no one has reached the max value yet, but see what can happen. so needs to be fixed.

@Jim8y
Copy link
Contributor

Jim8y commented Dec 28, 2023

Again, you dont fix it in a way that change the existing contract behavior. And you dont ensure someone has reached the maximum or not cause you dont know all contracts deployed. If you keep make changes like this, the neo project will be ruined one day cause you keep changing the existing contact behavior.

@cschuchardt88
Copy link
Member

You playing with people's potential money they have invested. This could be bad if a contract is doing StdLib.JsonSerialize and StdLib.JsonDeserialize for objects that deal with real funds or NFTs. Now developers use string as the json output, mostly. This will break oracle transactions, which are now bugged. I know this isn't a good solution, hints I said This PR isn't going to work.

This should of been developed correctly from the begin, the whole reason why we don't use floating points on the blockchain.

@shargon
Copy link
Member Author

shargon commented Dec 28, 2023

@Jim8y is right, a fork don't solve that the contract was not updated, maybe we need to store in the contract the last updated/deploy height, and use it as a fork height

@Jim8y
Copy link
Contributor

Jim8y commented Dec 28, 2023

You playing with people's potential money they have invested. This could be bad if a contract is doing StdLib.JsonSerialize and StdLib.JsonDeserialize for objects that deal with real funds or NFTs. Now developers use string as the json output, mostly. This will break oracle transactions, which are now bugged. I know this isn't a good solution, hints I said This PR isn't going to work.

This should of been developed correctly from the begin, the whole reason why we don't use floating points on the blockchain.

We are not playing around, it is a very serious problem. The very fundamental bottom line is we do not change the behavior of existing contracts, we must defence it firmly. Determinastic is one of the key feature of the blockchain, you break it, you break everything. I am not saying we should ignore it, but we should not fix it in a way that can potentially destroy the whole project. Reputation is the key in blokchain world, not a single project will want to exist in a chain that can change its contract's behavior arbitarily.

@cschuchardt88
Copy link
Member

I understand now. I didn't know what you meant by not fix it. Now I do. Thx for clearing that up.

@vncoelho
Copy link
Member

vncoelho commented Dec 28, 2023

@Jim8y is right, a fork don't solve that the contract was not updated, maybe we need to store in the contract the last updated/deploy height, and use it as a fork height

Perhaps this is a possible solution @shargon.
If a field is added to all deployed contracts and they keep calling the "legacy" version.
A mechanism can be created for those that want to remove this "legacy" flag.
Maybe "ContractManagement" could handle that.

@cschuchardt88
Copy link
Member

cschuchardt88 commented Dec 28, 2023

Let's not forget about oracle service as well. This would of affected any variable in the contract that they set a valid too.

@@ -50,10 +50,7 @@ public static JToken Serialize(StackItem item)
}
case Integer num:
{
var integer = num.GetInteger();
if (integer > JNumber.MAX_SAFE_INTEGER || integer < JNumber.MIN_SAFE_INTEGER)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hello, #2498.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

He put it in the JNumber. Different exception, but will fail as well.

Copy link
Member Author

@shargon shargon Dec 29, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It was moved to the constructor, that was the bug, when we used parse we allowed it...

@Jim8y
Copy link
Contributor

Jim8y commented Feb 15, 2024

@superboyiii, please check the state compatibility.

@Jim8y Jim8y requested a review from superboyiii February 15, 2024 09:25
@Jim8y
Copy link
Contributor

Jim8y commented Feb 15, 2024

@shargon format not a thing i can update, please check it.

@superboyiii
Copy link
Member

superboyiii commented Feb 20, 2024

@shargon @Jim8y Something is incompatible. Blockchain stopped syncing at block 1528989 on mainnet.
1708423247018
1708423292017

@cschuchardt88 cschuchardt88 added Blocked This issue can't be worked at the moment and removed Waiting for Review labels May 16, 2024
@Jim8y
Copy link
Contributor

Jim8y commented Jun 1, 2024

@shargon @Jim8y Something is incompatible. Blockchain stopped syncing at block 1528989 on mainnet. 1708423247018 1708423292017

Please consider close this pr. @shargon

@shargon
Copy link
Member Author

shargon commented Jun 1, 2024

@shargon @Jim8y Something is incompatible. Blockchain stopped syncing at block 1528989 on mainnet. 1708423247018 1708423292017

Please consider close this pr. @shargon

But it can't be related to this pr

@Jim8y Jim8y changed the base branch from master to HF_Echidna August 7, 2024 16:28
@Jim8y Jim8y added the Hardfork label Aug 7, 2024
@cschuchardt88
Copy link
Member

@shargon do you can start working on this again? If I remember we needed to change the vm as well. Since we are working on the vm now.

@Jim8y
Copy link
Contributor

Jim8y commented Nov 19, 2024

@shargon conflicts, lets add hardfork if possible and make it merged into the hardfork pr. Such that no more conflict to maintain.

/// <returns>The serialized object.</returns>
public static JToken Serialize(StackItem item)
public static JToken Serialize(StackItem item, Func<Hardfork, bool> hardforkChecker = null)
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@neo-project/core This methods seems that was only used in UT...

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Blocked This issue can't be worked at the moment Bug Used to tag confirmed bugs Hardfork
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Use BigInteger in Neo.Json
6 participants