META - Backward Compatibility

[piux2]

We can use this as a starting point to track potential backward compatibility issues we need to be aware of, and devise solutions for each along the way.

1. Native function implementations.

Context

It's important to be aware that all native functions could have backward compatibility issues when we update their native implementation. This is not introduced by the new binding solution; instead, it makes the potential issue more apparent.

During contract execution, the functionValue.nativeBody,

gno/gnovm/pkg/gnolang/op_call.go

Line 63 in 831bb6f

fv.nativeBody = m.Store.GetNative(fv.NativePkg, fv.NativeName)

which is a func(*Machine) returned from iterating through nativeFuncs in stdlibs/native.go, by finding matching package and names,

gno/gnovm/stdlibs/native.go

Line 26 in 831bb6f

var nativeFuncs = [...]nativeFunc{

returns a matching func(*Machine) when the VM executes the contract.

For example, in the standard contract stdlib/std/native.gno, PrevRealm() > getRealm(1), the nativeFuncs links Gno's getRealm(1)

gno/gnovm/stdlibs/std/native.gno

Line 57 in 831bb6f

func getRealm(height int) (address string, pkgPath string)

with the native implementation of X_getRealm().

gno/gnovm/stdlibs/std/native.go

Line 70 in 831bb6f

func X_getRealm(m *gno.Machine, height int) (address string, pkgPath string) {

gno/gnovm/stdlibs/native.go

Line 535 in 831bb6f

r0, r1 := libs_std.X_getRealm(

Backward compatibility issues:

updating the logic in stdlibs/std/native.go could cause existing contracts to fail to execute if contract developers assume the behaviors of PrevRealm have changed in a new release.

[zivkovicmilos]

cc @ajnavarro

[ajnavarro]

Storage related backward-incompatibility problems

Serialization System Changes

Our backward compatibility is tied to the serialization format used to store data on disk. Any change to this format could introduce backward incompatibilities.

Payload Size Changes in Set Operations

Optimizations in the size of payloads stored during Set operations could lead to backward incompatibilities. Suppose the size of the data being stored in subsequent versions of the VM (even if that data is representing the same internal VM state, but is downsized using compression or similar) is reduced or altered for efficiency.

State format change

Any change to the internal VM state format could break backward compatibility. For instance, introducing a garbage collector (GC) or altering the way the VM represents its internal state could modify the amount of data stored.

Proposal

To mitigate the risk of backward-incompatible changes, I propose we decouple the storage costs from the exact byte size stored by the VM. Instead, each type—such as Interfaces, Structs, Ints, Strings, etc.—should be assigned a fixed cost based on properties intrinsic to the object itself.

For example, in the case of strings, we could calculate storage costs based on the length of the string (i.e., string length * X). This would ensure that the cost calculation is stable and not influenced by changes to how the VM stores data internally.

[Kouteki]

Conclusion of the team discussion:

1. Use best judgement on a case-by-case basis
2. Breaking compatility is not a deal breaker
3. We'll rely on chain upgrades and snapshots when it comes to it
4. YOLO

[thehowl]

It's important to be aware that all native functions could have backward compatibility issues when we update their native implementation. This is not introduced by the new binding solution; instead, it makes the potential issue more apparent.

Yes, furthermore they need to be reimplemented by any and all alternative VM implementations.

Native bindings are a necessary evil which make our lives simpler for writing static analysis tools, as we don't need to add shims or hardcode Gno's standard library functions, but they should still be kept to be as little as possible.

[thehowl]

One general comment, additionally;

let's keep track of potential backward compatibility pitfalls, yes, but I'm looking more to have:

• Strong tests which strictly verify our current expected behaviour.
• "User Flow" integration test #2159 can help us try typical user flows and ensure they always work.
• Portal loop CI testing against its genesis.json #2862 tries to prevent failures restarting the portal loop; I think eventually on mainnet we should have a long test (similar, conceptually, to this issue) which tries to replay the entire chain up to the point of the test with the modified code, showing us if the chain would fail with the updated code.

A collection like this can tell us where to look; proper tooling can help us point to real problems before they happen.

Also related: #2550