Update Solana SDK to 1.16 and Anchor to the latest change including it

[vadorovsky]

The newest solana-program crate was pulled unintentionally anyway which resulted in unexpected errors due to incompatibility of dependencies. More context: coral-xyz/anchor#2512

[ananas-block]

why is program_id changed to __program_id?

[vadorovsky]

Because Anchor did so. We are not defining that variable ourselves, it's being generated inside #[program] attribute macro.

The change was made in this commit:

coral-xyz/anchor@2e89b79

[ananas-block]

I see thanks

[ananas-block]

is padding necessary because of repr c?

[vadorovsky]

It's necessary in zero-copy accounts. It was honestly our little f***-up that we didn't do it before. Fortunately, now bytemuck is smart enough to check for padding inside macros (bytemuck::Pod, which is getting used by anchor_lang::zero_copy), so I added it, because of the compilation error.

[vadorovsky]

If you want more context - bytemuck has this check:

https://github.com/Lokathor/bytemuck/blob/df5262b19ca4fde16bc1a15eecb7438f0ecdbe42/derive/src/traits.rs#L540C1-L544

which checks whether your struct and [u8; LEN_OF_ALL_YOUR_DEFINED_FIELDS] are transmuttable (which requires them to be of the equal size). Failure of this check means that your struct is not aligned and that the compiler added the padding. To pass this check, you need to define the padding yourself, so the compiler doesn't do it for you.

Non-explicit (autogenerated by compiler) padding is problematic in bytemuck, because Rust doesn't initialize the memory used for padding. And bytemuck uses unsafe code to use your structs as arrays of bytes. Having uninitialized memory there might end up in undefined behavior.

When you have

pub struct A {
    pub foo: u32, // 4 bytes; it's the biggest field, so the whole struct has to be aligned to 4 bytes
    pub bar: u8, // 1 byte
    _padding: [u8; 3], // 3 bytes for alignment
   // 4 + 1 + 3 = 8
   // 8 | 4
}

pub struct B {
    pub foo: u32,
    pub bar: u8,
    // compiler will add 3 bytes of uninitialized memory here
    // 5 bytes are safe to access, reading last 3 bytes with `unsafe` is undefined behavior
}

Bytemuck (which is used by Anchor, when doing zero-copy) is going to treat both structures as [u8; 8], doing unsafe stuff on it, so accessing uninitialized bytes of B is undefined behavior.

[ananas-block]

Ok makes sense. I was aware that the padding was not ideal before, that's why I put the two u8 variables last this way it did not mess with the overall alignment of the other variables.

And the following wouldn't work because bytemuck treats both variables ad [u8;8] right?

        pub number_of_leaves: u8,
        pub insert_leaves_index: u8,
        _padding: [u8; 6],
}

[vadorovsky]

Exactly, that wouldn't work, because padding has to be added explicitly to each field (so they match the biggest field with size). So both u8 fields have to be filled up until 8 bytes.