How to extract the correct rust types from Sui ?

[Sceat]

In efforts of having a working indexer in javascript, I'm generating BCS types from this file https://github.com/MystenLabs/sui/blob/testnet-v1.34.0/crates/sui-core/tests/staged/sui.yaml

The problem is many of them are missing and some are also incorrect, like below where TransactionEvents is replaced by TransactionEventsDigest

// generated from the yaml file above
const CheckpointTransaction = bcs.struct('CheckpointTransaction', {
  transaction: Transaction,
  effects: TransactionEffects,
  events: bcs.option(TransactionEventsDigest),
  input_objects: bcs.vector(SuiObject),
  output_objects: bcs.vector(SuiObject),
})

// expected correct bcs
const CheckpointTransaction = bcs.struct('CheckpointTransaction', {
  transaction: Transaction,
  effects: TransactionEffects,
  events: bcs.option(TransactionEvents),
  input_objects: bcs.vector(SuiObject),
  output_objects: bcs.vector(SuiObject),
})

How can we get a consistent types file after each release ? (used for https://github.com/aresrpg/sui-checkpoint-reader/)

[amnn]

@bmwill, @phoenix-o -- any idea what the best way is to make sure that someone can generate the necessary BCS bindings for the format that data ingestion reads from?

IIRC, sui.yaml is generated from types in the core protocol, but it may not include the types used by data ingestion, so there may be some discrepancies among types that share the same name. Can we generate a similar YAML file for the full checkpoint data?

[bmwill]

#19516 should address this

[Sceat]

Dope @bmwill !

While we are on the topic of bcs types, is the SDK up to date about type fetching ? In order to generate bcs types from packages (https://github.com/Sceat/sui-bcs) I'm using getNormalizedMoveModulesByPackage which output any enum as a type.Struct which indeed fails when using getNormalizedMoveStruct

JsonRpcError: No struct was found with struct name ItemCategory

[Sceat]

Hi @bmwill could you also include those ? It's needed to parse snapshots

export const LiveObject = bcs.enum('LiveObject', {
  Normal: SuiObject,
  Wrapped: ObjectKey,
})

export const Manifest = bcs.enum('Manifest', {
  V1: ManifestV1,
})

actually @amnn it would be so much easier to have those bcs types included in the ts SDK

[Sceat]

also saw that type_params was renamed to type_args ? where can we be informed of such breaking changes

[amnn]

it would be so much easier to have those bcs types included in the ts SDK

The ideal place for this would be a separated SDK specific to data ingestion, in TypeScript, similar to our sui-data-ingestion-core crate in Rust but we just don't have the cycles to commit to it at the moment -- most of our in-house indexers are built in Rust.

also saw that type_params was renamed to type_args ? where can we be informed of such breaking changes

I looked into this, it seems like this change is unintentional, and an artifact of how the sui.yaml file is generated (@bmwill, I'm not sure if you have more context on why it would do this), the type being serialized there did change (StructTag -> StructInput for types in transactions), but the name of the field on our side did not change:StructTag and StructInput define that field as being type_args with a possible alias of type_params:

sui/crates/sui-types/src/type_input.rs

Lines 20 to 22 in 76d629e

	// alias for compatibility with old json serialized data.
	#[serde(rename = "type_args", alias = "type_params")]
	pub type_params: Vec<TypeInput>,

sui/external-crates/move/crates/move-core-types/src/language_storage.rs

Lines 149 to 151 in 76d629e

	// alias for compatibility with old json serialized data.
	#[serde(rename = "type_args", alias = "type_params")]
	pub type_params: Vec<TypeTag>,

Note that this is not a breaking change in the BCS schema per se, because the BCS schema is not name sensitive, but the best way to watch out for changes to the types you care about is to look for diffs in this file:

https://github.com/MystenLabs/sui/blob/76d629e7765e4220342f94e7438ebd17cf22c5a7/crates/sui-core/tests/staged/sui.yaml