-
Notifications
You must be signed in to change notification settings - Fork 1.4k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Fix IDL #2824
Fix IDL #2824
Conversation
@acheroncrypto is attempting to deploy a commit to the coral-xyz Team on Vercel. A member of the Team first needs to authorize it. |
This looks great! Is the client backwards compatible with the old IDL when resolving external relations? Right now if you put a has_one on an account not from your program, if it has a published IDL it can still be resolved. Also any plans on resolution coming to rust? |
I haven't tried it but my intuition would say no because the old IDL is not compatible with the new client. All existing tests in the repo regarding relations work though. |
The spec (IDL version) field is very important for all of us that have various code generation tools in the wild. |
Yeah, would be great to support Rust too #2814. |
`solang` is currently not compatible with the new IDL spec
New IDL
This is a rewrite of the IDL which concerns:
The following sections serve as a way to provide an overview of the changes, however, it's not feasible to go over every single change. For this reason, only the larger changes to the public API will be in scope, and everything else, such as the implementation details, will be skipped.
Problem
Solana's programming model does not have any kind of IDL (or ABI) that can be used for interacting with programs easily. The Anchor IDL specification was created in order to fix this problem — while it managed to decrease friction and increase composability, the current iteration of the IDL has major problems that are not easily fixable without breaking changes.
IDL related issues are also the most reported issue by category in the Anchor repository.
What's new
There is a lot to talk about, we'll go over the main changes while briefly explaining why the change is necessary by comparing the old (before this PR) and the new (this PR) implementation.
Here is a comparison of the old and new IDL for this program.
Type definition files:
Address
The program address used to be stored in
metadata.address
field and was only populated after the program got deployed. That meant rebuilding a deployed program would cause the address information to get lost.Now,
address
is a required top-level field that's updated on each build.Metadata
metadata
field used to be an untyped object. Now, it has the following fields:name
andversion
fields used to be top-level fields but they are more fitting as a property ofmetadata
because they're not being used for program interactions.spec
: IDL specification version.description
: Description of the program.repository
: URL to the program's repository.dependencies
: Program dependencies. This is currently empty by default, however, including Anchor and Solana versions can be useful.contact
: Program contact information similar tosecurity.txt
.Potential fields to add:
origin
: Origin (which tool was used to generate the IDL)toolchain
: Toolchain information from[toolchain]
ofAnchor.toml
Accounts and events as type
Account and event type information used to be stored inside their own properties,
accounts
andevents
respectively. This created a problem where types weren't being found in some cases because type definitions didn't exist in thetypes
field.Now, all defined types are stored in
types
field, including account an event types.accounts
andevents
field still exist and they act as a pointer to the actual type definition:Discriminator
Discriminator data was not part of the IDL because Anchor discriminators are deterministic and are calculated by their type name. However, there are several problems with this approach:
In the new version, all instructions, accounts and events have a new field
discriminator
that is the byte representation of their discriminator:With this change, TypeScript client uses the discriminators from the IDL and no longer derives them separately.
Unit and tuple struct
Unit struct:
generates:
and tuple (unnamed) struct:
generates:
In TS package, unit structs are passed in as
{}
and tuple structs as[...]
similar to how enum variants work.Generics
Generic types and generic array length is supported both in the IDL and in TS package. Here is an example using both:
generates the following IDL type:
can be used as an instruction argument (or type field):
the instruction argument in the IDL:
calling from TS:
Serialization
One of the early decisions Anchor made was using
borsh
as the default serialization method. While it is still the most used serialization method in the Solana ecosystem, there are other serialization methods that are being used, such asbytemuck
for zero copy operations.The old IDL did not have any fields to represent which serialization format was being used, forcing all data to be de/serialized using
borsh
.In the new IDL, all types have a new field called
serialization
. The field defaults toborsh
and it's not stored if it's the default value in order to reduce duplication and save space.Current supported serialization methods are:
borsh
(default)bytemuck
bytemuckunsafe
For example, creating a zero copy struct:
generates:
Representation
Rust allows modifying memory representations of user-defined types with the
repr
attribute but this data was not stored in the IDL. For this reason, using anything other than the default representation was very hard to work with.In the new IDL, memory representation information is stored in the
repr
field. It has 3 properties:kind
Supported values:
rust
(default)c
transparent
For example:
generates:
align
Alignment must be a power of 2, for example:
generates:
packed
generates:
Generation
For starters, Anchor initially started generating IDLs by parsing the program source code. While this was very fast, it also had severe limitations for generation because everything needed to be parsed and implemented manually, and there was no way to evaluate expressions.
On Anchor
0.29.0
, a new IDL generation method was introduced (#2011) that allowed compiling program code in order to generate the IDL. However, this resulted in another problem — multiple ways to generate the IDL. This is a problem because:To solve this problem, a new generation method has been added that leverages both parsing and building. It can be thought of as the merge of the existing
idl-parse
andidl-build
features.Expression evaluation
A simple way to demonstrate expression evaluation is using the
#[constant]
attribute:With the old method (parsing):
and the new method (build):
Type alias resolution
Type alias support was added on Anchor
0.29.0
(#2637). However, aliases with generics were not supported:Although the new IDL specification supports type aliases as a user-defined type, all type aliases currently resolve to the actual type they hold. For example, when
OptionalElements
from above is used in a struct field:generates:
Resolving type aliases before serializing them into the IDL doesn't change the functionality but it makes working with types easier from the client side.
External types
One of the biggest pain points of the old IDL was the lack of support for external types. Some improvements have been made with the
idl-build
feature introduced in0.29.0
(#2011), however, it was only limited to a subset of Anchor programs that specifically hadidl-build
feature. For example, usingUnixTimestamp
fromsolana-program
:would not work with the
idl-build
feature. In fact,idl-build
wouldn't even work if theUnixTimestamp
type alias was defined inside our crate (#2640).On the other hand, with the new generation method, the above example generates:
This kind of resolution for non-Anchor crates is not perfect; currently only type aliases are supported but it can be extended to support structs and enums in the future.
To recap, any type can be used from other Anchor programs and some from non-Anchor crates. So what if you want to include a type that doesn't fit into the previous categories? This is where the next section, customization, comes in.
Customization
Customization refers to manually specifying how a type should be stored in the IDL instead of letting Anchor decide it. This is particularly useful with the New Type Idiom for external types.
For example, let's say you use
u256
from an external math library:Wrap it:
Implement
AnchorDe/Serialize
:Implement
IdlBuild
:Any type can be included in the IDL as long as it can be represented in the IDL.
Case conversion
Lack of consistent casing has caused a lot of problems for Anchor developers, and the IDL is not an exception. Even in the TypeScript library, some things require using camelCase while others require using PascalCase or snake_case.
The internals of the TS library are filled with case conversion logic before making string comparison and this also forces other libraries who build on top of Anchor to do the same.
Case conversion becomes inevitable when something is being used cross-language, especially when the languages (Rust, JS) have strictly different conventions on this topic. However, it is still possible to make it consistent so that people know what to expect. Here are the main principles this PR follows regarding case conversion:
my_instruction
is stored asmy_instruction
instead ofmyInstruction
.pubkey
(renamed frompublicKey
),struct
,borsh
.IDL
constant is no longer exported fromtarget/types
(the type export remains).Account resolution
Account resolution refers to the ability of clients to resolve accounts without having to manually specify them when sending transactions. This feature has been supported behind
seeds
feature flag for a while now but most projects either don't use it or don't even know it exists.address
fieldInstruction accounts have a new field called
address
that is used to store constant account public keys. Currently the following methods are supported:#[account(address = <>)]
constraintAddress of the
Program<T>
typeAddress of the
Sysvar<T>
typeExample
generates:
pda
fieldThis field is generated from PDAs that are validated with the
seeds
constraint. It's not a new field, it works similar to how it used to work with slight changes and many bug fixes.Example
generates:
relations
fieldThis field is generated from the
has_one
constraint. It's not a new field, however, its behavior has been modified — therelations
field is now being stored in the account that is getting resolved (it used to be the reverse).Example
generates:
Resolution in TypeScript
There are too many changes to the account resolution logic in the TS library, however, we can skip a good chunk of them since they're mostly internal.
One change that affects everyone is the change in the
accounts
method. Even though the TS library had some support for account resolution, it had no type-level support for it — all accounts were essentially typed as partial, and there was no way to know which accounts were resolvable and which were not.There are now 3 methods to specify accounts with the transaction builder:
accounts
: This method is now fully type-safe based on the resolution fields in the IDL, making it much easier to only specify the accounts that are actually needed.accountsPartial
: This method keeps the old behavior and let's you specify all accounts including the resolvable ones.accountsStrict
: If you don't want to use account resolution and specify all accounts manually (unchanged).Another change that affects most projects is the removal of "magic" account names. The TS library used to autofill common program and sysvar accounts based on their name, e.g.
systemProgram
, however, this is no longer necessary with the introduction of theaddress
field which is used to resolve all program and sysvars by default.resolution
featureAlong with the above changes, the
seeds
feature has been renamed toresolution
and is now enabled by default unless explicitly set tofalse
inAnchor.toml
.Downsides
While the vast majority of changes are advantageous, there are some downsides to consider.
Generation times
Rust is not known for its super-fast compile times. The current generation times are not even comparable to the old generation method (parse) because the current method requires building a binary while the old method only parses the files which was almost instant.
There are a few possible ways to reduce the effect of this problem:
anchor build
command by defaultanchor build
IDL size
The new IDL stores significantly more data than it used to which results in larger IDL sizes.
In order to offset some of the increase in the IDL size, most fields in the IDL are skipped on serialization, including
bool
fields likesigner
:Existing tooling
A great number of dev-tools in the Solana ecosystem depends on IDLs, these tools will need to be updated in order to be compatible with the new IDL.
Feedback
If you used Anchor IDL's in any sort of way and ran into problems in the past, this is a great time to try the new one and give feedback.
Here is the list of things that would be great to have feedback on:
Fixes
This PR resolves 47 open issues and 8 PRs with regards to the IDL.
Resolves #22, #45, #192, #232, #279, #325, #607, #617, #632, #674, #723, #736, #780, #785, #896, #899, #904, #912, #959, #971 (partial), #1058, #1190, #1213, #1458, #1513, #1550, #1560, #1566, #1641, #1680, #1830, #1849, #1859, #1927, #1971, #1972, #2058, #2104, #2138, #2187, #2286, #2349, #2431, #2441, #2442, #2531, #2545, #2625, #2640, #2653, #2687, #2688, #2710, #2748, #2788.