Proposal for subxt-python

[philoniare]

Project Abstract

subxt-python is a one-to-one wrapper library of the official subxt library, built with PyO3

Grant level

• Level 1: Up to $10,000, 2 approvals
• Level 2: Up to $30,000, 3 approvals
• Level 3: Unlimited, 5 approvals (for >$100k: Web3 Foundation Council approval)

Application Checklist

• The application template has been copied and aptly renamed (project_name.md).
• I have read the application guidelines.
• Payment details have been provided (Polkadot AssetHub (DOT, USDC & USDT) address in the application and bank details via email, if applicable).
• I am aware that, in order to receive a grant, I (and the entity I represent) have to successfully complete a KYC/KYB check.
• The software delivered for this grant will be released under an open-source license specified in the application.
• The initial PR contains only one commit (squash and force-push if needed).
• The grant will only be announced once the first milestone has been accepted (see the announcement guidelines).
• I prefer the discussion of this application to take place in a private Element/Matrix channel. My username is: @_______:matrix.org (change the homeserver if you use a different one)

[PieWol]

Hey @philoniare ,
thanks for the application.

As you mentioned the py-substrate-interface already exists as a python library. I am really interested in the reason why you think a new python library as a direct mirror of the subxt specification would be so superior to the existing python library. Would you mind to elaborate on this point? Thank you 👍

[philoniare]

@PieWol thanks for the question, I've asked myself the exact same question when I was thinking of this proposal. Here's my rationale, but I think you guys are better positioned to validate if the idea is good or not as you have a better bigger picture in mind of the ecosystem.

Better positioned for long-term maintenance and upgradeability:

• Consistency with subxt -> allows devs familiar with subxt to easily transition to Python and vice verse, reducing the learning curve and enabling seamless collaboration between Rust and Python teams. Allows small teams to quickly iterate on the Python version and seamlessly migrate to Rust edition when in need of optimized production performance
• subxt is actively developed and maintained by the Parity team, incorporating the latest optimizations, performance improvements, and features. By building on top of subxt, subxt-python inherits these benefits, such as the type-safe API, compile-time checks, and efficient data serialization and deserialization.
• By leveraging the power of Rust through PyO3 bindings, subxt-python can achieve better performance compared to a pure Python implementation, like py-substrate-interface.
• Building subxt-python as a direct mirror of subxt allows for easier maintenance and updates, as the Rust code serves as a reference implementation. By aligning with subxt, subxt-python can benefit from this community support, ensuring long-term sustainability and accessibility to a larger developer base.

Better feature set:

• Better type hints as it mirrors the subxt. py-substrate-interface seems lacking to me in this regards, for example, for GenericCall type there's no docs and the dev is not sure what data to pass in. They would mainly go through tutorials in order to find what to pass in.
• Big advantage of subxt is the codegen feature that allows generating Rust code from the metadata of a Substrate node. This allows communicating with the node without manual string calls like in py-substrate-interface: result = substrate.query('System', 'Account', ['F4xQKRUagnSGjFqafyhajLs94e7Vvzvr8ebwYJceKpr8R7T'])
• Better signer abstraction
• Better support for custom middleware
• More advanced subscription API - ability to subscribe to multiple storage items or events concurrently
• Better support for runtime versioning - haven't seen anything related to this on py-substrate-interface

[PieWol]

Great, thanks for elaborating. I think the idea to wrap subxt into a python library is a nice way to cater the subxt functionality to a bigger community of developers. Have you thought about how the python library would be maintained to track the progress of subxt in the future?

I think the following from your "future plans" is too much of an effort for a negligible gain.

Optimize the library for performance and efficiency to be on par with the native subxt library

I will wait a little longer to give others the chance to comment on this proposal before submitting it to the committee.

[philoniare]

@PieWol great to hear from you!

I think the following from your "future plans" is too much of an effort for a negligible gain.

Yep, agreed.

Have you thought about how the python library would be maintained to track the progress of subxt in the future?
This is definitely a crucial consideration to ensure the long-term viability and usefulness of subxt-python.

We will establish a clear versioning scheme for subxt-python that aligns with the versioning of the subxt Rust library. This will help users understand which versions of subxt-python are compatible with specific versions of subxt. We will strive to maintain compatibility with the latest stable version of subxt and provide clear documentation on version compatibility. In case the community finds the library useful, I'd be happy to apply for a maintenance grant and keep the library up to date and also welcome community contributors. There's definitely a lot more Python devs than Rust devs at the moment. Another thing I'm considering is setting up an automatic tests that tracks each release and alerts devs in case the latest build is no longer compatible so that quick action can be taken.

I will wait a little longer to give others the chance to comment on this proposal before submitting it to the committee.

Sounds good to me, looking forward to hearing more feedback

[semuelle]

Thanks for the application, @philoniare. Some minor questions below.

[philoniare]

Thanks for the application, @philoniare. Some minor questions below.

Thank you for taking the time to review the application. The questions have been addressed.

[semuelle]

Sorry for the long wait, @philoniare. I'm willing to give this project a try.

[philoniare]

Not at all, thanks for the faith @semuelle! I do believe that python interfaces will be increasingly popular as smaller teams join the ecosystem due to the introduction of Agile Coretime. It'd be great if we can greet them with a well-developed well-documented API along with tutorials for the most common off-chain use-cases, such as building custom indexers.

[PieWol]

Thanks for the latest changes @philoniare ,
happy to go forward with this.

[github-actions]

Congratulations and welcome to the Web3 Foundation Grants Program! Please refer to our Milestone Delivery repository for instructions on how to submit milestones and invoices, our FAQ for frequently asked questions and the support section of our README for more ways to find answers to your questions.

Before you start, take a moment to read through our announcement guidelines for all communications related to the grant or make them known to the right person in your organisation. In particular, please don't announce the grant publicly before at least the first milestone of your project has been approved. At that point or shortly before, you can get in touch with us at [email protected] and we'll be happy to collaborate on an announcement about the work you’re doing.

Lastly, please remember to let us know in case you run into any delays or deviate from the deliverables in your application. You can either leave a comment here or directly request to amend your application via PR. We wish you luck with your project! 🚀