Tracking issue for translating XCM tests from Zombienet to XCM simulator.

[ruseinov]

The goal is to be able to run as many as possible, if not all, XCM tests in the CI, without waiting for services to come up and other pain-points of actually spinning up multiple chains. This will also simplify testing locally, as unit-tests are easier and faster to run. The only concern here is that importing several runtimes might make the compilation pretty slow and resource-hungry, something to take into account.

This way we'll always know if any of the assumptions have changed and communicate that to the teams accordingly. Currently XCM tests reside in integration-tests directory and leverage Zombienet to run, but that has to be run manually.

How to approach this

The idea is to end up with something similar to the example here, introducing dev-dependencies that would import all the relevant runtimes, parachain and relay, as git dependencies, referencing a relative path for each of them and a polkadot master branch. And implement unit-tests per chain for all the scenarios we currently have in integration-tests repo and beyond.

Additionally it would be nice to have those tests comprehensively documented in order to provide meaningful usage examples for the community to take inspiration from.

A list of parachain XCM tests that need to be translated as seen in integration-tests directory:

• XCM-Simulator tests: Collectives #2205
• XCM-Simulator tests: Statemine #2206
• XCM-Simulator tests: Statemint #2207

[bkchr]

Zombienet is the way to, but the mentioned integrration tests are using some layer put on top of Zombienet. The solution here should be that we write these XCM in "pure" Zombienet. Currently not that trivial, but you can already "branch out" to some js script where you can use polkadot-js. We should use this way for writing the tests and then we should use the correct Statemine/Statemint/etc runtimes to run these XCM tests in CI.

[ruseinov]

Zombienet is the way to, but the mentioned integrration tests are using some layer put on top of Zombienet. The solution here should be that we write these XCM in "pure" Zombienet. Currently not that trivial, but you can already "branch out" to some js script where you can use polkadot-js. We should use this way for writing the tests and then we should use the correct Statemine/Statemint/etc runtimes to run these XCM tests in CI.

The idea behind this issue is to replicate as many of those tests as possible via XCM-simulator in order to have something that documents XCM usage and configuration well for ecosystem teams as discussed with @joepetrowski.
Perhaps implementing the Zombienet tests properly as you mentioned is something that needs to be mentioned in it's own issue and indeed used for the CI, while XCM-simulator tests would be more of a usage example doc for the community.

@joepetrowski would be nice if you could chip in with our vision of the subject.

[bkchr]

Okay! Yeah, I also just realized that xcm-simulator is using the runtimes directly. I had thought that this was mainly using some mocked stuff, but that way your issue makes more sense! Ty!

[xlc]

FYI this is how we plan to write e2e xcm tests with Chopsticks https://github.com/AcalaNetwork/e2e-tests/blob/master/tests/karura-statemine/transfer-asset.test.ts

It not only use real runtime but also mainnet state