Investigate zombienet-bridges-* test failures

[serban300]

During the last few days, the zombienet-bridges-* tests started failing all the time.

For example: https://gitlab.parity.io/parity/mirrors/polkadot-sdk/-/jobs/6946518

We need to investigate what is causing this

[serban300]

From what I understand, this is where it started to fail: #5074

[serban300]

This was caused by adding the CheckMetadataHash signed extension to the Rococo and Westend bridge hubs.

https://github.com/paritytech/polkadot-sdk/pull/5074/files/d52473dabd32e6c5cd109fdc71943a04932728ab#r1711034440

[pepoviola]

Hi @serban300, did you have the error log? Is produced by pjs on zombienet?
Thx!

[serban300]

Hi @pepoviola . I have the logs. It's an error in the substrate-relay:

2024-08-09 08:00:57 +00 INFO bridge Initialize bridge call encoded as hex string: "0x3001236e8c192e93841d1a07c92855d27111ba8b89f0832b5ecf750ce5b4ee8cc0a3506d5bef830a0d3f72743f54d7e81d5a876d7fa62ee5783de0c4b4ec53ce4265bb86472652635dfffef5231195a82c01c5af5fec1ea927a4f6963e918778c988c00c0642414245b5010101000000c74c1e1100000000aa4b04ad4a4057cfa86ceb8f64101dd9b9fd1dabf0ed5527be2ac84980e497081cf8cb8c0934688f17dc6cee771d995fedb762a5bb44f9ff00e6c29eaf7e2b0244488b78710fc5af554eff781354c7c2d6e7b38021219427df8fa615ef8c440d04424545468403191353773272c901c20c34481a62776bfc4d651e112fef4cc1702e806df3f30c054241424501014e387fa1233a457b8cae9331b45fd76e46343ba3067f2633880dbc94db1ad342d3b1feffb71f3dd9cad2cfe6ebd09e1794d054b6253abac993690441f2b3a5860c968550f9c1c98355cec02072f5c051ffea6b8e0a67bea2f01fdd17cbc134c861010000000000000087bb0b038291ecd4ba2c47f84209d703adfb4e8381573a494f6de2004575c2db01000000000000004f9d1507a5fcdeb2036cff42eeff77d4c24e58ca42a090c6d4df28c2f19980440100000000000000000000000000000000"
2024-08-09 08:00:57 +00 ERROR bridge Failed to submit Westend-headers bridge initialization transaction to BridgeHubRococo: SubmitTransaction("BridgeHubRococo", FailedToSubmitTransaction { chain: "BridgeHubRococo", error: FailedStateCall { chain: "BridgeHubRococo", hash: "0x0f09…78b7", method: "TaggedTransactionQueue_validate_transaction", arguments: Bytes([2, 185, 8, 132, 0, 142, 175, 4, 21, 22, 135, 115, 99, 38, 201, 254, 161, 126, 37, 252, 82, 135, 97, 54, 147, 201, 18, 144, 156, 178, 38, 170, 71, 148, 242, 106, 72, 1, 112, 215, 242, 2, 69, 3, 217, 239, 78, 146, 148, 246, 106, 42, 93, 162, 209, 69, 7, 254, 4, 48, 83, 62, 66, 217, 234, 109, 87, 203, 124, 106, 77, 172, 237, 205, 178, 57, 9, 183, 173, 123, 41, 83, 29, 96, 75, 53, 9, 81, 78, 65, 166, 86, 86, 19, 224, 241, 215, 199, 237, 5, 21, 141, 0, 0, 0, 48, 1, 35, 110, 140, 25, 46, 147, 132, 29, 26, 7, 201, 40, 85, 210, 113, 17, 186, 139, 137, 240, 131, 43, 94, 207, 117, 12, 229, 180, 238, 140, 192, 163, 80, 109, 91, 239, 131, 10, 13, 63, 114, 116, 63, 84, 215, 232, 29, 90, 135, 109, 127, 166, 46, 229, 120, 61, 224, 196, 180, 236, 83, 206, 66, 101, 187, 134, 71, 38, 82, 99, 93, 255, 254, 245, 35, 17, 149, 168, 44, 1, 197, 175, 95, 236, 30, 169, 39, 164, 246, 150, 62, 145, 135, 120, 201, 136, 192, 12, 6, 66, 65, 66, 69, 181, 1, 1, 1, 0, 0, 0, 199, 76, 30, 17, 0, 0, 0, 0, 170, 75, 4, 173, 74, 64, 87, 207, 168, 108, 235, 143, 100, 16, 29, 217, 185, 253, 29, 171, 240, 237, 85, 39, 190, 42, 200, 73, 128, 228, 151, 8, 28, 248, 203, 140, 9, 52, 104, 143, 23, 220, 108, 238, 119, 29, 153, 95, 237, 183, 98, 165, 187, 68, 249, 255, 0, 230, 194, 158, 175, 126, 43, 2, 68, 72, 139, 120, 113, 15, 197, 175, 85, 78, 255, 120, 19, 84, 199, 194, 214, 231, 179, 128, 33, 33, 148, 39, 223, 143, 166, 21, 239, 140, 68, 13, 4, 66, 69, 69, 70, 132, 3, 25, 19, 83, 119, 50, 114, 201, 1, 194, 12, 52, 72, 26, 98, 119, 107, 252, 77, 101, 30, 17, 47, 239, 76, 193, 112, 46, 128, 109, 243, 243, 12, 5, 66, 65, 66, 69, 1, 1, 78, 56, 127, 161, 35, 58, 69, 123, 140, 174, 147, 49, 180, 95, 215, 110, 70, 52, 59, 163, 6, 127, 38, 51, 136, 13, 188, 148, 219, 26, 211, 66, 211, 177, 254, 255, 183, 31, 61, 217, 202, 210, 207, 230, 235, 208, 158, 23, 148, 208, 84, 182, 37, 58, 186, 201, 147, 105, 4, 65, 242, 179, 165, 134, 12, 150, 133, 80, 249, 193, 201, 131, 85, 206, 192, 32, 114, 245, 192, 81, 255, 234, 107, 142, 10, 103, 190, 162, 240, 31, 221, 23, 203, 193, 52, 200, 97, 1, 0, 0, 0, 0, 0, 0, 0, 135, 187, 11, 3, 130, 145, 236, 212, 186, 44, 71, 248, 66, 9, 215, 3, 173, 251, 78, 131, 129, 87, 58, 73, 79, 109, 226, 0, 69, 117, 194, 219, 1, 0, 0, 0, 0, 0, 0, 0, 79, 157, 21, 7, 165, 252, 222, 178, 3, 108, 255, 66, 238, 255, 119, 212, 194, 78, 88, 202, 66, 160, 144, 198, 212, 223, 40, 194, 241, 153, 128, 68, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 15, 9, 23, 56, 219, 180, 224, 95, 2, 181, 3, 227, 131, 89, 79, 235, 60, 28, 139, 150, 35, 208, 101, 236, 240, 59, 69, 55, 255, 119, 120, 183]), error: RpcError(Call(ErrorObject { code: ServerError(4003), message: "Client error: Execution failed: Execution aborted due to trap: wasm trap: wasm `unreachable` instruction executed\nWASM backtrace:\nerror while executing at wasm backtrace:\n    0: 0x4c01d4 - <unknown>!rust_begin_unwind\n    1: 0x3c9db2 - <unknown>!core::panicking::panic_fmt::hd70a1becd9b46a99\n    2: 0xc452f - <unknown>!TaggedTransactionQueue_validate_transaction", data: None })) } })

Because it doesn't support the CheckMetadataExtension for Rococo/Westend bridge hubs

[pepoviola]

Hi @pepoviola . I have the logs. It's an error in the substrate-relay:

2024-08-09 08:00:57 +00 INFO bridge Initialize bridge call encoded as hex string: "0x3001236e8c192e93841d1a07c92855d27111ba8b89f0832b5ecf750ce5b4ee8cc0a3506d5bef830a0d3f72743f54d7e81d5a876d7fa62ee5783de0c4b4ec53ce4265bb86472652635dfffef5231195a82c01c5af5fec1ea927a4f6963e918778c988c00c0642414245b5010101000000c74c1e1100000000aa4b04ad4a4057cfa86ceb8f64101dd9b9fd1dabf0ed5527be2ac84980e497081cf8cb8c0934688f17dc6cee771d995fedb762a5bb44f9ff00e6c29eaf7e2b0244488b78710fc5af554eff781354c7c2d6e7b38021219427df8fa615ef8c440d04424545468403191353773272c901c20c34481a62776bfc4d651e112fef4cc1702e806df3f30c054241424501014e387fa1233a457b8cae9331b45fd76e46343ba3067f2633880dbc94db1ad342d3b1feffb71f3dd9cad2cfe6ebd09e1794d054b6253abac993690441f2b3a5860c968550f9c1c98355cec02072f5c051ffea6b8e0a67bea2f01fdd17cbc134c861010000000000000087bb0b038291ecd4ba2c47f84209d703adfb4e8381573a494f6de2004575c2db01000000000000004f9d1507a5fcdeb2036cff42eeff77d4c24e58ca42a090c6d4df28c2f19980440100000000000000000000000000000000"
2024-08-09 08:00:57 +00 ERROR bridge Failed to submit Westend-headers bridge initialization transaction to BridgeHubRococo: SubmitTransaction("BridgeHubRococo", FailedToSubmitTransaction { chain: "BridgeHubRococo", error: FailedStateCall { chain: "BridgeHubRococo", hash: "0x0f09…78b7", method: "TaggedTransactionQueue_validate_transaction", arguments: Bytes([2, 185, 8, 132, 0, 142, 175, 4, 21, 22, 135, 115, 99, 38, 201, 254, 161, 126, 37, 252, 82, 135, 97, 54, 147, 201, 18, 144, 156, 178, 38, 170, 71, 148, 242, 106, 72, 1, 112, 215, 242, 2, 69, 3, 217, 239, 78, 146, 148, 246, 106, 42, 93, 162, 209, 69, 7, 254, 4, 48, 83, 62, 66, 217, 234, 109, 87, 203, 124, 106, 77, 172, 237, 205, 178, 57, 9, 183, 173, 123, 41, 83, 29, 96, 75, 53, 9, 81, 78, 65, 166, 86, 86, 19, 224, 241, 215, 199, 237, 5, 21, 141, 0, 0, 0, 48, 1, 35, 110, 140, 25, 46, 147, 132, 29, 26, 7, 201, 40, 85, 210, 113, 17, 186, 139, 137, 240, 131, 43, 94, 207, 117, 12, 229, 180, 238, 140, 192, 163, 80, 109, 91, 239, 131, 10, 13, 63, 114, 116, 63, 84, 215, 232, 29, 90, 135, 109, 127, 166, 46, 229, 120, 61, 224, 196, 180, 236, 83, 206, 66, 101, 187, 134, 71, 38, 82, 99, 93, 255, 254, 245, 35, 17, 149, 168, 44, 1, 197, 175, 95, 236, 30, 169, 39, 164, 246, 150, 62, 145, 135, 120, 201, 136, 192, 12, 6, 66, 65, 66, 69, 181, 1, 1, 1, 0, 0, 0, 199, 76, 30, 17, 0, 0, 0, 0, 170, 75, 4, 173, 74, 64, 87, 207, 168, 108, 235, 143, 100, 16, 29, 217, 185, 253, 29, 171, 240, 237, 85, 39, 190, 42, 200, 73, 128, 228, 151, 8, 28, 248, 203, 140, 9, 52, 104, 143, 23, 220, 108, 238, 119, 29, 153, 95, 237, 183, 98, 165, 187, 68, 249, 255, 0, 230, 194, 158, 175, 126, 43, 2, 68, 72, 139, 120, 113, 15, 197, 175, 85, 78, 255, 120, 19, 84, 199, 194, 214, 231, 179, 128, 33, 33, 148, 39, 223, 143, 166, 21, 239, 140, 68, 13, 4, 66, 69, 69, 70, 132, 3, 25, 19, 83, 119, 50, 114, 201, 1, 194, 12, 52, 72, 26, 98, 119, 107, 252, 77, 101, 30, 17, 47, 239, 76, 193, 112, 46, 128, 109, 243, 243, 12, 5, 66, 65, 66, 69, 1, 1, 78, 56, 127, 161, 35, 58, 69, 123, 140, 174, 147, 49, 180, 95, 215, 110, 70, 52, 59, 163, 6, 127, 38, 51, 136, 13, 188, 148, 219, 26, 211, 66, 211, 177, 254, 255, 183, 31, 61, 217, 202, 210, 207, 230, 235, 208, 158, 23, 148, 208, 84, 182, 37, 58, 186, 201, 147, 105, 4, 65, 242, 179, 165, 134, 12, 150, 133, 80, 249, 193, 201, 131, 85, 206, 192, 32, 114, 245, 192, 81, 255, 234, 107, 142, 10, 103, 190, 162, 240, 31, 221, 23, 203, 193, 52, 200, 97, 1, 0, 0, 0, 0, 0, 0, 0, 135, 187, 11, 3, 130, 145, 236, 212, 186, 44, 71, 248, 66, 9, 215, 3, 173, 251, 78, 131, 129, 87, 58, 73, 79, 109, 226, 0, 69, 117, 194, 219, 1, 0, 0, 0, 0, 0, 0, 0, 79, 157, 21, 7, 165, 252, 222, 178, 3, 108, 255, 66, 238, 255, 119, 212, 194, 78, 88, 202, 66, 160, 144, 198, 212, 223, 40, 194, 241, 153, 128, 68, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 15, 9, 23, 56, 219, 180, 224, 95, 2, 181, 3, 227, 131, 89, 79, 235, 60, 28, 139, 150, 35, 208, 101, 236, 240, 59, 69, 55, 255, 119, 120, 183]), error: RpcError(Call(ErrorObject { code: ServerError(4003), message: "Client error: Execution failed: Execution aborted due to trap: wasm trap: wasm `unreachable` instruction executed\nWASM backtrace:\nerror while executing at wasm backtrace:\n    0: 0x4c01d4 - <unknown>!rust_begin_unwind\n    1: 0x3c9db2 - <unknown>!core::panicking::panic_fmt::hd70a1becd9b46a99\n    2: 0xc452f - <unknown>!TaggedTransactionQueue_validate_transaction", data: None })) } })

Because it doesn't support the CheckMetadataExtension for Rococo/Westend bridge hubs

Ahh oks, I though that the error can be caused by an outdated version of pjs in zombienet 👍.
Thx!