transaction transform should NOT canonicalize cbor data outside of the transaction body

[AndrewWestberg]

cardano-cli creates datum values that are outside of the transaction body using indeterminate arrays. A hash value is then placed inside of the transaction body. In order to be able to sign with cardano-hw-cli, only things inside the tx body should be canonicalized. If any datum values outside are changed, hashes don't match and the transaction doesn't work. Leaving data outside of the tx-body as non-canonical should not change the ability of the ledger device to witness the transaction.

Example:

$ cat tmp/tx.draft
{
    "type": "Unwitnessed Tx BabbageEra",
    "description": "Ledger Cddl Format",
    "cborHex": "84aa008382582066cec965a9554d39854dbcacd3d0d41099dc014157b579770ff0294f0b5139810082582066cec965a9554d39854dbcacd3d0d41099dc014157b579770ff0294f0b51398101825820f71ab92331f4b6d5ae65b8264815dc2cb5f002985f44043aea31bb2ff571404b020d81825820949d496d900345508ed6e0b27b2a79a284fd3c37b7a3b7b01cece027fe1577aa0212828258200b4c56fdae7c23748c837e2443dfebebd020239c107aed9b227851ebabe4a829018258200b4c56fdae7c23748c837e2443dfebebd020239c107aed9b227851ebabe4a829020182a300581d704347ac2bcda3f6516082cba79661714b836e68eba49d99429d73eec801821a00152d2ca1581cfd3a69817fe5b9ff39fb2fac2be2c7f2007746e827ee31868fe667cda1454e45574d5f01028201d8185829d8799f581c3ab25c853f0f188f43b34b2df5cb98737a4de7e19028ec9d1a414a5505454e45574d5fffa200581d609bb822cdf2c6c79657c0e8ec24308487dbd35750c084e271620d9d80011a14d3429f10a200581d609bb822cdf2c6c79657c0e8ec24308487dbd35750c084e271620d9d80011a0037f9dd111a00145163021a000d8b970e84581c13e17b6072008140a59a22402bd5ef44baf75cc7f6913b4cf219981e581c4c1017ed05f703f6cf31c6a743f2c69534dbe4846e0d51ed4cb24b85581c6fff6d33958f65022de64aabd3c16a7a7e07bad7a94a4585ce1dac76581c9bb822cdf2c6c79657c0e8ec24308487dbd35750c084e271620d9d8009a1581c3ab25c853f0f188f43b34b2df5cb98737a4de7e19028ec9d1a414a55a1464e45574d5f34200b5820864a2c18c923fe7066e40ac90d750097edf43d31c105516c0fe33f6817f0c576a10582840002d87a80821a00335d601a361892eb840100d8799f581c3ab25c853f0f188f43b34b2df5cb98737a4de7e19028ec9d1a414a5505454e45574d5fff821a00510ed71a5525e458f5f6"
}

$ cbor-diag --to diag <<< 84aa008382582066cec965a...
[
    {
        0: [
            [
                h'66cec965a9554d39854dbcacd3d0d41099dc014157b579770ff0294f0b513981',
                0,
            ],
            [
                h'66cec965a9554d39854dbcacd3d0d41099dc014157b579770ff0294f0b513981',
                1,
            ],
            [
                h'f71ab92331f4b6d5ae65b8264815dc2cb5f002985f44043aea31bb2ff571404b',
                2,
            ],
        ],
        13: [
            [
                h'949d496d900345508ed6e0b27b2a79a284fd3c37b7a3b7b01cece027fe1577aa',
                2,
            ],
        ],
        18: [
            [
                h'0b4c56fdae7c23748c837e2443dfebebd020239c107aed9b227851ebabe4a829',
                1,
            ],
            [
                h'0b4c56fdae7c23748c837e2443dfebebd020239c107aed9b227851ebabe4a829',
                2,
            ],
        ],
        1: [
            {
                0: h'704347ac2bcda3f6516082cba79661714b836e68eba49d99429d73eec8',
                1: [
                    1387820_2,
                    {
                        h'fd3a69817fe5b9ff39fb2fac2be2c7f2007746e827ee31868fe667cd': {h'4e45574d5f': 1},
                    },
                ],
                2: [
                    1,
                    24_0(h'd8799f581c3ab25c853f0f188f43b34b2df5cb98737a4de7e19028ec9d1a414a5505454e45574d5fff'),
                ],
            },
            {
                0: h'609bb822cdf2c6c79657c0e8ec24308487dbd35750c084e271620d9d80',
                1: 349389471_2,
            },
        ],
        16: {
            0: h'609bb822cdf2c6c79657c0e8ec24308487dbd35750c084e271620d9d80',
            1: 3668445_2,
        },
        17: 1331555_2,
        2: 887703_2,
        14: [
            h'13e17b6072008140a59a22402bd5ef44baf75cc7f6913b4cf219981e',
            h'4c1017ed05f703f6cf31c6a743f2c69534dbe4846e0d51ed4cb24b85',
            h'6fff6d33958f65022de64aabd3c16a7a7e07bad7a94a4585ce1dac76',
            h'9bb822cdf2c6c79657c0e8ec24308487dbd35750c084e271620d9d80',
        ],
        9: {
            h'3ab25c853f0f188f43b34b2df5cb98737a4de7e19028ec9d1a414a55': {h'4e45574d5f34': -1},
        },
        11: h'864a2c18c923fe7066e40ac90d750097edf43d31c105516c0fe33f6817f0c576',
    },
    {
        5: [
            [0, 2, 122_0([]), [3366240_2, 907580139_2]],
            [
                1,
                0,
                121_0([_
                    h'3ab25c853f0f188f43b34b2df5cb98737a4de7e19028ec9d1a414a55',
                    5,
                    h'4e45574d5f',
                ]),
                [5312215_2, 1428546648_2],
            ],
        ],
    },
    true,
    null,
]

Notice that there is an indeterminate array outside of the transaction body in the 5: section inside 121_0 tag.

After running through transform. This data has been corrupted by making it into a determinate length array. This value should not affect signing on the ledger because it is OUTSIDE the transaction body.

After:

$ cat tmp/tx.draft1
{
    "type": "Unwitnessed Tx BabbageEra",
    "description": "Ledger Cddl Format",
    "cborHex": "84aa008382582066cec965a9554d39854dbcacd3d0d41099dc014157b579770ff0294f0b5139810082582066cec965a9554d39854dbcacd3d0d41099dc014157b579770ff0294f0b51398101825820f71ab92331f4b6d5ae65b8264815dc2cb5f002985f44043aea31bb2ff571404b020182a300581d704347ac2bcda3f6516082cba79661714b836e68eba49d99429d73eec801821a00152d2ca1581cfd3a69817fe5b9ff39fb2fac2be2c7f2007746e827ee31868fe667cda1454e45574d5f01028201d8185829d8799f581c3ab25c853f0f188f43b34b2df5cb98737a4de7e19028ec9d1a414a5505454e45574d5fffa200581d609bb822cdf2c6c79657c0e8ec24308487dbd35750c084e271620d9d80011a14d3429f021a000d8b9709a1581c3ab25c853f0f188f43b34b2df5cb98737a4de7e19028ec9d1a414a55a1464e45574d5f34200b5820864a2c18c923fe7066e40ac90d750097edf43d31c105516c0fe33f6817f0c5760d81825820949d496d900345508ed6e0b27b2a79a284fd3c37b7a3b7b01cece027fe1577aa020e84581c13e17b6072008140a59a22402bd5ef44baf75cc7f6913b4cf219981e581c4c1017ed05f703f6cf31c6a743f2c69534dbe4846e0d51ed4cb24b85581c6fff6d33958f65022de64aabd3c16a7a7e07bad7a94a4585ce1dac76581c9bb822cdf2c6c79657c0e8ec24308487dbd35750c084e271620d9d8010a200581d609bb822cdf2c6c79657c0e8ec24308487dbd35750c084e271620d9d80011a0037f9dd111a0014516312828258200b4c56fdae7c23748c837e2443dfebebd020239c107aed9b227851ebabe4a829018258200b4c56fdae7c23748c837e2443dfebebd020239c107aed9b227851ebabe4a82902a10582840002d87a80821a00335d601a361892eb840100d87983581c3ab25c853f0f188f43b34b2df5cb98737a4de7e19028ec9d1a414a5505454e45574d5f821a00510ed71a5525e458f5f6"
}%                                                                                                                                                                                                                                                        

$ cbor-diag --to diag <<< 84aa008382582066cec965a9...
[
    {
        0: [
            [
                h'66cec965a9554d39854dbcacd3d0d41099dc014157b579770ff0294f0b513981',
                0,
            ],
            [
                h'66cec965a9554d39854dbcacd3d0d41099dc014157b579770ff0294f0b513981',
                1,
            ],
            [
                h'f71ab92331f4b6d5ae65b8264815dc2cb5f002985f44043aea31bb2ff571404b',
                2,
            ],
        ],
        1: [
            {
                0: h'704347ac2bcda3f6516082cba79661714b836e68eba49d99429d73eec8',
                1: [
                    1387820_2,
                    {
                        h'fd3a69817fe5b9ff39fb2fac2be2c7f2007746e827ee31868fe667cd': {h'4e45574d5f': 1},
                    },
                ],
                2: [
                    1,
                    24_0(h'd8799f581c3ab25c853f0f188f43b34b2df5cb98737a4de7e19028ec9d1a414a5505454e45574d5fff'),
                ],
            },
            {
                0: h'609bb822cdf2c6c79657c0e8ec24308487dbd35750c084e271620d9d80',
                1: 349389471_2,
            },
        ],
        2: 887703_2,
        9: {
            h'3ab25c853f0f188f43b34b2df5cb98737a4de7e19028ec9d1a414a55': {h'4e45574d5f34': -1},
        },
        11: h'864a2c18c923fe7066e40ac90d750097edf43d31c105516c0fe33f6817f0c576',
        13: [
            [
                h'949d496d900345508ed6e0b27b2a79a284fd3c37b7a3b7b01cece027fe1577aa',
                2,
            ],
        ],
        14: [
            h'13e17b6072008140a59a22402bd5ef44baf75cc7f6913b4cf219981e',
            h'4c1017ed05f703f6cf31c6a743f2c69534dbe4846e0d51ed4cb24b85',
            h'6fff6d33958f65022de64aabd3c16a7a7e07bad7a94a4585ce1dac76',
            h'9bb822cdf2c6c79657c0e8ec24308487dbd35750c084e271620d9d80',
        ],
        16: {
            0: h'609bb822cdf2c6c79657c0e8ec24308487dbd35750c084e271620d9d80',
            1: 3668445_2,
        },
        17: 1331555_2,
        18: [
            [
                h'0b4c56fdae7c23748c837e2443dfebebd020239c107aed9b227851ebabe4a829',
                1,
            ],
            [
                h'0b4c56fdae7c23748c837e2443dfebebd020239c107aed9b227851ebabe4a829',
                2,
            ],
        ],
    },
    {
        5: [
            [0, 2, 122_0([]), [3366240_2, 907580139_2]],
            [
                1,
                0,
                121_0([
                    h'3ab25c853f0f188f43b34b2df5cb98737a4de7e19028ec9d1a414a55',
                    5,
                    h'4e45574d5f',
                ]),
                [5312215_2, 1428546648_2],
            ],
        ],
    },
    true,
    null,
]

The datum values are now in a determinate array and even though signing is allowed, the transaction cannot succeed because the hash value for the datum has changed.

[gitmachtl]

$ cardano-hw-cli version  
Cardano HW CLI Tool version 1.12.0
Commit hash: d52a82c31831997d086a33e53d081b142f6afd89

[AndrewWestberg]

The workaround I've found is to go through the whole signing process with the ledgers and before submitting, fix the datum outside of the transaction body to put back in the indeterminate-length array.

[janmazak]

cardano-hw-cli encodes and decodes cbor by using a third-party library, trying to avoid having to write a custom one. Hence it is hard to encode one part of an object with definite length arrays and another with indefinite length arrays, and it's also hard to tell the cbor library to stop decoding at some point. And it's messy and error-prone to do some string cut/paste/replace operations on encoded cbors. I'll discuss this with IOG.

[AndrewWestberg]

Fixed on the IOG side