Deeply nested value

[remyhaemmerle-da]

The different ledger implementations do not handle consistently transactions with deeply nested values and crash more or less nicely when dealing with those.

We currently do not have clear specification for dealing with those transactions. in particular, LF transaction specification does not clearly specify the maximal depth of values. however, The engine ensures the values coming from the ledger API are not deeper than 100.

I propose to use the 100 limit as a hard limit:

1. A ledger should handle successfully the creation of a transaction that contains values with a nesting lower or equal to 100.
2. A ledger should not fails nicely when ask to create a transaction that contains values with a nesting strictly greater than 100.

This limits since to be bigger than all the current (practical) limits of our ledgers (sandbox classic, KV, Canton), so we can hope reaching a consistent behavior to all our ledgers in a backward compatible way.

We disregard the case of concurrent system where some older node (that do not handle this limit) communicate to newer node (that do handle the limit) as Canton is not production ready.

---

The solution we implemented produce a small slow down during (de)serialization + (de/en)coding.
For the record, here are the results of benchmark //ledger/participant-state/kvutils/tools:benchmark-codec

Reference (without changes from #10443 and #10393):

Benchmark                                     (ledgerExport)   Mode  Cnt     Score    Error  Units
DecodeTransactionBenchmark.run                                thrpt  200   120.164 ±  1.260  ops/s
DecodeValueBenchmark.run                                      thrpt  200   891.367 ±  5.419  ops/s
DeserializeAndDecodeTransactionBenchmark.run                  thrpt  200    82.018 ±  0.801  ops/s
DeserializeAndDecodeValueBenchmark.run                        thrpt  200   431.427 ±  3.739  ops/s
DeserializeTransactionBenchmark.run                           thrpt  200   316.590 ±  0.991  ops/s
DeserializeValueBenchmark.run                                 thrpt  200   872.121 ±  3.735  ops/s
EncodeAndSerializeTransactionBenchmark.run                    thrpt  200   121.585 ±  0.890  ops/s
EncodeAndSerializeValueBenchmark.run                          thrpt  200   425.170 ±  2.791  ops/s
EncodeTransactionBenchmark.run                                thrpt  200   316.702 ±  1.548  ops/s
EncodeValueBenchmark.run                                      thrpt  200  1188.807 ±  6.044  ops/s
SerializeTransactionBenchmark.run                             thrpt  200   445.964 ±  2.162  ops/s
SerializeValueBenchmark.run                                   thrpt  200  1296.371 ± 16.735  ops/s
engine.SpeedyToValueBenchmark.run                             thrpt  200  1089.514 ± 15.234  ops/s
engine.ValueTranslatorBenchmark.run                           thrpt  200   804.166 ±  6.815  ops/s
speedy.MachineImportValueBenchmark.run                        thrpt  200   988.040 ±  7.723  ops/s

After (with change from #10443 and #10393):

Benchmark                                     (ledgerExport)   Mode  Cnt     Score    Error  Units
DecodeTransactionBenchmark.run                                thrpt  200    98.487 ±  1.391  ops/s
DecodeValueBenchmark.run                                      thrpt  200   453.091 ±  3.579  ops/s
DeserializeAndDecodeTransactionBenchmark.run                  thrpt  200    77.268 ±  0.611  ops/s
DeserializeAndDecodeValueBenchmark.run                        thrpt  200   394.348 ±  3.151  ops/s
DeserializeTransactionBenchmark.run                           thrpt  200   444.050 ±  1.694  ops/s
DeserializeValueBenchmark.run                                 thrpt  200  3376.012 ± 25.028  ops/s
EncodeAndSerializeTransactionBenchmark.run                    thrpt  200   115.878 ±  0.837  ops/s
EncodeAndSerializeValueBenchmark.run                          thrpt  200   403.037 ±  2.711  ops/s
EncodeTransactionBenchmark.run                                thrpt  200   214.768 ±  1.445  ops/s
EncodeValueBenchmark.run                                      thrpt  200   476.360 ±  2.976  ops/s
SerializeTransactionBenchmark.run                             thrpt  200   611.573 ±  5.822  ops/s
SerializeValueBenchmark.run                                   thrpt  200  4737.347 ± 52.758  ops/s
engine.SpeedyToValueBenchmark.run                             thrpt  200  1069.260 ± 16.976  ops/s
engine.ValueTranslatorBenchmark.run                           thrpt  200   797.031 ±  7.184  ops/s
speedy.MachineImportValueBenchmark.run                        thrpt  200   985.888 ±  7.038  ops/s