Correctness issue - "Key length exceeds limit"

[apkar]

$ python test/correctness/document-correctness.py --doclayer-host localhost --doclayer-port 27018 forever doclayer mm --seed 4994950075151235634 --num-doc 300 --num-iter 1 --no-update --no-sort --no-numeric-fieldnames
Instance: 621285605982
========================================================
ID : 44348 iteration : 1
========================================================
 Key length exceeds limit
 .......
python /Users/bmuppana/src/fdb-document-layer/test/correctness/document-correctness.py --mongo-host localhost --mongo-port 27018 --doclayer-host localhost --doclayer-port 27018 forever doclayer mm --seed 4994950075151235634 --num-doc 300 --num-iter 1 --no-update --no-sort --no-numeric-fieldnames

Stack trace from trace logs

<Event Severity="40" Time="1547842023.948421" Type="BD_doIndexUpdate" ID="0000000000000000" error="Key length exceeds limit" Backtrace="atos -o fdbdoc.debug -arch x86_64 -l 0x10e5c4000 0x10e968c91 0x10e968d6f 0x10e7eedb3 0x10e800482 0x10e7ee699 0x10e7f53ce 0x10e7f4de5 0x10e7f7437 0x10e7f3af7 0x10e5eed2e 0x10e7f4ee4 0x10e7f52a0 0x10e7dd43e 0x10e7dc125 0x10e7dc07d 0x10e7dd582 0x10e7dbc37 0x10e744cdc 0x10e77f1d8 0x10e7fabd3 0x10e7fb8d2 0x10e7f99d7 0x10e77ce6e 0x10e77cc95 0x10e77d157 0x10e77c947 0x10e7c3b2e 0x10e8d61b5 0x10e8d6a57 0x10e8d6177 0x10e77ce6e 0x10e8d28b5 0x10e8d3509 0x10e8d5672 0x10e8d1af7 0x10e7ca18c 0x10e7ca118 0x..." Machine="127.0.0.1:27018" LogGroup="default" />

Found this against d2840e9. Consistently reproducible with the above seed.

[apkar]

Correctness is generating very large index keys. We should a couple of things part of this issue.

• Fail on index keys larger than 1024 bytes explicitly. Don't wait on FDB to fail. Errors coming from tr->set() are much harder to send back to the client as they don't come back in the same stack trace.
• Either make sure correctness wouldn't generate larger keys or implement index checks in MongoModel

[dongxinEric]

Instead of fixing this by limiting the document size generated by the random generator, we decided to do following:

• A bit background first: the Document Layer's key size limit (10K bytes) and value size limit (100K bytes) are controlled by the Knob of the backing FDB KVS and thus the error was reported from KVS, which means Document Layer itself does not do any checking there.
• We want to have Document Layer to do the check before it makes the way to KVS for a variety of reasons, one of which is better error reporting.
• We will then change the MongoModel in correctness tests to include this behavior too so that in the future, both MongoModel and the actual Document Layer instance will report the same error and that will be counted as a pass in the correctness tests.

[dongxinEric]

@apkar reminds me that there is one caveat of this solution:

• MongoModel never construct an actual FDB Key, since it dose not need to, and thus does not know the actual FDB Key length of the BSON document's key. Thus it is very hard to get the check precisely.

We would make MongoModel to do its best to estimate the size of the key and we would see how many errors we got from the correctness tests runs and then decide what to do from there.