[BUG] Request to index a large deeply nested doc that can potentially cause stack overflow and node to crash

[nkumar04]

Describe the bug
User with proper permission to OpenSearch endpoint can create a large deeply nested request that can potentially cause stack overflow and node to crash. For example, see the request below. Current implementation recursively goes inside each nested filed object and tries to parse it.

{
  "inner1": {
    "inner2": {
      "inner3": {
        ...
        #nested upto N (where N is very large number ~10000)
        ...
      }
    }
  }
}

To Reproduce
Steps to reproduce the behavior:
Use the python script (create_doc.py) below to generate a deeply nested doc.

import json
import sys
DEPTH=2000
n = len(sys.argv)
if n == 2:
    DEPTH = int(sys.argv[1])
obj = '{ "field": "value" }'
for i in range(DEPTH):
    field = "\"inner"+str(i+1)+"\""
    obj = "{ " + field + ": " + obj + "}"
json_doc = open("data.json", "w")
json_doc.write(obj)
json_doc.close()

Create a nested doc:
python3 create_doc.py 10000
Add the document for indexing:
curl -XPUT "localhost:9200/test-index-1/_doc/1?pretty" -H 'Content-Type: application/json' -d @data.json
Output from local run:

?  fatal error in thread [opensearch[runTask-0][write][T#2]], exiting
?  java.lang.StackOverflowError
?       at org.opensearch.common.settings.Setting.isDeprecated(Setting.java:388)
?       at org.opensearch.common.settings.Setting.checkDeprecation(Setting.java:563)
?       at org.opensearch.common.settings.Setting.getRaw(Setting.java:536)
?       at org.opensearch.common.settings.Setting.get(Setting.java:469)
?       at org.opensearch.common.settings.Setting.get(Setting.java:465)
?       at org.opensearch.Version.indexCreated(Version.java:140)
?       at org.opensearch.index.mapper.ObjectMapper.<init>(ObjectMapper.java:454)
?       at org.opensearch.index.mapper.ObjectMapper$Builder.createMapper(ObjectMapper.java:246)
?       at org.opensearch.index.mapper.ObjectMapper$Builder.build(ObjectMapper.java:224)
?       at org.opensearch.index.mapper.ObjectMapper$Builder.build(ObjectMapper.java:173)
?       at org.opensearch.index.mapper.DocumentParser.parseObject(DocumentParser.java:551)
?       at org.opensearch.index.mapper.DocumentParser.innerParseObject(DocumentParser.java:440)

Expected behavior
Exception while parsing the request doc should be handled gracefully without crashing the OpenSearch application.

Additional context
There are few settings in place to limit the number of field mappings (created manually or dynamically) and prevent documents from causing a mapping explosion. For example, INDEX_MAPPING_DEPTH_LIMIT_SETTING. Document parser parses the document and feed it to the mapper service to create or update the mapping tied to the index. Settings to limit the number of field mappings are enforced while creating or updating the mapping for index dynamically. But the prerequisite for these validation is successful completion of document parsing stage. Hence, we want to make sure that any exception at the time of parsing of a malformed document is handled gracefully

[dblock]

Thanks for reporting this! I see you have started on a fix, looking forward to it.