Fix bug where document embedding fails to be generated due to document has dot in field name

[yizheliu-amazon]

Description

Fix bug where document embedding fails to be generated due to document has dot in field name, which does not match field mapping exactly

Related Issues

Resolves #1042

Root cause

Such issue is caused by that we fail to unbox/unflatten field in ingested doc, then it causes mismatch between nested field in fieldMap config and ingested document schema.

For example, nested field can be represented as either {"a.b": "c"}, or {"a": {"b": "c"}}. If fieldMap is {"a": {"b": "b_embedding"}}, but ingested document is like {"a.b": "c"}, such schema mismatch fieldMap, and then we are not able to fetch "c" as value for inferencing.

About the change

Basically, the solution is to unbox/unflatten the field with dot from ingested doc.

Before this change, given fieldMap is below

{
  "description": "text embedding pipeline for exampe",
  "processors": [
    {
      "text_embedding": {
        "model_id": "EJTmBpQBz0MpoghfHptn",
        "field_map": {
          "nested_passages.level_2.level_3_text": "level_3_container.level_3_embedding"
        }
      }
    }
  ]
}

Simulate ingestion for below doc, there is no level_3_container.level_3_embedding generated for it.

{
  "docs": [
    {
      "_index": "neural-search-index-v2",
      "_id": "1",
      "_source": {
        "nested_passages": [
          {
            "level_2.level_3_text": "clown"
          },
          {
            "level_2": {
              "level_3_text": "batman"
            }
          }
        ]
      }
    }
  ]
}

After this change, embedding can be generated

{
  "docs": [
    {
      "doc": {
        "_index": "neural-search-index-v2",
        "_id": "1",
        "_source": {
          "nested_passages": [
            {
              "level_2": {
                "level_3_container": {
                  "level_3_embedding": [
                    0.17990997,
                    0.064552955,
                    ...
                  ]
                },
                "level_3_text": "clown"
              }
            },
            {
              "level_2": {
                "level_3_container": {
                  "level_3_embedding": [
                    0.20891039,
                    0.09188991,
                    ...
                  ]
                },
                "level_3_text": "batman"
              }
            }
          ]
        },
        "_ingest": {
          "timestamp": "2025-01-06T19:00:12.84077Z"
        }
      }
    }
  ]
}

Check List

• New functionality includes testing.
• New functionality has been documented.
• API changes companion pull request created.
• Commits are signed per the DCO using --signoff.
• Public documentation issue/PR created.

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

[heemin32]

Could you make the test naming consistent with others?
testUnflatten_whenSimpleDotNotation_thenSucceed

[yizheliu-amazon]

Sure. Will do.