bundle of a product does not return result

[tloubrieu-jpl]

🐛 Describe the bug

bundle of a product does not return result

📜 To Reproduce

Steps to reproduce the behavior:
0. prepare a test server with reference test datasets

1. Go to:
   http://localhost:8081/uid/urn:nasa:pds:insight_rad:data_derived:hp3_rad_der_00014_20181211_073042::1.0/referencing/bundle
2. the response is empty
3. the bundle is found by the registry api, http://localhost:8081/uid/urn:nasa:pds:insight_rad

🕵️ Expected behavior

The request http://localhost:8081/uid/urn:nasa:pds:insight_rad:data_derived:hp3_rad_der_00014_20181211_073042::1.0/referencing/bundle should return the description of the bundle of the requested product

📚 Version of Software Used

1.1.0-SNAPSHOT

🩺 Test Data / Additional context

🏞Screenshots

🖥 System Info

• OS: [e.g. iOS]
• Browser [e.g. chrome, safari]
• Version [e.g. 22]

---

🦄 Related requirements

🦄 #454

⚙️ Engineering Details

[tloubrieu-jpl]

This ticket shows the same behavior as NASA-PDS/pds-api#133 which was close because it was not reproduced.

The bug is not visible in the stable release deployed in production so this ticket is not critical but need to be understood.

[tloubrieu-jpl]

@al-niessner found out it is an effect of the migration from elasticsearch to opensearch.

[al-niessner]

@jordanpadams @tloubrieu-jpl

Invite your opensearch specialists to identify the problem. Let me set the stage first:

There are two opensearch queries -- bypasses registry for simple clarity:

q1 (query 1):

{
  "from": 0,
  "size": 10,
  "query": {
    "bool": {
      "must": [
        {
          "term": {
            "ref_lid_collection": {
              "value": "urn:nasa:pds:insight_rad:data_derived",
              "boost": 1
            }
          }
        }
      ],
      "adjust_pure_negative": true,
      "boost": 1
    }
  },
  "_source": {
    "includes": [
        "lid", "ref_lid_collection"
    ],
    "excludes": [
      "ops:Label_File_Info/ops:blob",
      "ops:Label_File_Info/ops:json_blob"
    ]
  }
}

and q2 (query 2):

{
  "from": 0,
  "size": 10,
  "query": {
    "bool": {
      "must": [
        {
          "term": {
            "product_class": {
              "value": "Product_Bundle",
              "boost": 1
            }
          }
        }
      ],
      "adjust_pure_negative": true,
      "boost": 1
    }
  },
  "_source": {
    "includes": [
        "lid", "ref_lid_collection"
    ],
    "excludes": [
      "ops:Label_File_Info/ops:blob",
      "ops:Label_File_Info/ops:json_blob"
    ]
  }
}

and if I difference the two queries to focus on the characters that are different rather than the entire JSON:

$ diff q1,json q2.json
9,10c9,10
<             "ref_lid_collection": {
<               "value": "urn:nasa:pds:insight_rad:data_derived",
---
>             "product_class": {
>               "value": "Product_Bundle",

Everything is the same but the name of the term and its value. If I then directly use a test opensearch:

$ curl --insecure --location --request GET 'https://localhost:9200/registry/_search' --header 'Accept: application/json' --header 'Content-Type: application/json' --user admin:admin -d @Projects/PDS/q4.json | jq
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100   619  100   160  100   459   5333  15300 --:--:-- --:--:-- --:--:-- 20633
{
  "took": 1,
  "timed_out": false,
  "_shards": {
    "total": 1,
    "successful": 1,
    "skipped": 0,
    "failed": 0
  },
  "hits": {
    "total": {
      "value": 0,
      "relation": "eq"
    },
    "max_score": null,
    "hits": []
  }
}

It seems term can no longer find a value in an array list.. We can see from q2 that the value does exist in the term given in q1. Running q2 it gives us the expected results:

$ curl --insecure --location --request GET 'https://localhost:9200/registry/_search' --header 'Accept: application/json' --header 'Content-Type: application/json' --user admin:admin -d @Projects/PDS/q2.json | jq
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100   895  100   430  100   465  22631  24473 --:--:-- --:--:-- --:--:-- 47105
{
  "took": 3,
  "timed_out": false,
  "_shards": {
    "total": 1,
    "successful": 1,
    "skipped": 0,
    "failed": 0
  },
  "hits": {
    "total": {
      "value": 1,
      "relation": "eq"
    },
    "max_score": 2.8526313,
    "hits": [
      {
        "_index": "registry",
        "_id": "urn:nasa:pds:insight_rad::2.1",
        "_score": 2.8526313,
        "_source": {
          "lid": "urn:nasa:pds:insight_rad",
          "ref_lid_collection": [
            "urn:nasa:pds:insight_rad:data_calibrated",
            "urn:nasa:pds:insight_rad:data_derived",
            "urn:nasa:pds:insight_rad:data_raw"
          ]
        }
      }
    ]
  }
}

Combined the correct q2 with q1 to make this request:

{
  "from": 0,
  "size": 10,
  "query": {
    "bool": {
      "must": [
        {
          "term": {
            "ref_lid_collection": {
              "value": "urn:nasa:pds:insight_rad:data_derived",
              "boost": 1
            }
          }
        },
        {
          "term": {
            "product_class": {
              "value": "Product_Bundle",
              "boost": 1
            }
          }
        }
      ],
      "adjust_pure_negative": true,
      "boost": 1
    }
  },
  "_source": {
    "includes": [
        "lid", "ref_lid_collection"
    ],
    "excludes": [
      "ops:Label_File_Info/ops:blob",
      "ops:Label_File_Info/ops:json_blob"
    ]
  }

Given the bundle can be found but that the value cannot be found in ref_lid_collection implies it should return nothing, which is what happens:

$ curl --insecure --location --request GET 'https://localhost:9200/registry/_search' --header 'Accept: application/json' --header 'Content-Type: application/json' --user admin:admin -d @Projects/PDS/q3.json | jq
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100   809  100   160  100   649   9411  38176 --:--:-- --:--:-- --:--:-- 50562
{
  "took": 2,
  "timed_out": false,
  "_shards": {
    "total": 1,
    "successful": 1,
    "skipped": 0,
    "failed": 0
  },
  "hits": {
    "total": {
      "value": 0,
      "relation": "eq"
    },
    "max_score": null,
    "hits": []
  }
}

So, I need help understanding what opensearch is doing oe think it is doing to make the queries work correctly and thus fix the registry. Maybe just need help seeing the wrong character.

Wanted to make sure that terms was required instead of term so created q4:

{
  "from": 0,
  "size": 10,
  "query": {
    "bool": {
      "must": [
        {
          "terms": {
            "ref_lid_collection": ["urn:nasa:pds:insight_rad:data_derived"],
              "boost": 1
          }
        }
      ],
      "adjust_pure_negative": true,
      "boost": 1
    }
  },
  "_source": {
    "includes": [
        "lid", "ref_lid_collection"
    ],
    "excludes": [
      "ops:Label_File_Info/ops:blob",
      "ops:Label_File_Info/ops:json_blob"
    ]
  }
}

but it too returns nothing:

$ curl --insecure --location --request GET 'https://localhost:9200/registry/_search' --header 'Accept: application/json' --header 'Content-Type: application/json' --user admin:admin -d @Projects/PDS/q4.json | jq
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100   619  100   160  100   459   5333  15300 --:--:-- --:--:-- --:--:-- 20633
{
  "took": 1,
  "timed_out": false,
  "_shards": {
    "total": 1,
    "successful": 1,
    "skipped": 0,
    "failed": 0
  },
  "hits": {
    "total": {
      "value": 0,
      "relation": "eq"
    },
    "max_score": null,
    "hits": []
  }
}

[al-niessner]

hypothesis: ref_lid_collection does not show up in localhost:9200/registry

Try to add it and then see if test passes.

[al-niessner]

Verified hypothesis. This going to be a very long post but it is critical to understand the situation. There are two possible things that happened:

1. opensearch changed dynamic behavior somewhere along the line
2. harvest changed from dynamic true to false
3. combination of the first two

opensearch does not allow the mapping to be changed then reindeced without changing names of the index. If we used alias I could have used the test database created by harvest. Since we do not use aliases, created a test database myself. It is really simple but it will eat up a bit space. It is the creation and testing of it that allowed me to discover the bit about dynamic being crucial.

The first thing I did was create a registry with a mapping using this data:

{
    "mappings":
    {
        "dynamic": "false",
        "dynamic_templates":
        [
            {
                "strings":
                {
                    "match_mapping_type": "string",
                    "mapping": { "type": "keyword" }
                }
            }
        ],
        "properties":
        {
            "name": { "type": "keyword" },
            "type": { "type": "keyword" },
            "number": { "type": "keyword" },
            "peter": { "type": "keyword" }
        }
    },
    "settings":
    {
        "index":
        {
            "number_of_shards": "2",
            "number_of_replicas": "1"
        }
    }
}

Use the command to initiate it:

 $ curl -XPUT --data @Projects/PDS/regmap.json --header 'Content-Type: application/json'  --insecure -u 'admin:admin' 'https://localhost:9200/registry' | jq
{
  "acknowledged": true,
  "shards_acknowledged": true,
  "index": "registry"
}

Then add the data with:

$ curl -XPUT --header 'Content-Type: application/json'  --insecure -u 'admin:admin' 'https://localhost:9200/registry/_bulk' --data '{ "create": { "_id": "1" }}
{ "name":"able", "type":"sir", "number":"0", "ref":["1", "3", "7"] }
{ "create": { "_id": "2" }}
{ "name":"baker", "type":"sir", "number":"0" }
{ "create": { "_id": "3" }}
{ "name":"cain", "type":"christian", "number":"1" }
{ "create": { "_id": "4" }}
{ "name":"didi", "type":"sir", "number":"0" }
{ "create": { "_id": "5" }}
{ "name":"edgar", "type":"christian", "number":"2" }
{ "create": { "_id": "6" }}
{ "name":"francis", "type":"christian", "number":"2" }
{ "create": { "_id": "7" }}
{ "name":"gable", "type":"christian", "number":"0" }
{ "create": { "_id": "8" }}
{ "name":"henry", "type":"sir", "number":"1" }
{ "create": { "_id": "9" }}
{ "name":"irene", "type":"christian", "number":"0" }

' | jq
{
  "took": 32,
  "errors": false,
  "items": [
    {
      "create": {
        "_index": "registry",
        "_id": "1",
        "_version": 1,
        "result": "created",
        "_shards": {
          "total": 2,
          "successful": 1,
          "failed": 0
        },
        "_seq_no": 0,
        "_primary_term": 1,
        "status": 201
      }
    },
    {
      "create": {
        "_index": "registry",
        "_id": "2",
        "_version": 1,
        "result": "created",
        "_shards": {
          "total": 2,
          "successful": 1,
          "failed": 0
        },
        "_seq_no": 1,
        "_primary_term": 1,
        "status": 201
      }
    },
    {
      "create": {
        "_index": "registry",
        "_id": "3",
        "_version": 1,
        "result": "created",
        "_shards": {
          "total": 2,
          "successful": 1,
          "failed": 0
        },
        "_seq_no": 2,
        "_primary_term": 1,
        "status": 201
      }
    },
    {
      "create": {
        "_index": "registry",
        "_id": "4",
        "_version": 1,
        "result": "created",
        "_shards": {
          "total": 2,
          "successful": 1,
          "failed": 0
        },
        "_seq_no": 0,
        "_primary_term": 1,
        "status": 201
      }
    },
    {
      "create": {
        "_index": "registry",
        "_id": "5",
        "_version": 1,
        "result": "created",
        "_shards": {
          "total": 2,
          "successful": 1,
          "failed": 0
        },
        "_seq_no": 3,
        "_primary_term": 1,
        "status": 201
      }
    },
    {
      "create": {
        "_index": "registry",
        "_id": "6",
        "_version": 1,
        "result": "created",
        "_shards": {
          "total": 2,
          "successful": 1,
          "failed": 0
        },
        "_seq_no": 4,
        "_primary_term": 1,
        "status": 201
      }
    },
    {
      "create": {
        "_index": "registry",
        "_id": "7",
        "_version": 1,
        "result": "created",
        "_shards": {
          "total": 2,
          "successful": 1,
          "failed": 0
        },
        "_seq_no": 5,
        "_primary_term": 1,
        "status": 201
      }
    },
    {
      "create": {
        "_index": "registry",
        "_id": "8",
        "_version": 1,
        "result": "created",
        "_shards": {
          "total": 2,
          "successful": 1,
          "failed": 0
        },
        "_seq_no": 1,
        "_primary_term": 1,
        "status": 201
      }
    },
    {
      "create": {
        "_index": "registry",
        "_id": "9",
        "_version": 1,
        "result": "created",
        "_shards": {
          "total": 2,
          "successful": 1,
          "failed": 0
        },
        "_seq_no": 6,
        "_primary_term": 1,
        "status": 201
      }
    }
  ]
}

Important: in the above command, that the ref is defined and not peter.

Using a simple query test:

{
  "from": 0,
  "size": 10,
  "query": {
    "bool": {
      "must": [
        {
          "term": {
            "ref": {
              "value": "7",
              "boost": 1
            }
          }
        }
      ],
      "adjust_pure_negative": true,
      "boost": 1
    }
  },
  "_source": {
    "includes": [
        "name", "ref"
    ],
    "excludes": [
      "ops:Label_File_Info/ops:blob",
      "ops:Label_File_Info/ops:json_blob"
    ]
  }
}

it fails to find the ref as shown:

$ curl --insecure --location --request GET 'https://localhost:9200/registry/_search' --header 'Accept: application/json' --header 'Content-Type: application/json' --user admin:admin -d @Projects/PDS/tq1.json | jq
{
  "took": 108,
  "timed_out": false,
  "_shards": {
    "total": 2,
    "successful": 2,
    "skipped": 0,
    "failed": 0
  },
  "hits": {
    "total": {
      "value": 0,
      "relation": "eq"
    },
    "max_score": null,
    "hits": []
  }
}

Now, the same search works if

1. peter is changed to ref
2. dynamic is set true (when inserting data opensearch adds ref to the mapping automagically)

Therefore this ticket cannot be fixed until the registry is fixed.

It also brings up another point, better to have references rather than <product>_ref or ref_lid_collections or ref_lidvid_collections. Breaking the references apart means more mappings to keep up to date. Does not matter if dynamic is true so much but will then need to scan all keywords and look for a pattern then look over it in the Java. It will be really slow vs just having references.

[al-niessner]

@tloubrieu-jpl @jordanpadams

See above comment about dynamic.

[jordanpadams]

@al-niessner so if I understanding this correctly, it sounds like we just need to change the OpenSearch config to have dynamic=true, and this problem should be fixed?

[al-niessner]

@jordanpadams

Yes and no. The default is dynamic=true. For some reason harvest or some harvesty process is setting it to false. I do not know why but it was a conscious decision by someone to make it false. Not a typo or forgot to set it true. So, yes, we can make it true again but that will come with consequences that someone thought better to avoid; so no.

[al-niessner]

@jordanpadams @tloubrieu-jpl

I would suggest making something like references and putting it in the mapping while leaving dynamic=false. Then I would put all references secondary, product, etc into references. You can keep the other but registry-api does not need them nor wants them. In any case, these are harvest changes.

[jordanpadams]

@al-niessner from the commit history here: https://github.com/NASA-PDS/registry-mgr/commits/main/src/main/resources/elastic/refs.json, it doesn't appear there is any particular reason mentioned for setting dynamic=false. but I also wouldn't say I am following all of this very intently.

we may need to have a discussion with @tloubrieu-jpl and @jimmie to discuss options. preferably the option that requires the least amount of refactoring of the existing indices and software.

[jordanpadams]

@al-niessner just as a heads up, no idea how it worked before, but apparently in our Build 12.0 testing, this requirement was tested and satisfied with https://github.com/NASA-PDS/registry-api-service/releases/tag/v0.4.2 . not worth rehashing but somehow it worked 🤷 .

definitely don't do anything different, was just trying to track down how this bug made it into the current release.

[al-niessner]

fixed by NASA-PDS/registry-mgr#50