[BUG][OpenSearch|OpenSearch Dashboards] error when saving search

[atalakey4work]

Describe the bug
After upgrading to OpenSearch chart version 2.25.0 and OpenSearch Dashboards chart version 2.22.0, we get error when we try to save a Discover search.

To Reproduce
Steps to reproduce the behavior:

1. Go to OpenSearch Dashboard Discover
2. Click on 'Save'
3. Inout any title
4. Click on 'Save'

Expected behavior
Discover search is saved with no errors

Chart Name
OpenSearch and OpenSearch Dashboards

Screenshots

Host/Environment (please complete the following information):

• Helm Version: OpenSearch (2.25.0), OpenSearch Dashboards (2.22.0)
• Kubernetes Version: v1.30.1

Additional context
Here is the error from the longs:

{"type":"log","@timestamp":"2024-09-22T04:16:47Z","tags":["error","opensearch","data"],"pid":1,"message":"[security_exception]: no permissions for [] and User [name=admin, backend_roles=[admin], requestedTenant=null]"}

{"type":"response","@timestamp":"2024-09-22T04:16:47Z","tags":[],"pid":1,"method":"post","statusCode":403,"req":{"url":"/api/saved_objects/search?overwrite=true","method":"post","headers":{"host":"[MASKED]","x-request-id":"cd1b40e738290336e5236bd11c7e11f3","x-real-ip":"11.0.0.5","x-forwarded-for":"[MASKED]","x-forwarded-host":"[MASKED]","x-forwarded-port":"443","x-forwarded-proto":"https","x-forwarded-scheme":"https","x-scheme":"https","content-length":"599","sec-ch-ua":"\"Chromium\";v=\"128\", \"Not;A=Brand\";v=\"24\", \"Google Chrome\";v=\"128\"","sec-ch-ua-platform":"\"[MASKED]\"","osd-xsrf":"osd-fetch","sec-ch-ua-mobile":"?0","user-agent":"[MASKED]","osd-version":"2.16.0","content-type":"application/json","accept":"*/*","origin":"https://[MASKED]","sec-fetch-site":"same-origin","sec-fetch-mode":"cors","sec-fetch-dest":"empty","referer":"https://[MASKED]/app/data-explorer/discover","accept-encoding":"gzip, deflate, br, zstd","accept-language":"en-US,en;q=0.9,ar;q=0.8","priority":"u=1, i"},"remoteAddress":"10.244.1.199","userAgent":"[MASKED]","referer":"https://[MASKED]/app/data-explorer/discover"},"res":{"statusCode":403,"responseTime":19,"contentLength":9},"message":"POST /api/saved_objects/search?overwrite=true 403 19ms - 9.0B"}

[gaiksaya]

[Triage]

Hi @atalakey4work ,
Looks like 2.25.0 points to 2.17.0 OpenSearch and 2.22.0 points to 2.16.0 OpenSearch-Dashboards.
Can you try using the same version of both the products and see if you are still seeing the issue?
Thanks!

[atalakey4work]

When i use the same version the issue is even worse. When i open a previously saved Discover search i get the permission error.

[peterzhuamazon]

Adding @ashwin-pc @AMoo-Miki @cwperks to take a look on this.
Seems like this is more related to the OSD core or security plugin.

Thanks.

[AMoo-Miki]

Do you have the OpenSearch logs for the 403? That would provide the reason for the permission being denied.

Also, what versions were being used prior to the upgrade?

I am not familiar with this repo but could this have anything to do with the strong password requirements?

[atalakey4work]

@AMoo-Miki I was able to resolve the error by updating the below in my opensearch_dashboarfs.yml

Before:
opensearch.requestHeadersAllowlist: ["Authorization", "securitytenant"]

After:
opensearch.requestHeadersAllowlist: ["Authorization"]

Now save works but whenever I open a saved search the saerch filters are not populated:

Here is a copy of the kibanaSavedObjectMeta.searchSourceJSON

{
  "query": {
    "query": "",
    "language": "kuery"
  },
  "highlightAll": true,
  "version": true,
  "aggs": {
    "2": {
      "date_histogram": {
        "field": "@timestamp",
        "fixed_interval": "30s",
        "time_zone": "",
        "min_doc_count": 1
      }
    }
  },
  "filter": [
    {
      "meta": {
        "alias": null,
        "negate": false,
        "disabled": false,
        "type": "phrase",
        "key": "kubernetes.namespace_name",
        "params": {
          "query": "nginx-ingress"
        },
        "indexRefName": "kibanaSavedObjectMeta.searchSourceJSON.filter[0].meta.index"
      },
      "query": {
        "match_phrase": {
          "kubernetes.namespace_name": "nginx-ingress"
        }
      },
      "$state": {
        "store": "appState"
      }
    }
  ],
  "indexRefName": "kibanaSavedObjectMeta.searchSourceJSON.index"
}