[BUG] security_analytics_exception, Can't upsert index template for concrete index!

[StefanSa]

What is the bug?
i would like to create a new detector, source is a datastream index with logfile content,
but get this error message when saving the detectors.

os-dash01 opensearch-dashboards[7806]:   body: {
os-dash01 opensearch-dashboards[7806]:     error: {
os-dash01 opensearch-dashboards[7806]:       root_cause: [Array],
os-dash01 opensearch-dashboards[7806]:       type: 'security_analytics_exception',
os-dash01 opensearch-dashboards[7806]:       reason: "Can't upsert index template for concrete index!",
os-dash01 opensearch-dashboards[7806]:       caused_by: [Object]
os-dash01 opensearch-dashboards[7806]:     },
os-dash01 opensearch-dashboards[7806]:     status: 500

How can one reproduce the bug?
Use the detector wizard and try to save.

What is the expected behavior?
Successfully create a detector without any problems.

What is your host/environment?

• OS: [Windows 11]
• Version [opensearch 2.9]
• 3 client, master and data nodes

Do you have any screenshots?

Do you have any additional context?
opensearch-dashboard log:

2023-08-02T09:10:15.913757+02:00 os-dash01 opensearch-dashboards[7806]: Security Analytics - DetectorsService - getDetector: StatusCodeError: [security_analytics_exception] Can't upsert index template for concrete index!
2023-08-02T09:10:15.913826+02:00 os-dash01 opensearch-dashboards[7806]:     at respond (/usr/share/opensearch-dashboards/node_modules/elasticsearch/src/lib/transport.js:349:15)
2023-08-02T09:10:15.913859+02:00 os-dash01 opensearch-dashboards[7806]:     at checkRespForFailure (/usr/share/opensearch-dashboards/node_modules/elasticsearch/src/lib/transport.js:306:7)
2023-08-02T09:10:15.913883+02:00 os-dash01 opensearch-dashboards[7806]:     at HttpConnector.<anonymous> (/usr/share/opensearch-dashboards/node_modules/elasticsearch/src/lib/connectors/http.js:173:7)
2023-08-02T09:10:15.913905+02:00 os-dash01 opensearch-dashboards[7806]:     at IncomingMessage.wrapper (/usr/share/opensearch-dashboards/node_modules/lodash/lodash.js:4991:19)
2023-08-02T09:10:15.913928+02:00 os-dash01 opensearch-dashboards[7806]:     at IncomingMessage.emit (node:events:525:35)
2023-08-02T09:10:15.913950+02:00 os-dash01 opensearch-dashboards[7806]:     at IncomingMessage.emit (node:domain:489:12)
2023-08-02T09:10:15.913975+02:00 os-dash01 opensearch-dashboards[7806]:     at endReadableNT (node:internal/streams/readable:1358:12)
2023-08-02T09:10:15.913997+02:00 os-dash01 opensearch-dashboards[7806]:     at processTicksAndRejections (node:internal/process/task_queues:83:21) {
2023-08-02T09:10:15.914020+02:00 os-dash01 opensearch-dashboards[7806]:   status: 500,
2023-08-02T09:10:15.914042+02:00 os-dash01 opensearch-dashboards[7806]:   displayName: 'InternalServerError',
2023-08-02T09:10:15.914068+02:00 os-dash01 opensearch-dashboards[7806]:   path: '/_plugins/_security_analytics/mappings',
2023-08-02T09:10:15.914094+02:00 os-dash01 opensearch-dashboards[7806]:   query: {},
2023-08-02T09:10:15.914127+02:00 os-dash01 opensearch-dashboards[7806]:   body: {
2023-08-02T09:10:15.914150+02:00 os-dash01 opensearch-dashboards[7806]:     error: {
2023-08-02T09:10:15.914172+02:00 os-dash01 opensearch-dashboards[7806]:       root_cause: [Array],
2023-08-02T09:10:15.914194+02:00 os-dash01 opensearch-dashboards[7806]:       type: 'security_analytics_exception',
2023-08-02T09:10:15.914216+02:00 os-dash01 opensearch-dashboards[7806]:       reason: "Can't upsert index template for concrete index!",
2023-08-02T09:10:15.914248+02:00 os-dash01 opensearch-dashboards[7806]:       caused_by: [Object]
2023-08-02T09:10:15.914271+02:00 os-dash01 opensearch-dashboards[7806]:     },
2023-08-02T09:10:15.914293+02:00 os-dash01 opensearch-dashboards[7806]:     status: 500
2023-08-02T09:10:15.914314+02:00 os-dash01 opensearch-dashboards[7806]:   },
2023-08-02T09:10:15.914347+02:00 os-dash01 opensearch-dashboards[7806]:   statusCode: 500,
2023-08-02T09:10:15.914370+02:00 os-dash01 opensearch-dashboards[7806]:   response: `{"error":{"root_cause":[{"type":"security_analytics_exception","reason":"Can't upsert index template for concrete index!"}],"type":"security_analytics_exception","reason":"Can't upsert index template for concrete index!","caused_by":{"type":"exception","reason":"java.lang.IllegalStateException: Can't upsert index template for concrete index!"}},"status":500}`,
2023-08-02T09:10:15.914395+02:00 os-dash01 opensearch-dashboards[7806]:   toString: [Function (anonymous)],
2023-08-02T09:10:15.914417+02:00 os-dash01 opensearch-dashboards[7806]:   toJSON: [Function (anonymous)]
2023-08-02T09:10:15.914440+02:00 os-dash01 opensearch-dashboards[7806]: }
2023-08-02T09:10:15.916031+02:00 os-dash01 opensearch-dashboards[7806]: {"type":"response","@timestamp":"2023-08-02T07:10:15Z","tags":[],"pid":7806,"method":"post","statusCode":200,"req":{"url":"/_plugins/_security_analytics/mappings","method":"post","headers":{"host":"os-dash01.feltengroup.local:5601","connection":"keep-alive","content-length":"173","sec-ch-ua":"\"Not/A)Brand\";v=\"99\", \"Microsoft Edge\";v=\"115\", \"Chromium\";v=\"115\"","content-type":"application/json","osd-xsrf":"osd-fetch","sec-ch-ua-mobile":"?0","user-agent":"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/115.0.0.0 Safari/537.36 Edg/115.0.1901.183","osd-version":"2.9.0","sec-ch-ua-platform":"\"Windows\"","accept":"*/*","origin":"https://os-dash01.feltengroup.local:5601","sec-fetch-site":"same-origin","sec-fetch-mode":"cors","sec-fetch-dest":"empty","referer":"https://os-dash01.feltengroup.local:5601/app/opensearch_security_analytics_dashboards","accept-encoding":"gzip, deflate, br","accept-language":"de,de-DE;q=0.9,en;q=0.8,en-GB;q=0.7,en-US;q=0.6","securitytenant":""},"remoteAddress":"172.16.33.110","userAgent":"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/115.0.0.0 Safari/537.36 Edg/115.0.1901.183","referer":"https://os-dash01.feltengroup.local:5601/app/opensearch_security_analytics_dashboards"},"res":{"statusCode":200,"responseTime":108,"contentLength":9},"message":"POST /_plugins/_security_analytics/mappings 200 108ms - 9.0B"}
2023-08-02T09:10:20.180392+02:00 os-dash01 opensearch-dashboards[7806]: {"type":"response","@timestamp":"2023-08-02T07:10:15Z","tags":[],"pid":7806,"method":"post","statusCode":200,"req":{"url":"/_plugins/_security_analytics/detectors","method":"post","headers":{"host":"os-dash01.feltengroup.local:5601","connection":"keep-alive","content-length":"2228","sec-ch-ua":"\"Not/A)Brand\";v=\"99\", \"Microsoft Edge\";v=\"115\", \"Chromium\";v=\"115\"","content-type":"application/json","osd-xsrf":"osd-fetch","sec-ch-ua-mobile":"?0","user-agent":"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/115.0.0.0 Safari/537.36 Edg/115.0.1901.183","osd-version":"2.9.0","sec-ch-ua-platform":"\"Windows\"","accept":"*/*","origin":"https://os-dash01.feltengroup.local:5601","sec-fetch-site":"same-origin","sec-fetch-mode":"cors","sec-fetch-dest":"empty","referer":"https://os-dash01.feltengroup.local:5601/app/opensearch_security_analytics_dashboards","accept-encoding":"gzip, deflate, br","accept-language":"de,de-DE;q=0.9,en;q=0.8,en-GB;q=0.7,en-US;q=0.6","securitytenant":""},"remoteAddress":"172.16.33.110","userAgent":"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/115.0.0.0 Safari/537.36 Edg/115.0.1901.183","referer":"https://os-dash01.feltengroup.local:5601/app/opensearch_security_analytics_dashboards"},"res":{"statusCode":200,"responseTime":4366,"contentLength":9},"message":"POST /_plugins/_security_analytics/detectors 200 4366ms - 9.0B"}

[StefanSa]

Interestingly, the detector seems to work, so it can read data from the datastream index, but it can't save the configuration.
When i open a two session, i see this:

[amsiglan]

Hi @StefanSa is the issue resolved on your end?

[StefanSa]

Hi @amsiglan
This seems to be fixed now.
I was able to create and save a Detector.
But now i have bigger problems to stop or delete it, see here:
#618

How do we get this new problem solved,
currently i can't use this really interesting tool because of such bugs.

[amsiglan]

Okay, closing this issue in favor of #618

[jhill-cmd]

Observing the same issue while creating a detector on a datastream-* (they have the same mapping)

but if i take on specific datastream, it gives: #768

[kritikashahi]

@StefanSa Can you shed light on what worked for you here. I am facing the same issue while creating detector based on datastream indices.