[Enterprise Search] Add .connector-secrets system index and GET/POST requests

[navarone-feekery]

Context

The Search team is currently implementing BYOI (Bring Your Own Index) for Connectors. This feature will allow users to create search indices for Connectors without having search- prefix the index name.
Native Connectors are run using the Enterprise Search system account, which has access to search indices because it can access all indices that follow the pattern search-*. To continue allowing Native Connectors to function after removing the prefix requirement, we need to switch to using API keys to manage index access. That brings us to this PR.

The goal of this PR is to have a place where Native Connectors can store API keys (secrets storage). A connector secret can be stored from either Kibana or the Connectors CLI. Kibana would only have write access to this index so it cannot expose secrets in the UI. Enterprise Search (which runs the Connectors CLI on cloud) will have both read and write access.

These changes follow the implementation design of Fleet's secrets storage (example PR). The implementation is largely identical, with a change in namespace.

Changes

• Introduce new internal system index called .connector-secrets. This will be used to store the API keys that Native Connectors will use.
• Add GET and POST requests for connector secrets
• Give the Kibana system account write access to .connector-secrets
• Give the Enterprise Search account read and write access to .connector-secrets

Not included

• To reduce PR size, DELETE requests will be added in a further PR

How to test this PR

The API endpoints can be tested in Kibana Dev Tools:

1. Run ES from source with ./gradlew run.

2. Once ES is running, change the default password for the elastic user from "password" to "changeme" to match Kibana's:

   curl -k -u elastic-admin:elastic-password -H 'Content-Type: application/json' \
   http://localhost:9200/_security/user/elastic/_password -d'{"password": "changeme"}'
   curl -k -u elastic:changeme -H 'Content-Type: application/json' \
    http://localhost:9200/_security/user/kibana_system/_password -d'{"password": "changeme"}'
   

3. Run Kibana with yarn start.

4. Log into Kibana and go to /app/dev_tools#/console.

5. Create a secret:

   POST /_connector/secret/
   {
     "value": "test123"
   }
   

6. This should succeed and return the id of the created secret doc, e.g.:

   {
     "id": "1234"
   }
   

7. Copy the id and try to get the secret:

   GET /_connector/secret/1234
   

8. This should succeed and return the secret id and value:

   {
     "id": "1234",
     "value": "test123"
   }
   

9. Try to get a different secret using a random value. It should fail with 404 and return an error.

   GET /_connector/secret/notrealid
   

[elasticsearchmachine]

Pinging @elastic/ent-search-eng (Team:Enterprise Search)

[mark-vieira]

@breskeby we should think about making it simpler to setup clusters that require security w/o having to use the default distribution. I suspect the vast majority of tests that use the default distro is for this reason alone.

Otherwise these changes LGTM.

[kderusso]

This looks great! A couple of questions and notes.

The only other thing that I'd request before merging is adding unit tests extending AbstractBWCSerializationTestCase for your request and response objects. After that I'm ready to 👍. Thank you!

[timgrein]

Really nice work and congrats on your (I think) first PR in ES 👏 And thank you for the detailed PR description!

Left some minor comments, which are non-blocking and having one question: Do you plan to add the Backwards compatibility tests, index service tests and other unit tests, too?

[jedrazb]

Great work! 🚀 Left some minor comments

[timgrein]

One additional question around endpoint naming 🙏

[carlosdelest]

Not for this PR, but we should consider creating separate directories for each Ent Search feature - this is becoming crowded

[kderusso]

++

[navarone-feekery]

@timgrein I've added BWC and index service tests. Let me know if there are more tests I should add.
@carlosdelest I've removed the unnecessary index tests now that we rely on origin settings + privileges.

[carlosdelest]

LGTM. Thanks for iterating on this! 🚀