Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Enterprise Search] Add .connector-secrets system index #104766

Merged
merged 26 commits into from
Jan 26, 2024

Conversation

navarone-feekery
Copy link
Contributor

@navarone-feekery navarone-feekery commented Jan 25, 2024

Context

These changes were already approved and merged in this PR.
However, they ended up blocking CI by introducing merge conflicts that weren't picked up by github. The change was reverted here.
This PR is a cherry-pick of the previous PR's commits, with an extra final commit added to address the merge conflict here.


Original PR Content:

The Search team is currently implementing BYOI (Bring Your Own Index) for Connectors. This feature will allow users to create search indices for Connectors without having search- prefix the index name.
Native Connectors are run using the Enterprise Search system account, which has access to search indices because it can access all indices that follow the pattern search-*. To continue allowing Native Connectors to function after removing the prefix requirement, we need to switch to using API keys to manage index access. That brings us to this PR.

The goal of this PR is to have a place where Native Connectors can store API keys (secrets storage). A connector secret can be stored from either Kibana or the Connectors CLI. Kibana would only have write access to this index so it cannot expose secrets in the UI. Enterprise Search (which runs the Connectors CLI on cloud) will have both read and write access.

These changes follow the implementation design of Fleet's secrets storage (example PR). The implementation is largely identical, with a change in namespace.

Changes

  • Introduce new internal system index called .connector-secrets. This will be used to store the API keys that Native Connectors will use.
  • Add GET and POST requests for connector secrets
  • Give the Kibana system account write access to .connector-secrets
  • Give the Enterprise Search account read and write access to .connector-secrets

Not included

  • To reduce PR size, DELETE requests will be added in a further PR

How to test this PR

The API endpoints can be tested in Kibana Dev Tools:

  1. Run ES from source with ./gradlew run.

  2. Once ES is running, change the default password for the elastic user from "password" to "changeme" to match Kibana's:

    curl -k -u elastic-admin:elastic-password -H 'Content-Type: application/json' \
    http://localhost:9200/_security/user/elastic/_password -d'{"password": "changeme"}'
    curl -k -u elastic:changeme -H 'Content-Type: application/json' \
     http://localhost:9200/_security/user/kibana_system/_password -d'{"password": "changeme"}'
    
  3. Run Kibana with yarn start.

  4. Log into Kibana and go to /app/dev_tools#/console.

  5. Create a secret:

    POST /_connector/secret/
    {
      "value": "test123"
    }
    
  6. This should succeed and return the id of the created secret doc, e.g.:

    {
      "id": "1234"
    }
    
  7. Copy the id and try to get the secret:

    GET /_connector/secret/1234
    
  8. This should succeed and return the secret id and value:

    {
      "id": "1234",
      "value": "test123"
    }
    
  9. Try to get a different secret using a random value. It should fail with 404 and return an error.

    GET /_connector/secret/notrealid
    

@navarone-feekery navarone-feekery requested a review from a team as a code owner January 25, 2024 14:56
Copy link
Contributor

Documentation preview:

@elasticsearchmachine
Copy link
Collaborator

Pinging @elastic/ent-search-eng (Team:Enterprise Search)

Copy link
Member

@jedrazb jedrazb left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM 🚀

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
:EnterpriseSearch/Application Enterprise Search >non-issue Team:Enterprise Search Meta label for Enterprise Search team v8.13.0
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

3 participants