[BUG] KEYSTORE_PASSWORD Failed to read keystore password on console

[tastyfrankfurt]

Describe the bug

When setting the KEYSTORE_PASSWORD environment variable the opensearch start command unsets this environment variable. The if statement then errors out as failed to read keystore password on conole.

Related component

Build

To Reproduce

1. Password protect the opensearch keystore using the opensearch-keystore command.
2. Set the environment variable KEYSTORE_PASSWORD on systemd
3. Attempt to start the service
4. Service fails to start and reports "Failed to read keystore password on console"

Expected behavior

Service starts and decrypts all secrets in the keystore.

Additional Details

Plugins
Please list all plugins currently enabled.

Screenshots
If applicable, add screenshots to help explain your problem.

Host/Environment (please complete the following information):

• OS: [e.g. iOS]
• Version [e.g. 22]

Additional context
Add any other context about the problem here.

[peternied]

[Triage - attendees 1 2 3 4 5 6]
@tastyfrankfurt Thanks for filing this issue, we'd welcome a pull request to resolve this issue.

@derek-ho Do you have any context around environment variable usage related to the recent changes for default password that would apply here?

[tastyfrankfurt]

@derek-ho For us this issue has been around since the projects inception, it would just be a matter of adjusting the OpenSearch file to handle the KEYSTORE_PASSWORD as an environment variable. We deploy OpenSearch using tar.gz using ansible on Ubuntu. We haven't had a requirement for this feature to work until now.

Happy to put in pull request, but will need to work through the process of how to do this appropriately for this project.

[derek-ho]

@tastyfrankfurt @peternied this is not related to the default password, but let me see if some of our experience might help. Can you share what platform you are using? Are you using the ansible-playbook? Here we have admin-password being set into the playbook: https://github.com/search?q=repo%3Aopensearch-project%2Fansible-playbook%20admin_password&type=code, but not too familiar with how the KEYSTORE_PASSWORD is being used by Opensearch, can you share some more around that? If I am reading the situation right, you may need to make a PR against the ansible playbook to pass in the env variable similar to this - https://github.com/opensearch-project/ansible-playbook/blob/d1a1af02b2a2b9994dc0748bf6abfb3b0c7c7c5d/roles/linux/opensearch/tasks/security.yml#L225. Let me know if that solves for your use case!

[derek-ho]

I think this is the correct file actually: https://github.com/opensearch-project/ansible-playbook/blob/d1a1af02b2a2b9994dc0748bf6abfb3b0c7c7c5d/roles/linux/opensearch/tasks/opensearch.yml

[tastyfrankfurt]

@derek-ho @peternied
The code i have issue with is in the link attached,

OpenSearch/distribution/src/bin/opensearch

Line 39 in e97bee8

unset KEYSTORE_PASSWORD

Basically lines 39 and 40, make the environment variable equal nothing. Also CHECK_KEYSTORE is set to true implicitly with no check for an existing environment variable. My code changes would be to
[[ -z "${CHECK_KEYSTORE }" ]] && CHECK_KEYSTORE=true

and delete lines 39 and 40