Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Updates rpm and deb distribution to adapt to admin password change #4332

Merged
merged 14 commits into from
Jan 19, 2024

Conversation

DarshitChanpura
Copy link
Member

Description

This PR is an outcome of a design decision proposed here: opensearch-project/security#3916

Since an initial admin password is required starting OpenSearch version 2.12 and later, this PR adds two checks:

  • A preinstall check to verify if OPENSEARCH_INITIAL_ADMIN_PASSWORD was set. If yes, do nothing. If not print a helpful message and exit the installation.
  • A postinstall check to verify if demo_config threw any errors. If so print a helpful a message and exit the postinstall checlk. (This will require users to re-install with admin password defined but that is not a big task as they can simply re-run installation again without needing to delete any pre-installed folders. I have tested this and it works)

Issues Resolved

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

Copy link
Contributor

@derek-ho derek-ho left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Need version check

Copy link

codecov bot commented Jan 9, 2024

Codecov Report

All modified and coverable lines are covered by tests ✅

Comparison is base (4b0b53d) 91.35% compared to head (b6d7976) 91.35%.
Report is 11 commits behind head on main.

Additional details and impacted files
@@           Coverage Diff           @@
##             main    #4332   +/-   ##
=======================================
  Coverage   91.35%   91.35%           
=======================================
  Files         190      190           
  Lines        6175     6175           
=======================================
  Hits         5641     5641           
  Misses        534      534           

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

@DarshitChanpura DarshitChanpura force-pushed the update-rpm-deb branch 2 times, most recently from 6f21de0 to d9a9a63 Compare January 10, 2024 02:11
Signed-off-by: Darshit Chanpura <[email protected]>
Signed-off-by: Darshit Chanpura <[email protected]>
@DarshitChanpura
Copy link
Member Author

Debian logs:
ubuntu@ip:~/test$ sudo dpkg -i opensearch-2.12.0-linux-x64.deb
Selecting previously unselected package opensearch.
(Reading database ... 109974 files and directories currently installed.)
Preparing to unpack opensearch-2.12.0-linux-x64.deb ...
Running OpenSearch Pre-Installation Script
ERROR: Opensearch 2.12 and later requires the env variable OPENSEARCH_INITIAL_ADMIN_PASSWORD to be defined to setup the opensearch-security demo configuration
dpkg: error processing archive opensearch-2.12.0-linux-x64.deb (--install):
 new opensearch package pre-installation script subprocess returned error exit status 1
Errors were encountered while processing:
 opensearch-2.12.0-linux-x64.deb
ubuntu@ip:~/test$ sudo env OPENSEARCH_INITIAL_ADMIN_PASSWORD=admin -i opensearch-2.12.0-linux-x64.deb
env: ‘-i’: No such file or directory
ubuntu@ip:~/test$ sudo env OPENSEARCH_INITIAL_ADMIN_PASSWORD=admin dpkg -i opensearch-2.12.0-linux-x64.deb
(Reading database ... 109974 files and directories currently installed.)
Preparing to unpack opensearch-2.12.0-linux-x64.deb ...
Running OpenSearch Pre-Installation Script
Unpacking opensearch (2.12.0) ...
Setting up opensearch (2.12.0) ...
Running OpenSearch Post-Installation Script
ERROR: Something went wrong during demo configuration installation. Please see the logs in /var/log/opensearch/install_demo_configuration.log.
dpkg: error processing package opensearch (--install):
 installed opensearch package post-installation script subprocess returned error exit status 1
Processing triggers for systemd (245.4-4ubuntu3.22) ...
Errors were encountered while processing:
 opensearch
ubuntu@ip:~/test$ cat /var/log/opensearch/install_demo_configuration.log
### OpenSearch Security Demo Installer
### ** Warning: Do not use on production or public reachable systems **
OpenSearch install type: rpm/deb on Linux 5.15.0-1048-aws amd64
OpenSearch config dir: /etc/opensearch/
OpenSearch config file: /etc/opensearch/opensearch.yml
OpenSearch bin dir: /usr/share/opensearch/bin/
OpenSearch plugins dir: /usr/share/opensearch/plugins/
OpenSearch lib dir: /usr/share/opensearch/lib/
Detected OpenSearch Version: 2.12.0
Detected OpenSearch Security Version: 2.12.0.0
Password admin is weak. Please re-try with a stronger password.
ubuntu@ip:~/test$
ubuntu@ip:~/test$
ubuntu@ip:~/test$
ubuntu@ip:~/test$ sudo env OPENSEARCH_INITIAL_ADMIN_PASSWORD=myStrongPassword123! dpkg -i opensearch-2.12.0-linux-x64.deb
(Reading database ... 110841 files and directories currently installed.)
Preparing to unpack opensearch-2.12.0-linux-x64.deb ...
Running OpenSearch Pre-Removal Script
Running OpenSearch Pre-Installation Script
Unpacking opensearch (2.12.0) over (2.12.0) ...
Setting up opensearch (2.12.0) ...
Running OpenSearch Post-Installation Script
### NOT starting on installation, please execute the following statements to configure opensearch service to start automatically using systemd
 sudo systemctl daemon-reload
 sudo systemctl enable opensearch.service
### You can start opensearch service by executing
 sudo systemctl start opensearch.service
### Create opensearch demo certificates in /etc/opensearch/
 See demo certs creation log in /var/log/opensearch/install_demo_configuration.log
### Upcoming breaking change in packaging
 In a future release of OpenSearch, we plan to change the permissions associated with access to installed files
 If you are configuring tools that require read access to the OpenSearch configuration files, we recommend you add the user that runs these tools to the 'opensearch' group
 For more information, see https://github.com/opensearch-project/opensearch-build/pull/4043
Processing triggers for systemd (245.4-4ubuntu3.22) ...
ubuntu@ip:~/test$ cat /var/log/opensearch/install_demo_configuration.log
### OpenSearch Security Demo Installer
### ** Warning: Do not use on production or public reachable systems **
OpenSearch install type: rpm/deb on Linux 5.15.0-1048-aws amd64
OpenSearch config dir: /etc/opensearch/
OpenSearch config file: /etc/opensearch/opensearch.yml
OpenSearch bin dir: /usr/share/opensearch/bin/
OpenSearch plugins dir: /usr/share/opensearch/plugins/
OpenSearch lib dir: /usr/share/opensearch/lib/
Detected OpenSearch Version: 2.12.0
Detected OpenSearch Security Version: 2.12.0.0
Admin password set successfully.
### Success
### Execute this script now on all your nodes and then start all nodes
### OpenSearch Security will be automatically initialized.
### If you like to change the runtime configuration
### change the files in ../../../config/opensearch-security and execute:
sudo "/usr/share/opensearch/plugins/opensearch-security/tools/securityadmin.sh" -cd "/etc/opensearch/opensearch-security" -icl -key "/etc/opensearch/kirk-key.pem" -cert "/etc/opensearch/kirk.pem" -cacert "/etc/opensearch/root-ca.pem" -nhnv
### or run ./securityadmin_demo.sh
### To use the Security Plugin ConfigurationGUI
### To access your secured cluster open https://<hostname>:<HTTP port> and log in with admin/<your-custom-admin-password>.
### (Ignore the SSL certificate warning because we installed self-signed demo certificates)
ubuntu@ip:~/test$
ubuntu@ip:~/test$
ubuntu@ip:~/test$ sudo systemctl enable opensearch
Synchronizing state of opensearch.service with SysV service script with /lib/systemd/systemd-sysv-install.
Executing: /lib/systemd/systemd-sysv-install enable opensearch
Created symlink /etc/systemd/system/multi-user.target.wants/opensearch.service → /lib/systemd/system/opensearch.service.
ubuntu@ip:~/test$ sudo systemctl start opensearch
ubuntu@ip:~/test$
ubuntu@ip:~/test$
ubuntu@ip:~/test$ curl https://localhost:9200 -ku admin:admin
ubuntu@ip:~/test$ curl https://localhost:9200 -ku admin:myStrongPassword123!
{
  "name" : "smoketestnode",
  "cluster_name" : "opensearch",
  "cluster_uuid" : "8WrWnRGxQhiXH2R4uBxU1Q",
  "version" : {
    "distribution" : "opensearch",
    "number" : "2.12.0",
    "build_type" : "deb",
    "build_hash" : "334636f62662a886a51edd39e7e81f8e80ab9e14",
    "build_date" : "2024-01-10T03:00:44.565275404Z",
    "build_snapshot" : false,
    "lucene_version" : "9.8.0",
    "minimum_wire_compatibility_version" : "7.10.0",
    "minimum_index_compatibility_version" : "7.0.0"
  },
  "tagline" : "The OpenSearch Project: https://opensearch.org/"
}
ubuntu@ip:~/test$

@DarshitChanpura
Copy link
Member Author

RPM logs:
[ec2-userip ~]$ sudo yum install opensearch-2.12.0-linux-x64.rpm
Updating Subscription Management repositories.
Unable to read consumer identity

This system is not registered with an entitlement server. You can use subscription-manager to register.

Last metadata expiration check: 0:17:50 ago on Wed 10 Jan 2024 05:24:48 AM UTC.
Dependencies resolved.
============================================================================================================================================================================================================
 Package                                           Architecture                                  Version                                          Repository                                           Size
============================================================================================================================================================================================================
Installing:
 opensearch                                        x86_64                                        2.12.0-1                                         @commandline                                        277 M

Transaction Summary
============================================================================================================================================================================================================
Install  1 Package

Total size: 277 M
Installed size: 466 M
Is this ok [y/N]: y
Downloading Packages:
Running transaction check
Transaction check succeeded.
Running transaction test
Transaction test succeeded.
Running transaction
  Preparing        :                                                                                                                                                                                    1/1
  Running scriptlet: opensearch-2.12.0-1.x86_64                                                                                                                                                         1/1
ERROR: Opensearch 2.12 and later requires the env variable OPENSEARCH_INITIAL_ADMIN_PASSWORD to be defined to setup the opensearch-security demo configuration
error: %prein(opensearch-2.12.0-1.x86_64) scriptlet failed, exit status 1

Error in PREIN scriptlet in rpm package opensearch
  Verifying        : opensearch-2.12.0-1.x86_64                                                                                                                                                         1/1
Installed products updated.

Failed:
  opensearch-2.12.0-1.x86_64

Error: Transaction failed
[ec2-userip ~]$ sudo yum list | grep opensearch
[ec2-userip ~]$
[ec2-userip ~]$
[ec2-userip ~]$ sudo env OPENSEARCH_INITIAL_ADMIN_PASSWORD=admin yum install opensearch-2.12.0-linux-x64.rpm
Updating Subscription Management repositories.
Unable to read consumer identity

This system is not registered with an entitlement server. You can use subscription-manager to register.

Last metadata expiration check: 0:18:57 ago on Wed 10 Jan 2024 05:24:48 AM UTC.
Dependencies resolved.
============================================================================================================================================================================================================
 Package                                           Architecture                                  Version                                          Repository                                           Size
============================================================================================================================================================================================================
Installing:
 opensearch                                        x86_64                                        2.12.0-1                                         @commandline                                        277 M

Transaction Summary
============================================================================================================================================================================================================
Install  1 Package

Total size: 277 M
Installed size: 466 M
Is this ok [y/N]: y
Downloading Packages:
Running transaction check
Transaction check succeeded.
Running transaction test
Transaction test succeeded.
Running transaction
  Preparing        :                                                                                                                                                                                    1/1
  Running scriptlet: opensearch-2.12.0-1.x86_64                                                                                                                                                         1/1
  Installing       : opensearch-2.12.0-1.x86_64                                                                                                                                                         1/1
  Running scriptlet: opensearch-2.12.0-1.x86_64                                                                                                                                                         1/1
ERROR: Something went wrong during demo configuration installation. Please see the logs in /var/log/opensearch/install_demo_configuration.log.
warning: %post(opensearch-2.12.0-1.x86_64) scriptlet failed, exit status 1

Error in POSTIN scriptlet in rpm package opensearch
/usr/lib/tmpfiles.d/opensearch.conf:1: Line references path below legacy directory /var/run/, updating /var/run/opensearch → /run/opensearch; please update the tmpfiles.d/ drop-in file accordingly.

  Verifying        : opensearch-2.12.0-1.x86_64                                                                                                                                                         1/1
Installed products updated.

Installed:
  opensearch-2.12.0-1.x86_64

Complete!
[ec2-userip ~]$
[ec2-userip ~]$
[ec2-userip ~]$ cat /var/log/opensearch/install_demo_configuration.log
### OpenSearch Security Demo Installer
### ** Warning: Do not use on production or public reachable systems **
OpenSearch install type: rpm/deb on Linux 5.14.0-362.13.1.el9_3.x86_64 amd64
OpenSearch config dir: /etc/opensearch/
OpenSearch config file: /etc/opensearch/opensearch.yml
OpenSearch bin dir: /usr/share/opensearch/bin/
OpenSearch plugins dir: /usr/share/opensearch/plugins/
OpenSearch lib dir: /usr/share/opensearch/lib/
Detected OpenSearch Version: 2.12.0
Detected OpenSearch Security Version: 2.12.0.0
Password admin is weak. Please re-try with a stronger password.
[ec2-userip ~]$
[ec2-userip ~]$ sudo yum list | grep opensearch
opensearch.x86_64                                    2.12.0-1                            @@commandline
[ec2-userip ~]$
[ec2-userip ~]$
[ec2-userip ~]$
[ec2-userip ~]$ sudo yum remove opensearch
Updating Subscription Management repositories.
Unable to read consumer identity

This system is not registered with an entitlement server. You can use subscription-manager to register.

Dependencies resolved.
============================================================================================================================================================================================================
 Package                                           Architecture                                  Version                                         Repository                                            Size
============================================================================================================================================================================================================
Removing:
 opensearch                                        x86_64                                        2.12.0-1                                        @@commandline                                        466 M

Transaction Summary
============================================================================================================================================================================================================
Remove  1 Package

Freed space: 466 M
Is this ok [y/N]: y
Running transaction check
Transaction check succeeded.
Running transaction test
Transaction test succeeded.
Running transaction
  Preparing        :                                                                                                                                                                                    1/1
  Running scriptlet: opensearch-2.12.0-1.x86_64                                                                                                                                                         1/1
  Erasing          : opensearch-2.12.0-1.x86_64                                                                                                                                                         1/1
  Verifying        : opensearch-2.12.0-1.x86_64                                                                                                                                                         1/1
Installed products updated.

Removed:
  opensearch-2.12.0-1.x86_64

Complete!
[ec2-userip ~]$ sudo rm -rf /etc/opensearch
[ec2-userip ~]$
[ec2-userip ~]$
[ec2-userip ~]$ sudo env OPENSEARCH_INITIAL_ADMIN_PASSWORD=myStrongPassword123! yum install opensearch-2.12.0-linux-x64.rpm
Updating Subscription Management repositories.
Unable to read consumer identity

This system is not registered with an entitlement server. You can use subscription-manager to register.

Last metadata expiration check: 0:21:41 ago on Wed 10 Jan 2024 05:24:48 AM UTC.
Dependencies resolved.
============================================================================================================================================================================================================
 Package                                           Architecture                                  Version                                          Repository                                           Size
============================================================================================================================================================================================================
Installing:
 opensearch                                        x86_64                                        2.12.0-1                                         @commandline                                        277 M

Transaction Summary
============================================================================================================================================================================================================
Install  1 Package

Total size: 277 M
Installed size: 466 M
Is this ok [y/N]: y
Downloading Packages:
Running transaction check
Transaction check succeeded.
Running transaction test
Transaction test succeeded.
Running transaction
  Preparing        :                                                                                                                                                                                    1/1
  Running scriptlet: opensearch-2.12.0-1.x86_64                                                                                                                                                         1/1
  Installing       : opensearch-2.12.0-1.x86_64                                                                                                                                                         1/1
  Running scriptlet: opensearch-2.12.0-1.x86_64                                                                                                                                                         1/1
/usr/lib/tmpfiles.d/opensearch.conf:1: Line references path below legacy directory /var/run/, updating /var/run/opensearch → /run/opensearch; please update the tmpfiles.d/ drop-in file accordingly.
### NOT starting on installation, please execute the following statements to configure opensearch service to start automatically using systemd
 sudo systemctl daemon-reload
 sudo systemctl enable opensearch.service
### You can start opensearch service by executing
 sudo systemctl start opensearch.service
### Create opensearch demo certificates in /etc/opensearch/
 See demo certs creation log in /var/log/opensearch/install_demo_configuration.log
### Upcoming breaking change in packaging
 In a future release of OpenSearch, we plan to change the permissions associated with access to installed files
 If you are configuring tools that require read access to the OpenSearch configuration files, we recommend you add the user that runs these tools to the 'opensearch' group
 For more information, see https://github.com/opensearch-project/opensearch-build/pull/4043

/usr/lib/tmpfiles.d/opensearch.conf:1: Line references path below legacy directory /var/run/, updating /var/run/opensearch → /run/opensearch; please update the tmpfiles.d/ drop-in file accordingly.

  Verifying        : opensearch-2.12.0-1.x86_64                                                                                                                                                         1/1
Installed products updated.

Installed:
  opensearch-2.12.0-1.x86_64

Complete!
[ec2-userip ~]$ sudo systemctl enable opensearch
Synchronizing state of opensearch.service with SysV service script with /usr/lib/systemd/systemd-sysv-install.
Executing: /usr/lib/systemd/systemd-sysv-install enable opensearch
[ec2-userip ~]$
[ec2-userip ~]$
[ec2-userip ~]$
[ec2-userip ~]$ sudo systemctl start opensearch
[ec2-userip ~]$
[ec2-userip ~]$
[ec2-userip ~]$ cat /var/log/opensearch/install_demo_configuration.log
### OpenSearch Security Demo Installer
### ** Warning: Do not use on production or public reachable systems **
OpenSearch install type: rpm/deb on Linux 5.14.0-362.13.1.el9_3.x86_64 amd64
OpenSearch config dir: /etc/opensearch/
OpenSearch config file: /etc/opensearch/opensearch.yml
OpenSearch bin dir: /usr/share/opensearch/bin/
OpenSearch plugins dir: /usr/share/opensearch/plugins/
OpenSearch lib dir: /usr/share/opensearch/lib/
Detected OpenSearch Version: 2.12.0
Detected OpenSearch Security Version: 2.12.0.0
Admin password set successfully.
### Success
### Execute this script now on all your nodes and then start all nodes
### OpenSearch Security will be automatically initialized.
### If you like to change the runtime configuration
### change the files in ../../../config/opensearch-security and execute:
sudo "/usr/share/opensearch/plugins/opensearch-security/tools/securityadmin.sh" -cd "/etc/opensearch/opensearch-security" -icl -key "/etc/opensearch/kirk-key.pem" -cert "/etc/opensearch/kirk.pem" -cacert "/etc/opensearch/root-ca.pem" -nhnv
### or run ./securityadmin_demo.sh
### To use the Security Plugin ConfigurationGUI
### To access your secured cluster open https://<hostname>:<HTTP port> and log in with admin/<your-custom-admin-password>.
### (Ignore the SSL certificate warning because we installed self-signed demo certificates)
[ec2-userip ~]$
[ec2-userip ~]$
[ec2-userip ~]$
[ec2-userip ~]$ curl https://localhost:9200 -ku admin:admin
[ec2-userip ~]$
[ec2-userip ~]$
[ec2-userip ~]$ curl https://localhost:9200 -ku admin:myStrongPassword123!
{
  "name" : "smoketestnode",
  "cluster_name" : "opensearch",
  "cluster_uuid" : "3WxosaT_S0aW-FpZjpSIbw",
  "version" : {
    "distribution" : "opensearch",
    "number" : "2.12.0",
    "build_type" : "rpm",
    "build_hash" : "334636f62662a886a51edd39e7e81f8e80ab9e14",
    "build_date" : "2024-01-10T05:09:33.291113994Z",
    "build_snapshot" : false,
    "lucene_version" : "9.8.0",
    "minimum_wire_compatibility_version" : "7.10.0",
    "minimum_index_compatibility_version" : "7.0.0"
  },
  "tagline" : "The OpenSearch Project: https://opensearch.org/"
}
[ec2-userip ~]$

@DarshitChanpura
Copy link
Member Author

DarshitChanpura commented Jan 10, 2024

@peterzhuamazon @prudhvigodithi @rishabh6788 Could you please add your reviews to this PR?

@DarshitChanpura
Copy link
Member Author

Another issue just raised by @DarshitChanpura is that preinstall might not correctly interpret if an old install with admin pass already set existed, thus will skip the password check here.

Handled in the latest commit.

@DarshitChanpura
Copy link
Member Author

Debian successful upgrade logs:
ubuntu@ip:~/test$ sudo dpkg -i opensearch-2.12.0-linux-x64.deb
(Reading database ... 111709 files and directories currently installed.)
Preparing to unpack opensearch-2.12.0-linux-x64.deb ...
Running OpenSearch Pre-Installation Script
Unpacking opensearch (2.12.0) over (2.11.1) ...
dpkg: warning: unable to delete old directory '/etc/opensearch/opensearch-reports-scheduler': Directory not empty
dpkg: warning: unable to delete old directory '/etc/opensearch/opensearch-performance-analyzer': Directory not empty
dpkg: warning: unable to delete old directory '/etc/opensearch/opensearch-observability': Directory not empty
dpkg: warning: unable to delete old directory '/etc/opensearch/opensearch-notifications-core': Directory not empty
dpkg: warning: unable to delete old directory '/etc/opensearch/opensearch-notifications': Directory not empty
Setting up opensearch (2.12.0) ...

Configuration file '/etc/opensearch/jvm.options'
 ==> Modified (by you or by a script) since installation.
 ==> Package distributor has shipped an updated version.
   What would you like to do about it ?  Your options are:
    Y or I  : install the package maintainer's version
    N or O  : keep your currently-installed version
      D     : show the differences between the versions
      Z     : start a shell to examine the situation
 The default action is to keep your current version.
*** jvm.options (Y/I/N/O/D/Z) [default=N] ? N
Installing new version of config file /etc/opensearch/log4j2.properties ...
Installing new version of config file /etc/opensearch/opensearch-security/roles.yml ...

Configuration file '/etc/opensearch/opensearch.yml'
 ==> Modified (by you or by a script) since installation.
 ==> Package distributor has shipped an updated version.
   What would you like to do about it ?  Your options are:
    Y or I  : install the package maintainer's version
    N or O  : keep your currently-installed version
      D     : show the differences between the versions
      Z     : start a shell to examine the situation
 The default action is to keep your current version.
*** opensearch.yml (Y/I/N/O/D/Z) [default=N] ? N
Running OpenSearch Post-Installation Script
### NOT starting on installation, please execute the following statements to configure opensearch service to start automatically using systemd
 sudo systemctl daemon-reload
 sudo systemctl enable opensearch.service
### You can start opensearch service by executing
 sudo systemctl start opensearch.service
### Create opensearch demo certificates in /etc/opensearch/
 See demo certs creation log in /var/log/opensearch/install_demo_configuration.log
### Upcoming breaking change in packaging
 In a future release of OpenSearch, we plan to change the permissions associated with access to installed files
 If you are configuring tools that require read access to the OpenSearch configuration files, we recommend you add the user that runs these tools to the 'opensearch' group
 For more information, see https://github.com/opensearch-project/opensearch-build/pull/4043
Processing triggers for libc-bin (2.31-0ubuntu9.14) ...
Processing triggers for systemd (245.4-4ubuntu3.22) ...
ubuntu@ip:~/test$ sudo systemctl enable opensearch-performance-analyzer.service
Created symlink /etc/systemd/system/multi-user.target.wants/opensearch-performance-analyzer.service → /lib/systemd/system/opensearch-performance-analyzer.service.
ubuntu@ip:~/test$ sudo systemctl enable opensearch
Synchronizing state of opensearch.service with SysV service script with /lib/systemd/systemd-sysv-install.
Executing: /lib/systemd/systemd-sysv-install enable opensearch
ubuntu@ip:~/test$ sudo systemctl start opensearch
ubuntu@ip:~/test$
ubuntu@ip:~/test$
ubuntu@ip:~/test$ curl https://localhost:9200 -ku admin:myStrongPassword123!
ubuntu@ip:~/test$ curl https://localhost:9200 -ku admin:admin
{
  "name" : "ip",
  "cluster_name" : "opensearch",
  "cluster_uuid" : "P6tHXsLgQwCuWE4seDRLNQ",
  "version" : {
    "distribution" : "opensearch",
    "number" : "2.12.0",
    "build_type" : "deb",
    "build_hash" : "de26624e75ca8e81bc0a1d4416c70b06d9dac7f4",
    "build_date" : "2024-01-10T19:31:34.117062540Z",
    "build_snapshot" : false,
    "lucene_version" : "9.9.1",
    "minimum_wire_compatibility_version" : "7.10.0",
    "minimum_index_compatibility_version" : "7.0.0"
  },
  "tagline" : "The OpenSearch Project: https://opensearch.org/"
}
ubuntu@ip:~/test$

Copy link
Contributor

@smortex smortex left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I added a few in-line comments. Most of them apply both to the deb and the rpm scripts, but I have not duplicated them.

scripts/pkg/build_templates/opensearch/deb/debian/preinst Outdated Show resolved Hide resolved
scripts/pkg/build_templates/opensearch/deb/debian/preinst Outdated Show resolved Hide resolved
scripts/pkg/build_templates/opensearch/deb/debian/preinst Outdated Show resolved Hide resolved
@DarshitChanpura
Copy link
Member Author

Logs after the change:

Debian:
ubuntu@ip:~/test$ sudo dpkg -i opensearch-2.12.0-linux-x64.deb
Selecting previously unselected package opensearch.
(Reading database ... 110309 files and directories currently installed.)
Preparing to unpack opensearch-2.12.0-linux-x64.deb ...
Running OpenSearch Pre-Installation Script
Unpacking opensearch (2.12.0) ...
Setting up opensearch (2.12.0) ...
Running OpenSearch Post-Installation Script
ERROR: Something went wrong during demo configuration installation. Please see the logs in /var/log/opensearch/install_demo_configuration.log
dpkg: error processing package opensearch (--install):
 installed opensearch package post-installation script subprocess returned error exit status 1
Processing triggers for systemd (245.4-4ubuntu3.22) ...
Errors were encountered while processing:
 opensearch
ubuntu@ip:~/test$ sudo dpkg -i opensearch-2.11.1-linux-x64.deb
dpkg: warning: downgrading opensearch from 2.12.0 to 2.11.1
(Reading database ... 111176 files and directories currently installed.)
Preparing to unpack opensearch-2.11.1-linux-x64.deb ...
Running OpenSearch Pre-Removal Script
Running OpenSearch Pre-Installation Script
Unpacking opensearch (2.11.1) over (2.12.0) ...
Setting up opensearch (2.11.1) ...
Installing new version of config file /etc/opensearch/jvm.options ...
Installing new version of config file /etc/opensearch/log4j2.properties ...
Installing new version of config file /etc/opensearch/opensearch-security/roles.yml ...
Installing new version of config file /etc/opensearch/opensearch.yml ...
Running OpenSearch Post-Installation Script
### NOT starting on installation, please execute the following statements to configure opensearch service to start automatically using systemd
 sudo systemctl daemon-reload
 sudo systemctl enable opensearch.service
### You can start opensearch service by executing
 sudo systemctl start opensearch.service
### Create opensearch demo certificates in /etc/opensearch/
 See demo certs creation log in /var/log/opensearch/install_demo_configuration.log
### Upcoming breaking change in packaging
 In a future release of OpenSearch, we plan to change the permissions associated with access to installed files
 If you are configuring tools that require read access to the OpenSearch configuration files, we recommend you add the user that runs these tools to the 'opensearch' group
 For more information, see https://github.com/opensearch-project/opensearch-build/pull/4043
Processing triggers for libc-bin (2.31-0ubuntu9.14) ...
Processing triggers for systemd (245.4-4ubuntu3.22) ...
ubuntu@ip:~/test$
ubuntu@ip:~/test$
ubuntu@ip:~/test$ sudo dpkg -i opensearch-2.12.0-linux-x64.deb
(Reading database ... 111709 files and directories currently installed.)
Preparing to unpack opensearch-2.12.0-linux-x64.deb ...
Running OpenSearch Pre-Removal Script
Running OpenSearch Pre-Installation Script
Unpacking opensearch (2.12.0) over (2.11.1) ...
dpkg: warning: unable to delete old directory '/etc/opensearch/opensearch-reports-scheduler': Directory not empty
dpkg: warning: unable to delete old directory '/etc/opensearch/opensearch-performance-analyzer': Directory not empty
dpkg: warning: unable to delete old directory '/etc/opensearch/opensearch-observability': Directory not empty
dpkg: warning: unable to delete old directory '/etc/opensearch/opensearch-notifications-core': Directory not empty
dpkg: warning: unable to delete old directory '/etc/opensearch/opensearch-notifications': Directory not empty
Setting up opensearch (2.12.0) ...

Configuration file '/etc/opensearch/jvm.options'
 ==> Modified (by you or by a script) since installation.
 ==> Package distributor has shipped an updated version.
   What would you like to do about it ?  Your options are:
    Y or I  : install the package maintainer's version
    N or O  : keep your currently-installed version
      D     : show the differences between the versions
      Z     : start a shell to examine the situation
 The default action is to keep your current version.
*** jvm.options (Y/I/N/O/D/Z) [default=N] ? N
Installing new version of config file /etc/opensearch/log4j2.properties ...
Installing new version of config file /etc/opensearch/opensearch-security/roles.yml ...

Configuration file '/etc/opensearch/opensearch.yml'
 ==> Modified (by you or by a script) since installation.
 ==> Package distributor has shipped an updated version.
   What would you like to do about it ?  Your options are:
    Y or I  : install the package maintainer's version
    N or O  : keep your currently-installed version
      D     : show the differences between the versions
      Z     : start a shell to examine the situation
 The default action is to keep your current version.
*** opensearch.yml (Y/I/N/O/D/Z) [default=N] ? N
Running OpenSearch Post-Installation Script
### NOT starting on installation, please execute the following statements to configure opensearch service to start automatically using systemd
 sudo systemctl daemon-reload
 sudo systemctl enable opensearch.service
### You can start opensearch service by executing
 sudo systemctl start opensearch.service
### Create opensearch demo certificates in /etc/opensearch/
 See demo certs creation log in /var/log/opensearch/install_demo_configuration.log
### Upcoming breaking change in packaging
 In a future release of OpenSearch, we plan to change the permissions associated with access to installed files
 If you are configuring tools that require read access to the OpenSearch configuration files, we recommend you add the user that runs these tools to the 'opensearch' group
 For more information, see https://github.com/opensearch-project/opensearch-build/pull/4043
Processing triggers for libc-bin (2.31-0ubuntu9.14) ...
Processing triggers for systemd (245.4-4ubuntu3.22) ...
ubuntu@ip:~/test$
ubuntu@ip:~/test$
ubuntu@ip:~/test$ sudo systemctl enable opensearch
Synchronizing state of opensearch.service with SysV service script with /lib/systemd/systemd-sysv-install.
Executing: /lib/systemd/systemd-sysv-install enable opensearch
ubuntu@ip:~/test$ sudo systemctl start opensearch
ubuntu@ip:~/test$
ubuntu@ip:~/test$
ubuntu@ip:~/test$ curl https://localhost:9200 -ku admin:myStrongPassword123!
ubuntu@ip:~/test$
ubuntu@ip:~/test$
ubuntu@ip:~/test$ curl https://localhost:9200 -ku admin:admin
{
  "name" : "ip",
  "cluster_name" : "opensearch",
  "cluster_uuid" : "tFzkQZ8JSryfqNyXDKkWwA",
  "version" : {
    "distribution" : "opensearch",
    "number" : "2.12.0",
    "build_type" : "deb",
    "build_hash" : "84b3ebe9b9d0f903fce84a9cbafba63650353c0b",
    "build_date" : "2024-01-11T01:39:14.529344933Z",
    "build_snapshot" : false,
    "lucene_version" : "9.9.1",
    "minimum_wire_compatibility_version" : "7.10.0",
    "minimum_index_compatibility_version" : "7.0.0"
  },
  "tagline" : "The OpenSearch Project: https://opensearch.org/"
}
ubuntu@ip:~/test$
RPM:
[ec2-user@ip ~]$ sudo yum install opensearch-2.12.0-linux-x64.rpm
Updating Subscription Management repositories.
Unable to read consumer identity

This system is not registered with an entitlement server. You can use subscription-manager to register.

Last metadata expiration check: 3:25:02 ago on Thu 11 Jan 2024 12:34:46 AM UTC.
Dependencies resolved.
==============================================================================================================================================================
 Package                               Architecture                      Version                                Repository                               Size
==============================================================================================================================================================
Installing:
 opensearch                            x86_64                            2.12.0-1                               @commandline                            278 M

Transaction Summary
==============================================================================================================================================================
Install  1 Package

Total size: 278 M
Installed size: 466 M
Is this ok [y/N]: y
Downloading Packages:
Running transaction check
Transaction check succeeded.
Running transaction test
Transaction test succeeded.
Running transaction
  Preparing        :                                                                                                                                      1/1
  Running scriptlet: opensearch-2.12.0-1.x86_64                                                                                                           1/1
package opensearch is not installed
Updating Subscription Management repositories.
Unable to read consumer identity

This system is not registered with an entitlement server. You can use subscription-manager to register.

Error: No matching Packages to list
ERROR: Opensearch 2.12 and later requires the env variable OPENSEARCH_INITIAL_ADMIN_PASSWORD to be defined to setup the opensearch-security demo configuration
error: %prein(opensearch-2.12.0-1.x86_64) scriptlet failed, exit status 1

Error in PREIN scriptlet in rpm package opensearch
  Verifying        : opensearch-2.12.0-1.x86_64                                                                                                           1/1
Installed products updated.

Failed:
  opensearch-2.12.0-1.x86_64

Error: Transaction failed
[ec2-user@ip ~]$
[ec2-user@ip ~]$
[ec2-user@ip ~]$ sudo yum install opensearch-2.11.1-linux-x64.rpm
Updating Subscription Management repositories.
Unable to read consumer identity

This system is not registered with an entitlement server. You can use subscription-manager to register.

Last metadata expiration check: 3:25:36 ago on Thu 11 Jan 2024 12:34:46 AM UTC.
Dependencies resolved.
==============================================================================================================================================================
 Package                               Architecture                      Version                                Repository                               Size
==============================================================================================================================================================
Installing:
 opensearch                            x86_64                            2.11.1-1                               @commandline                            748 M

Transaction Summary
==============================================================================================================================================================
Install  1 Package

Total size: 748 M
Installed size: 1.0 G
Is this ok [y/N]: y
Downloading Packages:
Running transaction check
Transaction check succeeded.
Running transaction test
Transaction test succeeded.
Running transaction
  Preparing        :                                                                                                                                      1/1
  Running scriptlet: opensearch-2.11.1-1.x86_64                                                                                                           1/1
  Installing       : opensearch-2.11.1-1.x86_64                                                                                                           1/1
  Running scriptlet: opensearch-2.11.1-1.x86_64                                                                                                           1/1
/usr/lib/tmpfiles.d/opensearch.conf:1: Line references path below legacy directory /var/run/, updating /var/run/opensearch → /run/opensearch; please update the tmpfiles.d/ drop-in file accordingly.
### NOT starting on installation, please execute the following statements to configure opensearch service to start automatically using systemd
 sudo systemctl daemon-reload
 sudo systemctl enable opensearch.service
### You can start opensearch service by executing
 sudo systemctl start opensearch.service
### Create opensearch demo certificates in /etc/opensearch/
 See demo certs creation log in /var/log/opensearch/install_demo_configuration.log
### Upcoming breaking change in packaging
 In a future release of OpenSearch, we plan to change the permissions associated with access to installed files
 If you are configuring tools that require read access to the OpenSearch configuration files, we recommend you add the user that runs these tools to the 'opensearch' group
 For more information, see https://github.com/opensearch-project/opensearch-build/pull/4043

/usr/lib/tmpfiles.d/opensearch-dashboards.conf:1: Line references path below legacy directory /var/run/, updating /var/run/opensearch-dashboards → /run/opensearch-dashboards; please update the tmpfiles.d/ drop-in file accordingly.
/usr/lib/tmpfiles.d/opensearch.conf:1: Line references path below legacy directory /var/run/, updating /var/run/opensearch → /run/opensearch; please update the tmpfiles.d/ drop-in file accordingly.

  Verifying        : opensearch-2.11.1-1.x86_64                                                                                                           1/1
Installed products updated.

Installed:
  opensearch-2.11.1-1.x86_64

Complete!
[ec2-user@ip ~]$
[ec2-user@ip ~]$
[ec2-user@ip ~]$ sudo yum install opensearch-2.11.1-linux-x64.rpm
Updating Subscription Management repositories.
Unable to read consumer identity

This system is not registered with an entitlement server. You can use subscription-manager to register.

Last metadata expiration check: 3:25:36 ago on Thu 11 Jan 2024 12:34:46 AM UTC.
Dependencies resolved.
==============================================================================================================================================================
 Package                               Architecture                      Version                                Repository                               Size
==============================================================================================================================================================
Installing:
 opensearch                            x86_64                            2.11.1-1                               @commandline                            748 M

Transaction Summary
==============================================================================================================================================================
Install  1 Package

Total size: 748 M
Installed size: 1.0 G
Is this ok [y/N]: y
Downloading Packages:
Running transaction check
Transaction check succeeded.
Running transaction test
Transaction test succeeded.
Running transaction
  Preparing        :                                                                                                                                      1/1
  Running scriptlet: opensearch-2.11.1-1.x86_64                                                                                                           1/1
  Installing       : opensearch-2.11.1-1.x86_64                                                                                                           1/1
  Running scriptlet: opensearch-2.11.1-1.x86_64                                                                                                           1/1
/usr/lib/tmpfiles.d/opensearch.conf:1: Line references path below legacy directory /var/run/, updating /var/run/opensearch → /run/opensearch; please update the tmpfiles.d/ drop-in file accordingly.
### NOT starting on installation, please execute the following statements to configure opensearch service to start automatically using systemd
 sudo systemctl daemon-reload
 sudo systemctl enable opensearch.service
### You can start opensearch service by executing
 sudo systemctl start opensearch.service
### Create opensearch demo certificates in /etc/opensearch/
 See demo certs creation log in /var/log/opensearch/install_demo_configuration.log
### Upcoming breaking change in packaging
 In a future release of OpenSearch, we plan to change the permissions associated with access to installed files
 If you are configuring tools that require read access to the OpenSearch configuration files, we recommend you add the user that runs these tools to the 'opensearch' group
 For more information, see https://github.com/opensearch-project/opensearch-build/pull/4043

/usr/lib/tmpfiles.d/opensearch-dashboards.conf:1: Line references path below legacy directory /var/run/, updating /var/run/opensearch-dashboards → /run/opensearch-dashboards; please update the tmpfiles.d/ drop-in file accordingly.
/usr/lib/tmpfiles.d/opensearch.conf:1: Line references path below legacy directory /var/run/, updating /var/run/opensearch → /run/opensearch; please update the tmpfiles.d/ drop-in file accordingly.

  Verifying        : opensearch-2.11.1-1.x86_64                                                                                                           1/1
Installed products updated.

Installed:
  opensearch-2.11.1-1.x86_64

Complete!
[ec2-user@ip ~]$
[ec2-user@ip ~]$
[ec2-user@ip ~]$ sudo yum install opensearch-2.12.0-linux-x64.rpm
Updating Subscription Management repositories.
Unable to read consumer identity

This system is not registered with an entitlement server. You can use subscription-manager to register.

Last metadata expiration check: 3:27:19 ago on Thu 11 Jan 2024 12:34:46 AM UTC.
Dependencies resolved.
==============================================================================================================================================================
 Package                               Architecture                      Version                                Repository                               Size
==============================================================================================================================================================
Upgrading:
 opensearch                            x86_64                            2.12.0-1                               @commandline                            278 M

Transaction Summary
==============================================================================================================================================================
Upgrade  1 Package

Total size: 278 M
Is this ok [y/N]: y
Downloading Packages:
Running transaction check
Transaction check succeeded.
Running transaction test
Transaction test succeeded.
Running transaction
  Preparing        :                                                                                                                                      1/1
  Running scriptlet: opensearch-2.12.0-1.x86_64                                                                                                           1/2
opensearch-2.11.1-1.x86_64

  Upgrading        : opensearch-2.12.0-1.x86_64                                                                                                           1/2
warning: /etc/opensearch/jvm.options created as /etc/opensearch/jvm.options.rpmnew
warning: /etc/opensearch/opensearch.yml created as /etc/opensearch/opensearch.yml.rpmnew

  Running scriptlet: opensearch-2.12.0-1.x86_64                                                                                                           1/2
/usr/lib/tmpfiles.d/opensearch.conf:1: Line references path below legacy directory /var/run/, updating /var/run/opensearch → /run/opensearch; please update the tmpfiles.d/ drop-in file accordingly.
### NOT starting on installation, please execute the following statements to configure opensearch service to start automatically using systemd
 sudo systemctl daemon-reload
 sudo systemctl enable opensearch.service
### You can start opensearch service by executing
 sudo systemctl start opensearch.service
### Create opensearch demo certificates in /etc/opensearch/
 See demo certs creation log in /var/log/opensearch/install_demo_configuration.log
### Upcoming breaking change in packaging
 In a future release of OpenSearch, we plan to change the permissions associated with access to installed files
 If you are configuring tools that require read access to the OpenSearch configuration files, we recommend you add the user that runs these tools to the 'opensearch' group
 For more information, see https://github.com/opensearch-project/opensearch-build/pull/4043

  Running scriptlet: opensearch-2.11.1-1.x86_64                                                                                                           2/2
  Cleanup          : opensearch-2.11.1-1.x86_64                                                                                                           2/2
  Running scriptlet: opensearch-2.11.1-1.x86_64                                                                                                           2/2
/usr/lib/tmpfiles.d/opensearch-dashboards.conf:1: Line references path below legacy directory /var/run/, updating /var/run/opensearch-dashboards → /run/opensearch-dashboards; please update the tmpfiles.d/ drop-in file accordingly.
/usr/lib/tmpfiles.d/opensearch.conf:1: Line references path below legacy directory /var/run/, updating /var/run/opensearch → /run/opensearch; please update the tmpfiles.d/ drop-in file accordingly.

  Verifying        : opensearch-2.12.0-1.x86_64                                                                                                           1/2
  Verifying        : opensearch-2.11.1-1.x86_64                                                                                                           2/2
Installed products updated.

Upgraded:
  opensearch-2.12.0-1.x86_64

Complete!
[ec2-user@ip ~]$
[ec2-user@ip ~]$
[ec2-user@ip ~]$
[ec2-user@ip ~]$ sudo systemctl enable opensearch
Synchronizing state of opensearch.service with SysV service script with /usr/lib/systemd/systemd-sysv-install.
Executing: /usr/lib/systemd/systemd-sysv-install enable opensearch
[ec2-user@ip ~]$ sudo systemctl start opensearch
[ec2-user@ip ~]$
[ec2-user@ip ~]$
[ec2-user@ip ~]$
[ec2-user@ip ~]$ curl https://localhost:9200 -ku admin:myStrongPassword123!
[ec2-user@ip ~]$
[ec2-user@ip ~]$
[ec2-user@ip ~]$ curl https://localhost:9200 -ku admin:admin
{
  "name" : "ip.ec2.internal",
  "cluster_name" : "opensearch",
  "cluster_uuid" : "qLvF1naOTuiUGef_-IfGDg",
  "version" : {
    "distribution" : "opensearch",
    "number" : "2.12.0",
    "build_type" : "rpm",
    "build_hash" : "bd5b5ee0f124605a14ea5b69073a7400d7d33ca6",
    "build_date" : "2024-01-11T01:20:35.709440292Z",
    "build_snapshot" : false,
    "lucene_version" : "9.9.1",
    "minimum_wire_compatibility_version" : "7.10.0",
    "minimum_index_compatibility_version" : "7.0.0"
  },
  "tagline" : "The OpenSearch Project: https://opensearch.org/"
}
[ec2-user@ip ~]$

@DarshitChanpura
Copy link
Member Author

@peterzhuamazon Would you please re-review this?

@peterzhuamazon
Copy link
Member

Hi @DarshitChanpura @smortex,

Given the current implementation, we would expect user to install pkg with this method:

sudo env OPENSEARCH_INITIAL_ADMIN_PASSWORD=<somepassword> yum install opensearch

I feel like this is a hard breaking change for the installation.
Like @smortex mentioned in the #3916, we need to ensure user is well informed with the steps if we bound to make this change.

Right now:

  1. Preinst will only check if there is a password set if >= 2.12.0
  2. Preinst will not check if the password is weak
  3. If password is weak, install will still happen, then postinst will cache this issue.
  4. Postinst fail and force user to uninstall then reinstall the pkg just to check if the password is strong enough before next step to generate certs.
  5. Current implementation only check if a previous version is installed, but did not check if the previous version correctly setup password and certs, and rely on the post install to check.
  6. More......

Please let me know your thoughts on this.

cc: @bbarani @prudhvigodithi

Thanks.

@DarshitChanpura
Copy link
Member Author

@peterzhuamazon @prudhvigodithi @rishabh6788 Could I get some more reviews?

Signed-off-by: Darshit Chanpura <[email protected]>
@DarshitChanpura
Copy link
Member Author

@peterzhuamazon @prudhvigodithi @rishabh6788 Could I get some more reviews?

Lint checker fails due to timeout and is unrelated to code changes introduced in this PR

@peterzhuamazon
Copy link
Member

Awaiting approval @prudhvigodithi.

Thanks @DarshitChanpura for the sync up and improvement on the PR.

@prudhvigodithi
Copy link
Collaborator

LGTM, @DarshitChanpura if a user does not have a strong password it would still proceed with the installation and fail at the post install, we should think through how we can address this in pre install itself. WDYT @peterzhuamazon ?

@DarshitChanpura
Copy link
Member Author

@prudhvigodithi if a password is not passed at al, the script will fail at pre-install. If a weak password is passed, the script will fail at post-install, and user may have to uninstall and reinstall.
We initially planned to fail everything at pre-install, but since the weak password validation is handled by the demo config install script by a 3rd party library we will have to wait until post install (which is where the demo config is called). We were not able to replicate the weak password validation logic provided by java library into the pre-install bash script and hence the decision was to let it fail at post-install. To help the users, we modified the scripts to point the users to install_demo_configuration.log, which is where the execution output/error from the demo install is stored.
Another option, was to move the demo script to pre-install phase, but that is not possible since the script requires a bunch of folders to be setup which are done in post-install phase only.

Hope this answers your question.

@prudhvigodithi prudhvigodithi merged commit 5803531 into opensearch-project:main Jan 19, 2024
13 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants