[collect] Fails for the local node when using --nopasswd-sudo

[TrevorBenson]

Problem

Executing sos collect results in failures on the local node executing sos collect via sudo.

sudo sos collect --nopasswd-sudo --ssh-key /etc/project/id_edd2519 --ssh-user project-user --nodes 172.31.100.121,172.31.100.85 -o block

Beginning collection of sosreports from 2 nodes, collecting a maximum of 4 concurrently

system1  : Generating sos report...
system2        : Generating sos report...
system2        : Retrieving sos report...
system2        : Successfully collected sos report
system1  : Error running sos report: sudo attempt failed
[system1:sosreport] Error during sos execution: sudo attempt failed

When project-user executes sudo sos report directly on either system1 or system2 a valid sos archive is created in /var/tmp.

• OS: Enterprise Linux 8 (Rocky Linux 8.10 in this case)
• Version of sos: sos-4.7.2-2.el8_10.noarch

Environment Details

The environment has 2 nodes hardened with no direct shell access to root. Executing sos report, or sos collect requires using sudo. The project-user account exists on both nodes and has sudoers permissions to run sos.

The public ssh key /etc/project/id_edd2519.pub is part of /home/project-user/.ssh/authorized_keys on both system1 and system2.

Sudoers Configuration

$ sudo grep SOS /etc/sudoers.d/project
Cmnd_Alias SOS = /usr/sbin/sos, /usr/sbin/sosreport, /usr/bin/chown * /var/tmp/sosreport-*
project-user ALL=(root) EXEC:/home/project-user/project-installer, SOS

Additional Details

Checking the sos_logs/sos.log it mentions something wrong with the account password even when using --nopasswd-sudo and having confirmed that sudo works perfectly fine without a password request to execute sudo sos report on either system:

2024-10-31 20:56:52,518 DEBUG: [sos_collector:__init__] Executing /usr/sbin/sos collect --nopasswd-sudo --debug --verbose --ssh-key /etc/project/pki/salt-bootstrap --ssh-user project-user --nodes 172.31.100.121,172.31.100.85 -o block
2024-10-31 20:56:52,518 DEBUG: [sos_collector:__init__] Found cluster profiles: dict_keys(['ceph', 'jbon', 'juju', 'kubernetes', 'ocp', 'rhosp', 'ovirt', 'rhhi_virt', 'rhv', 'pacemaker', 'saltstack', 'satellite'])
2024-10-31 20:56:53,415 DEBUG: [sos_collector:configure_sos_cmd] Initial sos cmd set to sosreport --batch --chroot auto
2024-10-31 20:56:53,415 DEBUG: [sos_collector:prep] password not specified, assuming SSH keys
2024-10-31 20:56:53,764 DEBUG: [localhost:run_command] Running command hostname
2024-10-31 20:56:53,871 INFO: [system1:_get_hostname] Hostname set to system1
2024-10-31 20:56:53,871 INFO: [localhost:determine_host_policy] using local policy Rocky Linux
2024-10-31 20:56:53,872 INFO: [system1:_load_sos_info] sos version is 4.7.2-2
2024-10-31 20:56:53,873 DEBUG: [system1:run_command] Running command sudo -S sos report -l
2024-10-31 20:57:53,999 DEBUG: [system1:run_command] Running command sudo -S sos report --list-presets
2024-10-31 20:58:54,125 DEBUG: [ocp] oc base command set to oc
2024-10-31 20:58:54,126 DEBUG: [system1:run_command] Running command oc whoami
2024-10-31 20:58:54,127 DEBUG: [system1:_run_command_with_pexpect] The command was not found or was not executable: oc.
2024-10-31 20:58:54,128 DEBUG: [system1:run_command] Running command sudo -S stat /var/lib/mistral/overcloud/tripleo-ansible-inventory.yaml
2024-10-31 20:59:54,253 INFO: Cluster type could not be determined, but --nodes is provided. Attempting to continue using JBON cluster type and the node list
2024-10-31 20:59:54,254 DEBUG: [sos_collector:get_nodes_from_cluster] Node list: []
2024-10-31 20:59:54,254 DEBUG: [sos_collector:get_nodes] Force adding 172.31.100.121 to node list
2024-10-31 20:59:54,254 DEBUG: [sos_collector:get_nodes] Force adding 172.31.100.85 to node list
2024-10-31 20:59:54,255 DEBUG: [sos_collector:reduce_node_list] Node list reduced to ['172.31.100.121']
2024-10-31 20:59:54,256 INFO: [archive:TarFileArchive] initialised empty FileCacheArchive at '/var/tmp/sos.8rtucemv/sos-collector-2024-10-31-wodmb'
2024-10-31 20:59:54,264 INFO: [172.31.100.121:_connect] Opening SSH session to create control socket
2024-10-31 20:59:54,461 DEBUG: [172.31.100.121:_connect] Successfully created control socket at /var/tmp/sos.8rtucemv/.sos-collector-172.31.100.121
2024-10-31 20:59:54,562 DEBUG: [172.31.100.121:run_command] Running command hostname
2024-10-31 20:59:54,690 INFO: [system2:_get_hostname] Hostname set to system2
2024-10-31 20:59:54,690 INFO: [172.31.100.121:read_file] Reading file /etc/os-release
2024-10-31 20:59:54,691 DEBUG: [system2:read_file] Reading file /etc/os-release
2024-10-31 20:59:54,691 DEBUG: [system2:run_command] Running command cat /etc/os-release
2024-10-31 20:59:54,941 DEBUG: [system2:run_command] Running command rpm -qa --queryformat "%{NAME}|%{VERSION}|%{RELEASE}\n"
2024-10-31 20:59:55,724 DEBUG: [system2:run_command] Running command flatpak list --columns=name,version,branch
2024-10-31 20:59:55,851 INFO: [172.31.100.121:determine_host_policy] loaded policy Rocky Linux for host
2024-10-31 20:59:55,852 INFO: [system2:_load_sos_info] sos version is 4.7.2-2
2024-10-31 20:59:55,853 DEBUG: [system2:run_command] Running command sudo -S sos report -l
2024-10-31 20:59:58,218 DEBUG: [system2:_regex_sos_help] Error parsing sos help: list index out of range
2024-10-31 20:59:58,219 DEBUG: [system2:_regex_sos_help] Error parsing sos help: list index out of range
2024-10-31 20:59:58,220 DEBUG: [system2:_regex_sos_help] Error parsing sos help: list index out of range
2024-10-31 20:59:58,221 DEBUG: [system2:_regex_sos_help] Error parsing sos help: list index out of range
2024-10-31 20:59:58,222 DEBUG: [system2:_regex_sos_help] Error parsing sos help: list index out of range
2024-10-31 20:59:58,223 DEBUG: [system2:_regex_sos_help] Error parsing sos help: list index out of range
2024-10-31 20:59:58,224 DEBUG: [system2:_regex_sos_help] Error parsing sos help: list index out of range
2024-10-31 20:59:58,224 DEBUG: [system2:_regex_sos_help] Error parsing sos help: list index out of range
2024-10-31 20:59:58,225 DEBUG: [system2:_regex_sos_help] Error parsing sos help: list index out of range
2024-10-31 20:59:58,226 DEBUG: [system2:run_command] Running command sudo -S sos report --list-presets
2024-10-31 21:00:00,655 DEBUG: [system1:finalize_sos_cmd] Requested plugins ['block'] were requested to be enabled but do not exist
2024-10-31 21:00:00,656 INFO: [system2:finalize_sos_cmd] Final sos command set to /usr/sbin/sos report --batch  --chroot auto --only-plugins=block
2024-10-31 21:00:00,656 INFO: [system1:finalize_sos_cmd] Final sos command set to /usr/sbin/sos report --batch  --chroot auto
2024-10-31 21:00:00,658 DEBUG: [system2:run_command] Running command sudo -S /usr/sbin/sos report --batch  --chroot auto --only-plugins=block
2024-10-31 21:00:00,659 DEBUG: [system1:run_command] Running command sudo -S /usr/sbin/sos report --batch  --chroot auto
2024-10-31 21:00:00,660 DEBUG: [system1:run_command] Shell requested, command is now /bin/bash -c 'sudo -S /usr/sbin/sos report --batch  --chroot auto'
2024-10-31 21:00:00,660 DEBUG: [system2:run_command] Shell requested, command is now /bin/bash -c 'sudo -S /usr/sbin/sos report --batch  --chroot auto --only-plugins=block'
2024-10-31 21:00:04,910 INFO: [system2:finalize_sos_path] Final sos path: /var/tmp/sosreport-system2-2024-10-31-iexranf.tar.xz
2024-10-31 21:00:04,911 DEBUG: [system2:run_command] Running command sudo -S chmod o+r /var/tmp/sosreport-system2-2024-10-31-iexranf.tar.xz
2024-10-31 21:00:05,050 INFO: [system2:retrieve_sosreport] Retrieving sos report from 172.31.100.121
2024-10-31 21:00:05,051 DEBUG: [system2:run_command] Running command stat /var/tmp/sosreport-system2-2024-10-31-iexranf.tar.xz
2024-10-31 21:00:05,179 INFO: [system2:retrieve_file] Copying remote /var/tmp/sosreport-system2-2024-10-31-iexranf.tar.xz to local /var/tmp/sos.8rtucemv/
2024-10-31 21:00:05,218 DEBUG: [system2:run_command] Running command stat /var/tmp/sosreport-system2-2024-10-31-iexranf.tar.xz
2024-10-31 21:00:05,345 INFO: [system2:remove_file] Removing file /var/tmp/sosreport-system2-2024-10-31-iexranf.tar.xz
2024-10-31 21:00:05,346 DEBUG: [system2:run_command] Running command sudo -S rm -f /var/tmp/sosreport-system2-2024-10-31-iexranf.tar.xz
2024-10-31 21:00:05,483 DEBUG: [system2:run_command] Running command stat /var/tmp/sosreport-system2-2024-10-31-iexranf.tar.xz.sha256
2024-10-31 21:00:05,611 INFO: [system2:remove_file] Removing file /var/tmp/sosreport-system2-2024-10-31-iexranf.tar.xz.sha256
2024-10-31 21:00:05,612 DEBUG: [system2:run_command] Running command sudo -S rm -f /var/tmp/sosreport-system2-2024-10-31-iexranf.tar.xz.sha256
2024-10-31 21:01:00,790 DEBUG: [system1:execute_sos_command] Error running sos report. rc = 1 msg = sudo: Account or password is expired, reset your password and try again

Current password: 
sudo: timed out reading password

sudo: unable to change expired password: Authentication token manipulation error

sudo: a password is required


2024-10-31 21:01:00,790 INFO: [system1:execute_sos_command] Exception during sos report execution: sudo attempt failed
2024-10-31 21:01:00,791 ERROR: [system1:sosreport] Error during sos execution: sudo attempt failed
2024-10-31 21:01:00,791 INFO: 
Successfully captured 1 of 2 sosreports
2024-10-31 21:01:00,792 DEBUG: [sos_collector:close_all_connections] Closing connection to localhost
2024-10-31 21:01:00,792 DEBUG: [system1:disconnect] Successfully disconnected from node
2024-10-31 21:01:00,792 DEBUG: [sos_collector:close_all_connections] Closing connection to 172.31.100.121
2024-10-31 21:01:00,793 DEBUG: [system2:disconnect] Successfully disconnected from node
2024-10-31 21:01:00,793 INFO: Creating archive of sosreports...

The project-user's password has not expired:

$ sudo chage -l project-user
Last password change					: Oct 30, 2024
Password expires					: Oct 30, 2025
Password inactive					: never
Account expires						: never
Minimum number of days between password change		: 0
Maximum number of days between password change		: 365
Number of days of warning before password expires	: 7

However, the hardening of the systems does appear to set the root user to have an expired password:

$ sudo chage -l root
Last password change					: password must be changed
Password expires					: password must be changed
Password inactive					: password must be changed
Account expires						: never
Minimum number of days between password change		: 0
Maximum number of days between password change		: 99999
Number of days of warning before password expires	: 7

This is identical for both system1 and system2. The only time this seems to be an issue is when sos collect attempts to collect the report from the "local" (collector) node. Whenever this same configuration is found on the remote node there is no issue with using --nopasswd-sudo.

Extra Tests

1. Combinations of the following flags:

   • --batch
   • --nopasswd-sudo
   • --no-local
   • --nodes including the IP of the local (collector) system1 (normally excluded and relying on local collection)
     • I suspected this might be a workaround, however w/ --no-local including the IP of system1 still gets filtered out instead of attempting to "loopback" ssh to itself.

2. Using system2 to attempt collection.

   • Confirmed the issue appears to stick to the collector/local node, as system2 now fails and system1 is properly collected.