Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Automatus struggles with profile-specific test scenarios #10369

Open
jan-cerny opened this issue Mar 27, 2023 · 0 comments
Open

Automatus struggles with profile-specific test scenarios #10369

jan-cerny opened this issue Mar 27, 2023 · 0 comments
Labels
Test Suite Update in Test Suite.

Comments

@jan-cerny
Copy link
Collaborator

Description of problem:

During his work on #10348, @maage has discovered that with some rules there are unexpected errors when Automatus runs in rule mode. This happens when testing rule sssd_ldap_configure_tls_ca_dir on a RHEL 8 virtual machine.

The trigger seems to be that the test scenarios for this rule have profile limitation in their headers but the rule isn't part of the given profile. In rule sssd_ldap_configure_tls_ca_dir all the test scenarios have # profiles = xccdf_org.ssgproject.content_profile_stig but the rule isn't part of RHEL 8 STIG profile. It is only part of RHEL 7 STIG profile and Oracle Linux 7 profile. But still the rule is present in RHEL 8 data stream.

As a consequence of the rule not being present in the profile, for the .fail.sh test scenarios, Automatus hits the issue in OpenSCAP OpenSCAP/openscap#1963 that means that it generates an empty Ansible Playbook and then Automatus tries to apply this empty Ansible Playbook which doesn't remediate anything and then the final scan isn't fine.

The main question is the expected behavior for the profile-specific scenarios in a situation when the given rule isn't part of the profile but exists in the given SCAP source data stream.

SCAP Security Guide Version:

current upstream as of 2022-03-27 as of HEAD 2c18502

Operating System Version:

F 37

Steps to Reproduce:

  1. python3 tests/automatus.py rule --libvirt qemu:///system ssgts_rhel8 --remediate-using ansible --dontclean sssd_ldap_configure_tls_ca_dir

Actual Results:

INFO - The base image option has not been specified, choosing libvirt-based test environment.
INFO - Logging into /home/jcerny/work/git/scap-security-guide/logs/rule-custom-2023-03-27-1320/test_suite.log
INFO - xccdf_org.ssgproject.content_rule_sssd_ldap_configure_tls_ca_dir
INFO - Script domain_not_there.fail.sh using profile xccdf_org.ssgproject.content_profile_stig OK
ERROR - Rule evaluation resulted in fail, instead of expected pass during final stage 
ERROR - The check after remediation failed for rule 'xccdf_org.ssgproject.content_rule_sssd_ldap_configure_tls_ca_dir'.
INFO - Script ldap_tls_cacertdir.pass.sh using profile xccdf_org.ssgproject.content_profile_stig OK
INFO - Script ldap_tls_cacertdir_bad_value.fail.sh using profile xccdf_org.ssgproject.content_profile_stig OK
ERROR - Rule evaluation resulted in fail, instead of expected pass during final stage 
ERROR - The check after remediation failed for rule 'xccdf_org.ssgproject.content_rule_sssd_ldap_configure_tls_ca_dir'.
INFO - Script ldap_tls_cacertdir_not_absolute_path.fail.sh using profile xccdf_org.ssgproject.content_profile_stig OK
ERROR - Rule evaluation resulted in fail, instead of expected pass during final stage 
ERROR - The check after remediation failed for rule 'xccdf_org.ssgproject.content_rule_sssd_ldap_configure_tls_ca_dir'.
INFO - Script ldap_tls_cacertdir_not_there.fail.sh using profile xccdf_org.ssgproject.content_profile_stig OK
ERROR - Rule evaluation resulted in fail, instead of expected pass during final stage 
ERROR - The check after remediation failed for rule 'xccdf_org.ssgproject.content_rule_sssd_ldap_configure_tls_ca_dir'.

Expected Results:

Probably no errors. But the expected results is a question that needs to be answered.

Additional Information/Debugging Steps:

  • notice that the Playbook logs/rule-custom-2023-03-27-1320/xccdf_org.ssgproject.content_rule_sssd_ldap_configure_tls_ca_dir.yml doesn't contain any tasks
  • notice how the # profile header is handled by Automatus
  • notice that Automatus evaluates TS even the rule isn't a part of the STIG profile
@jan-cerny jan-cerny added the Test Suite Update in Test Suite. label Mar 27, 2023
@jan-cerny jan-cerny mentioned this issue Mar 27, 2023
Merged
jan-cerny added a commit to jan-cerny/scap-security-guide that referenced this issue Mar 30, 2023
@jan-cerny
Add a warning to AutoMatus
06b3498 
This adds a warning that prevents a confusion in situation when
a test scenario has a profile in its header but the rule isn't
a part of that profile but is present in the built data stream.

For more context, see:
ComplianceAsCode#10369
@jan-cerny jan-cerny mentioned this issue Mar 30, 2023
Merged
jan-cerny added a commit to jan-cerny/scap-security-guide that referenced this issue Mar 30, 2023
@jan-cerny
Add a warning to AutoMatus
a8f42b9 
This adds a warning that prevents a confusion in situation when
a test scenario has a profile in its header but the rule isn't
a part of that profile but is present in the built data stream.

For more context, see:
ComplianceAsCode#10369
jan-cerny added a commit to jan-cerny/scap-security-guide that referenced this issue Apr 4, 2023
@jan-cerny
Add a warning to AutoMatus
bbbc85f 
This adds a warning that prevents a confusion in situation when
a test scenario has a profile in its header but the rule isn't
a part of that profile but is present in the built data stream.

For more context, see:
ComplianceAsCode#10369
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
Test Suite Update in Test Suite.
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@jan-cerny