When reviewing a PR which changed rules only applicable to OL, though an explicit defition of prodtype: ol7, it was noted that CI tests only relevant for RHEL and Fedora were executed and failed with the following output:
---------Bash Remediation Logs---------
ERROR - Rule 'xccdf_org.ssgproject.content_rule_accounts_authorized_local_users_sidadm_orasid' isn't present in benchmark 'xccdf_org.ssgproject.content_benchmark_RHEL-8' in '/tmp/ssgts-ds-gzmbdvrc'
ERROR - No tests found matching the rule ID(s) 'accounts_authorized_local_users_sidadm_orasid, java_jre_insecure_prompt_locked, java_jre_enable_jws, java_jre_security_revocation_check_locked, java_jre_validation_crl_locked, java_jre_validation_ocsp, java_jre_deployment_config_mandatory, java_jre_lock_untrusted_sources, java_jre_deployment_properties_exists, java_jre_validation_crl, java_jre_disable_untrusted_sources, java_jre_validation_ocsp_locked, java_jre_accepted_sites_exists, java_jre_unsigned_applications, java_jre_unsigned_applications_locked, java_jre_security_revocation_check, java_jre_blacklist_check_locked, java_jre_enable_jws_locked, java_jre_deployment_config_properties, java_jre_blacklist_check, java_jre_disable_untrusted_sources_locked, java_jre_lock_untrusted_sources_locked, java_jre_accepted_sites_properties, java_jre_insecure_prompt, java_jre_deployment_config_exists'
WARNING - Nothing has been tested!
The output is correct since the rule is indeed not available for RHEL8, as in this example.
In such cases, testing this rule in RHEL is not necessary since the result will always fail.
It would be good to have an option or other mechanism to not test rules in some products when the rules are not applicable for the respective product.
When reviewing a PR which changed rules only applicable to OL, though an explicit defition of prodtype: ol7, it was noted that CI tests only relevant for RHEL and Fedora were executed and failed with the following output:
The output is correct since the rule is indeed not available for RHEL8, as in this example.
In such cases, testing this rule in RHEL is not necessary since the result will always fail.
It would be good to have an option or other mechanism to not test rules in some products when the rules are not applicable for the respective product.
References:
PR: ComplianceAsCode/content#9398
Tests CS8: https://github.com/ComplianceAsCode/content/runs/7998438537?check_suite_focus=true
Tests CS9: https://github.com/ComplianceAsCode/content/runs/7998421987?check_suite_focus=true