The automation framework supports testing the job outcomes based on conditions that you can set in the YAML file. Currently, tests of type ‘stats’ and ‘alert’ are available. More types will be added in the future.
Statistics-based tests are supported by all add-ons that have an automation job. If there is a relevant statistic used by an add-on, a test can be created for it. A job can have tests for multiple statistics and multiple tests can be created for the same statistic. Note that statistics-based tests will function correctly only if in memory statistics have been enabled.
jobs: - type: example parameters: jobSpecificParam: 'value' tests: - name: 'test one' # Name of the test, optional type: stats # Specifies that the test is of type 'stats' statistic: 'stats.addon.something' # Name of an integer / long statistic operator: '>=' # One of '==', '!=', '>=', '>', '<', '<=' value: 10 # Value to compare statistic against onFail: 'info' # String: One of 'warn', 'error', 'info', mandatory
Alert tests are supported by the activeScan and passiveScan-wait jobs. These tests can be used to validate the presence/absence of specific alerts in the active/passive scan. It is mandatory for the alerts specified in the plan to have a scanRuleId, against which the generated alerts will always be matched. All other fields describing an alert are optional regexes, and will be matched against only if they are specified. A job can have tests for multiple alerts, and multiple tests can be created for alerts having the same scanRuleId.
jobs: - type: activeScan # The active scanner - this actively attacks the target so should only be used with permission parameters: context: # String: Name of the context to attack, default: first context policy: # String: Name of the scan policy to be used, default: Default Policy maxRuleDurationInMins: # Int: The max time in minutes any individual rule will be allowed to run for, default: 0 unlimited maxScanDurationInMins: # Int: The max time in minutes the active scanner will be allowed to run for, default: 0 unlimited tests: - name: 'test one' # Name of the test, optional type: alert # Specifies that the test is of type 'alert' action: passIfPresent/passIfAbsent # String: The condition (presence/absence) of the alert, default: passIfAbsent scanRuleId: # Integer: The id of the scanRule which generates the alert, mandatory alertName: # String: The name of the alert generated, optional url: http://www.example.com/path # String: The url of the request corresponding to the alert generated, optional method: # String: The method of the request corresponding to the alert generated, optional attack: # String: The actual attack which generated the alert, optional param: # String: The parameter which was modified to generate the alert, optional evidence: # String: The evidence corresponding to the alert generated, optional confidence: # String: The confidence of the alert, one of 'False Positive', 'Low', 'Medium', 'High', 'Confirmed', optional risk: # String: The risk of the alert, one of 'Informational', 'Low', 'Medium', 'High', optional otherInfo: # String: Additional information corresponding to the alert, optional onFail: 'info' # String: One of 'warn', 'error', 'info', mandatory