Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Update field mapping documentation for Security Analytics #2422

Merged
merged 14 commits into from
Jan 23, 2023
37 changes: 31 additions & 6 deletions _security-analytics/sec-analytics-config/detectors-config.md
Original file line number Diff line number Diff line change
Expand Up @@ -28,21 +28,46 @@ Defining a new detector involves naming the detector, selecting a data source an
* Use the **Log type**, **Rule severity**, and **Source** dropdown menus to filter the rules you want to select from.
* Use the **Search** bar to search for specific rules.

To quickly select one or more known rules and dismiss others, first deselect all rules by moving the **rule name** toggle to the left, then search for your target rule names and select each individually by moving its toggle to the right.
To quickly select one or more known rules and dismiss others, first deselect all rules by moving the **Rule name** toggle to the left, then search for your target rule names and select each individually by moving its toggle to the right.
{: .tip }

1. In the **Detector schedule** section, set how often the detector will run. Specify a unit of time and a corresponding number to set the interval.
1. Select the **Next** button in the lower-right corner of the screen to continue. The Configure field mapping page appears.

## Step 2. Making field mappings
## Step 2. Create field mappings

Field mapping matches field names for the rule with field names from the log being used to provide data. The mappings are automatically applied once the detector is defined in previous steps. This page offers the user the option to map log-specific field names to the internal rule field names.
The field mapping step matches field names from the rule with field names from the log index being used to provide data. Creating field mappings allows the system to accurately convey event data from the log to the detector and then use the data for triggering alerts.
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
The field mapping step matches field names from the rule with field names from the log index being used to provide data. Creating field mappings allows the system to accurately convey event data from the log to the detector and then use the data for triggering alerts.
The field mapping step matches field names from the rule with field names from the log index being used to provide data. Creating field mappings allows the system to accurately convey event data from the log to the detector and then use the data to trigger alerts.


For example, if you prefer to have the log field name UserID rather than EventID correspond to the event_uid rule field name, you can use the **Log field name** dropdown menu to select **UserID**.
The data source (log index), log type, and detection rules specified in the first step determine which fields are available for mapping. For example, when "Windows logs" is selected as the log type, this parameter along with the specific detection rules determine the list of rule field names available for the mapping. Similarly, the selected data source (log index) determines the list of log field names that are available for the mapping.
cwillum marked this conversation as resolved.
Show resolved Hide resolved

<img src="{{site.url}}{{site.baseurl}}/images/Security/field_map.png" alt="Rule and log field mapping example">
#### A note on field names

To make any changes to the automatically populated mappings, use the dropdown arrows across from the rule field names to specify a preferred log field name for the mapping. After completing the mappings, select the **Next** button in the lower-right corner of the screen. The Set up alerts page appears and displays settings for an alert trigger.
The field mapping process requires that you are familiar with the field names in the log index and have an understanding of the data contained in those fields. If you have an understanding of the log fields in the index, the mapping is typically a simple and straightforward process.
cwillum marked this conversation as resolved.
Show resolved Hide resolved

Security Analytics takes advantage of pre-packaged Sigma rules for security event detection. Therefore, the rule field names are derived from a Sigma rule field standard. To make them easier to identify, however, we create aliases for the Sigma rule fields based on the open source Elastic Common Schema (ECS) specification. These alias rule field names are the field names used in these steps.
cwillum marked this conversation as resolved.
Show resolved Hide resolved

Although the ECS rule field names are largely self explanatory, you can find pre-defined mappings of the Sigma rule field names with ECS rule field names for all supported log types in the GitHub Security Analytics repository. First navigate to the [OSMappings](https://github.com/opensearch-project/security-analytics/tree/main/src/main/resources/OSMapping) folder, select the folder named for the log type, and open the `fieldmappings.yml` file. For example, to see the Sigma rule fields that correspond to ECS rule fields for the Windows log type, open the [fieldmappings.yml file](https://github.com/opensearch-project/security-analytics/blob/main/src/main/resources/OSMapping/windows/fieldmappings.yml) in the **windows** folder.
cwillum marked this conversation as resolved.
Show resolved Hide resolved

### Pending field mappings

Once you navigate to the Configure field mapping page, the system attempts to automatically map fields between the two sources. Those field names that are not automatically mapped appear in the **Pending field mapping** table. In this table you can manually map rule fields to log fields.
cwillum marked this conversation as resolved.
Show resolved Hide resolved
<br><img src="{{site.url}}{{site.baseurl}}/images/Security/pending-mappings.png" alt="Field mapping example for pending mappings" width="900">

* The Rule field name column lists field names based on all of the pre-packaged rules associated with the selected log type.
cwillum marked this conversation as resolved.
Show resolved Hide resolved
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The bulleted list should ideally be introduced by a brief sentence ending in a colon.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Added ...
"While mapping fields, consider the following:"

* The log field name column includes a dropdown list for each of the rule fields. Each dropdown list contains field names extracted from the log index.
* To map a rule field name to a log field name, use the dropdown arrow to open the list of log fields and select the log field name from the list. To search for names in the log field list, enter text in the **Select a mapping field** box.
<br><img src="{{site.url}}{{site.baseurl}}/images/Security/log-field.png" alt="Field mapping example for pending mappings" width="600">
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reference the image in the text that precedes it. ", as shown in the following image." usually works.

* Once the log field name is selected and mapped to the rule field name, the icon in the Status column to the right changes to a green check mark.
* Make as many matches between field names as possible to complete an accurate mapping for rule and log fields.

### Default field mappings

The **Default mapped fields** table contains mappings that the system made automatically after defining the detector. As shown in the image that follows, when the field names are similar to one another the system can successfully match the two.
cwillum marked this conversation as resolved.
Show resolved Hide resolved
<br><img src="{{site.url}}{{site.baseurl}}/images/Security/default-mappings.png" alt="Field mapping example for pending mappings" width="900">

Nevertheless, it's a good idea to review the mappings and verify that they are correct and matched as expected. If you find a mapping that doesn't appear to be accurate, you can use the dropdown list as described in the [Pending field mappings](#pending-field-mappings) section above to correct the field mapping.
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
Nevertheless, it's a good idea to review the mappings and verify that they are correct and matched as expected. If you find a mapping that doesn't appear to be accurate, you can use the dropdown list as described in the [Pending field mappings](#pending-field-mappings) section above to correct the field mapping.
Nevertheless, it's a good idea to review the mappings and verify that they are correct and matched as expected. If you find a mapping that doesn't appear to be accurate, you can use the dropdown list as described in the [Pending field mappings](#pending-field-mappings) section to correct the field mapping.

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can we use something simpler than "Nevertheless"? It may not be understood by Global English speakers.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I added this ...
"Although these automatic matches are normally dependable, it's still a good idea to review the mappings ..."


After completing the mappings, select the **Next** button in the lower-right corner of the screen. The **Set up alerts** page appears and displays settings for an alert trigger.

## Step 3. Setting up alerts

Expand Down
Binary file added images/Security/default-mappings.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added images/Security/log-field.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added images/Security/pending-mappings.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.