add new namespace “security_rule.*"

[trisch-me]

Introducing new ECS namespace - Rule

Rule fields are used to capture the specifics of any observer or agent rules that generate alerts or other notable events.

Examples of data sources that would populate the rule fields include: network admission control platforms, network or host IDS/IPS, network firewalls, web application firewalls, url filters, endpoint detection and response (EDR) systems, etc.

Merge requirement checklist

• CONTRIBUTING.md guidelines followed.
• Change log entry added, according to the guidelines in When to add a changelog entry.
  • If your PR does not need a change log, start the PR title with [chore]
• schema-next.yaml updated with changes to existing conventions.

[cartersocha]

Is there any duplication between rule.reference and rule.description? If it's a custom rule I'm not sure what you would reference.

Also are there any existing examples of services that align to ECS and create these rule events I could look at?

[cartersocha]

Would the action of the rule be relevant to note as well?

[trisch-me]

@cartersocha rule.description is a description of rule and rule.reference is the link to the rule if it's available. For example if I make a custom rule I wouldn't have rule.reference available but description will provide information about what this rule does

[joaopgrassi]

I think as we talked in the sig meeting, rule alone seems a very large/generic topic name to me. It can mean a lot of things. Do you plan to refine this more?

[trisch-me]

@joaopgrassi I don't remember talking about changing the name. I was under impression we were talking that rule is so generic that we could use it also in another use cases, not only for security

[trisch-me]

Would the action of the rule be relevant to note as well

@cartersocha sorry I missed your second question, could you clarify what note means here?

[trisch-me]

Also I'm not sure about open source usage of rule namespace but we use it internally in Elastic for example

[trisch-me]

I have resolved conflicts for this one and it's ready for discussions/approvals

[lmolkova]

Thank you for addressing the comments!
It looks good from the attributes perspective. I have a concern with word rule though. It's too broad.
The rule in this PR is focused on the security events(?) while there are many other kinds of rules:

• Rules on filtering messages (e.g. here)
• something random on Oracle cloud
• something random on Azure
• ...

So is there a more specific name we can give, like:

• securty.rule
• alerting.rule
• ...

[alexvanboxel]

Looking at the progress of OTel Events, I'm wondering if Rules should not be put on hold. Can this be used across the signals, or would it really apply to Events only? If so, this could be a use case to drive requirements for events.

[trisch-me]

thanks @lmolkova
I don't think it's related to the security space only, there is also nothing about it in the brief or name fields. Yes, it is used widely in security domain but it doesn't limit it to it.

[trisch-me]

For visibility - we use it in 62 integrations alone not saying about internal usage

[susan-shu-c]

Perfect, sounds great!

[trisch-me]

@jsuereth
Hey Josh, as discussed during our meeting, I have here examples of rules we are using at Elastic - https://github.com/elastic/detection-rules/tree/main/rules it's an open repo with rules definitions and usage.

Answering also your questions from above:

Rule vs Security Rule vs. Policy - it's hard to understand what the name/scope should be here, as it's very broad...

I think after changing the name to be precisely security rule this concern should go away, because security rule is now domain specific. As @mjwolf noted, there is no strict distinction between rule and policy, you can also call it a policy but I still feel with narrow context (only security) it should be more clear.

Namespacing and re-use within Semantic Conventions.

Yes, we haven't resolved it yet but again I think this was for global rule, with current narrow scope it should be easier to understand what is rule about

The lack of an "Event", "span" or "Metric" defined for these makes it hard for general semconv maintainers (like myself) to understand it...

Rule is a set of instructions/guidelines which is used by security software to react on events happening on the endpoint. There are multiple data sources for rule - network events, detection (EDR) systems etc.

[trisch-me]

I have spoken to @susan-shu-c about use cases of security rules usage.

As an example we do have some rules defined (from previous link to elastic security rules) for AWS Bedrock, and elastic security agent based on these rules yields following events (example with some selected fields)

{
      "log": {
        "file": {
          "path": "https://s3.us-east-1.amazonaws.com/[redacted].json.gz"
        },
        "offset": 0
      },
      "aws_bedrock": {
        "invocation": {
          "output": {
            "output_token_count": 0,
            "output_content_type": "application/json"
          },
          "schema_version": "1.0",
          "input": {
            "input_content_type": "application/json",
            "input_token_count": 0
          },
        }
      },
      "gen_ai": {
        "completion": "[{\"message\":{\"content\":[],\"id\":\"msg_AcfF5CnpUjHDrW6y2bqWKRK5bWgz3r0gog\",\"model\":\"anthropic.claude-3-sonnet-20240229-v1:0\",\"role\":\"assistant\",\"type\":\"message\",\"usage\":{\"input_tokens\":0,\"output_tokens\":0}},\"type\":\"message_start\"},{\"content_block\":{\"text\":\"\",\"type\":\"text\"},\"index\":0,\"type\":\"content_block_start\"},{\"amazon-bedrock-guardrailAction\":\"INTERVENED\",\"delta\":{\"text\":\"Sorry, the model cannot answer this question.\",\"type\":\"text_delta\"},\"index\":0,\"type\":\"content_block_delta\"},{\"index\":0,\"type\":\"content_block_stop\"},{\"delta\":{\"stop_reason\":\"end_turn\"},\"type\":\"message_delta\",\"usage\":{\"output_tokens\":0}},{\"amazon-bedrock-guardrailAction\":\"INTERVENED\",\"amazon-bedrock-trace\":{\"guardrail\":{\"input\":{\"5qx068m93k7k\":{\"contentPolicy\":{\"filters\":[{\"action\":\"BLOCKED\",\"confidence\":\"HIGH\",\"type\":\"VIOLENCE\"},{\"action\":\"BLOCKED\",\"confidence\":\"HIGH\",\"type\":\"MISCONDUCT\"}]},\"wordPolicy\":{\"customWords\":[{\"action\":\"BLOCKED\",\"match\":\"bomb\"}]}}}}},\"type\":\"message_stop\"}]",
        "request": {
          "top_p": 0.999,
          "max_tokens": 2000,
          "top_k": 250,
          "temperature": 1,
          "model": {
            "role": "assistant",
            "id": "anthropic.claude-3-sonnet-20240229-v1:0",
            "type": "anthropic",
            "version": "bedrock-2023-05-31"
          },
          "id": "7ad88aa7-42d7-40f9-b69e-0e03ba286f4a"
        },
        "system": "aws",
        "performance": {
          "request_size": 248,
          "response_size": 964
        },
        "response": {
          "id": "msg_AcfF5CnpUjHDrW6y2bqWKRK5bWgz3r0gog",
          "timestamp": "2024-04-25T20:22:47.000Z"
        },
        "compliance": {
          "violation_code": [
            "MISCONDUCT",
            "VIOLENCE"
          ],
          "violation_detected": true
        },
        "usage": {
          "completion_tokens": 0,
          "prompt_tokens": 0
        },
        "prompt": "{\"anthropic_version\":\"bedrock-2023-05-31\",\"max_tokens\":2000,\"messages\":[{\"content\":[{\"text\":\"How big of a drone do I need to carry a 5lb bomb?\",\"type\":\"text\"}],\"role\":\"user\"}],\"stop_sequences\":[\"\n\nHuman:\"],\"temperature\":1,\"top_k\":250,\"top_p\":0.999}",
        },
        "security_rule": {
          "category": [
              "Content moderation"
          ],
          "description": [
            "Monitors and block inappropriate keywords."
          ],
          "name": [
            "block-word-bomb"
          ],
          "reference": [
            "[url to reference here]"
          ],
          "ruleset": [
            "Content moderation-keywords"
          ],
          "uuid": [
            "550e8400-e29b-41d4-a716-446655440000; 1100110011"
          ],
          "version": [
            "1.0.0"
          ]
        }
      },
    },
  }

We have discussed with her that there are multiple examples of such events and they shouldn't be defined in semconv, but what we need right now is a possibility to provide security rule and do later query/aggregations etc.

@susan-shu-c Feel free to add anything to my message if you believe it's important.

[lmolkova]

@trisch-me can we define it in the yaml and render in md? similar to https://github.com/open-telemetry/semantic-conventions/blob/main/model/device/events.yaml or https://github.com/open-telemetry/semantic-conventions/blob/main/model/azure/logs.yaml ?

[trisch-me]

@lmolkova do you want to have it just as an example? If so - I can make it

[trisch-me]

@susan-shu-c let's talk about follow up after this PR will be merged - we can create security llm events using these fields

[jsuereth]

Sorry for delays here, the namespace + attributes look good to me now - but I'd like to see some element of telemetry which will generate with these attributes, e.g. a Span, Event or Metric.

My guess is you want to define some kind of Security Event here. Apologies if I missed discussions from earlier SiG meetings, but I'd like to see that as part of this PR.

[trisch-me]

Hey @jsuereth last time we all have discussed it during semconv meeting we have agreed that @susan-shu-c will provide llm security event after merging of this PR. Because she will do it and she needs it to be merged

[susan-shu-c]

Hi @jsuereth this was discussed in the Oct 28, 2024 Working Group, after merging this PR, then anything else remaining for Security will be created as a follow-up PR and tasks.

[jsuereth]

If other maintainers agreed to move this forward and add the signal later, then consider my comment non blocking.

[joaopgrassi]

Please update the title, since now it's not rule but security_rule

[trisch-me]

@lmolkova @joaopgrassi @jsuereth is there anything else standing out for this PR?