Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Error loading WAF log to OpenSearch: "mapper_parsing_exception" #436

Open
kynging opened this issue Feb 29, 2024 · 0 comments
Open

Error loading WAF log to OpenSearch: "mapper_parsing_exception" #436

kynging opened this issue Feb 29, 2024 · 0 comments
Labels
bug Something isn't working

Comments

@kynging
Copy link

kynging commented Feb 29, 2024

We found aes-siem-es-loader giving below error message when loading some of the WAF log records to OpenSearch:

{
    "level": "ERROR",
    "message": "1 of logs were NOT loaded into OpenSearch Service",
    "location": "process_record:545",
    "timestamp": "2024-02-28 08:04:48,438+0000",
    "service": "es-loader",
    "cold_start": false,
    "function_name": "aes-siem-es-loader",
    "function_memory_size": "2048",
    "function_arn": "arn:aws:lambda:us-east-1:1234567890:function:aes-siem-es-loader",
    "function_request_id": "72a18220-fdaa-4222-8968-a602fca37f6d",
    "s3_key": "AWSLogs/1234567890/WAF/ap-east-1/2024/02/28/08/aws-waf-logs-xxx-1-2024-02-28-08-03-40-5996b31a-9556-4bc1-9730-8983ecf1690a.gz",
    "s3_bucket": "aes-siem-1234567890-log",
    "message_error": [
        {
            "type": "mapper_parsing_exception",
            "reason": "object mapping for [ruleGroupList.nonTerminatingMatchingRules.ruleMatchDetails.matchedData] tried to parse field [null] as object, but found a concrete value",
            "log_number": 42
        }
    ],
    "xray_trace_id": "1-65dee91f-6087a22c081977d222e15070"
}

Content of log number 42 is as below:

{"timestamp":1709108635066,"formatVersion":1,"webaclId":"arn:aws:wafv2:ap-east-1:1234567890:regional/webacl/xxx/b954fa85-5178-4201-8948-221e9bf0d032","terminatingRuleId":"AWS-AWSManagedRulesCommonRuleSet","terminatingRuleType":"MANAGED_RULE_GROUP","action":"BLOCK","terminatingRuleMatchDetails":[{"conditionType":"XSS","location":"BODY","matchedData":["<?","xml"],"matchedFieldName":""}],"httpSourceName":"ALB","httpSourceId":"1234567890-app/qa-xxx-alb-new/72afb40a799b993a","ruleGroupList":[{"ruleGroupId":"AWS#AWSManagedRulesAmazonIpReputationList","terminatingRule":null,"nonTerminatingMatchingRules":[],"excludedRules":null,"customerConfig":null},{"ruleGroupId":"AWS#AWSManagedRulesCommonRuleSet","terminatingRule":{"ruleId":"CrossSiteScripting_BODY","action":"BLOCK","ruleMatchDetails":null},"nonTerminatingMatchingRules":[{"ruleId":"CrossSiteScripting_BODY_RC_COUNT","action":"COUNT","ruleMatchDetails":[{"conditionType":"XSS","location":"BODY","matchedData":["<?","xml"],"matchedFieldName":""}]},{"ruleId":"SizeRestrictions_BODY","action":"COUNT","overriddenAction":"BLOCK","ruleMatchDetails":[]}],"excludedRules":null,"customerConfig":null}],"rateBasedRuleList":[],"nonTerminatingMatchingRules":[],"requestHeadersInserted":null,"responseCodeSent":null,"httpRequest":{"clientIp":"[103.60.248.229](http://103.60.248.229/)","country":"HK","headers":[{"name":"Cache-control","value":"no-cache"},{"name":"Cache-store","value":"no-store"},{"name":"Pragma","value":"no-cache"},{"name":"User-Agent","value":"Apache-Maven/3.6.3 (Java 1.8.0_181; Mac OS X 10.16)"},{"name":"Content-Length","value":"10492"},{"name":"Host","value":"xxx.xxx.global"},{"name":"Connection","value":"Keep-Alive"},{"name":"Expect","value":"100-continue"},{"name":"Accept-Encoding","value":"gzip,deflate"},{"name":"Authorization","value":"Basic YWRtaW46b3ZlcnNlYXM="}],"uri":"/repository/maven-snapshots/com/kun/kun-dependencies/1.0-SNAPSHOT/kun-dependencies-1.0-20240228.082354-59.pom","args":"","httpVersion":"HTTP/1.1","httpMethod":"PUT","requestId":"1-65deed9a-4d373a463405c2ed763a5638"},"labels":[{"name":"awswaf:managed:aws:core-rule-set:CrossSiteScripting_Body_RC_COUNT"},{"name":"awswaf:managed:aws:core-rule-set:CrossSiteScripting_Body"},{"name":"awswaf:managed:aws:core-rule-set:SizeRestrictions_Body"}],"oversizeFields":["REQUEST_BODY"],"requestBodySize":10492,"requestBodySizeInspectedByWAF":8192}

This seems to be a bug.

@nakajiak nakajiak added the bug Something isn't working label Mar 1, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
Development

No branches or pull requests

2 participants