Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Incorrect output in iis/error module on Travis on 6.x #8224

Closed
kvch opened this issue Sep 4, 2018 · 5 comments
Closed

Incorrect output in iis/error module on Travis on 6.x #8224

kvch opened this issue Sep 4, 2018 · 5 comments
Labels
bug Filebeat Filebeat module Team:Integrations Label for the Integrations team :Testing

Comments

@kvch
Copy link
Contributor

kvch commented Sep 4, 2018

Test logs of iis/error module are not parsed correctly on 6.x. I have tried to generate the expected output using GENERATE=1 INTEGRATION_TESTS=1 TESTING_FILEBEAT_MODULES=iis nosetests tests/system/test_modules.py but it does not help.
I cannot reproduce it locally and it does not fail on Jenkins.

@ruflin Do you have an idea what went wrong?

This is the error message:

AssertionError: The following expected object was not found:
 {
  "nginx.access.user_agent.major": "49",
  "nginx.access.user_agent.os_name": "Mac OS X",
  "prospector.type": "log",
  "@timestamp": "2016-12-07T10:05:07.000Z",
  "nginx.access.response_code": "200",
  "nginx.access.body_sent.bytes": "571",
  "nginx.access.geoip.location.lat": 52.5167,
  "nginx.access.referrer": "-",
  "nginx.access.user_agent.device": "Other",
  "nginx.access.http_version": "1.1",
  "nginx.access.geoip.continent_name": "Europe",
  "nginx.access.geoip.region_name": "Land Berlin",
  "input.type": "log",
  "nginx.access.url": "/ocelot",
  "nginx.access.method": "GET",
  "nginx.access.geoip.country_iso_code": "DE",
  "nginx.access.user_agent.os_major": "10",
  "nginx.access.user_agent.os": "Mac OS X 10.12",
  "nginx.access.remote_ip_list": [
    "10.0.0.2",
    "10.0.0.1",
    "85.181.35.98"
  ],
  "nginx.access.user_agent.os_minor": "12",
  "fileset.module": "nginx",
  "offset": 341,
  "fileset.name": "access",
  "nginx.access.user_agent.name": "Firefox",
  "nginx.access.user_name": "-",
  "nginx.access.remote_ip": "85.181.35.98",
  "nginx.access.geoip.location.lon": 13.4,
  "nginx.access.geoip.city_name": "Berlin",
  "nginx.access.user_agent.minor": "0"
}
Searched in: 
[
  {
    "nginx": {
      "access": {
        "body_sent": {
          "bytes": "571"
        },
        "url": "/ocelot",
        "referrer": "-",
        "response_code": "200",
        "remote_ip_list": [
          "10.0.0.2",
          "10.0.0.1",
          "127.0.0.1"
        ],
        "user_agent": {
          "major": "49",
          "os_name": "Mac OS X",
          "name": "Firefox",
          "os_minor": "12",
          "os_major": "10",
          "device": "Other",
          "os": "Mac OS X 10.12",
          "minor": "0"
        },
        "http_version": "1.1",
        "user_name": "-",
        "method": "GET",
        "remote_ip": "10.0.0.2"
      }
    },
    "beat": {
      "hostname": "eac06347dff0",
      "name": "eac06347dff0",
      "version": "6.5.0"
    },
    "@timestamp": "2016-12-07T10:05:07.000Z",
    "read_timestamp": "2018-09-04T17:15:54.086Z",
    "fileset": {
      "name": "access",
      "module": "nginx"
    },
    "source": "/go/src/github.com/elastic/beats/filebeat/module/nginx/access/test/test.log",
    "host": {
      "name": "eac06347dff0"
    },
    "offset": 0,
    "input": {
      "type": "log"
    },
    "prospector": {
      "type": "log"
    }
  },
  {
    "nginx": {
      "access": {
        "body_sent": {
          "bytes": "612"
        },
        "url": "/stringpatch",
        "referrer": "-",
        "response_code": "404",
        "remote_ip_list": [
          "172.17.0.1"
        ],
        "user_agent": {
          "major": "15",
          "os_name": "Windows 7",
          "name": "Firefox Alpha",
          "patch": "a2",
          "device": "Other",
          "os": "Windows 7",
          "minor": "0"
        },
        "http_version": "1.1",
        "user_name": "-",
        "method": "GET",
        "remote_ip": "172.17.0.1"
      }
    },
    "beat": {
      "hostname": "eac06347dff0",
      "name": "eac06347dff0",
      "version": "6.5.0"
    },
    "@timestamp": "2017-05-29T19:02:48.000Z",
    "read_timestamp": "2018-09-04T17:15:54.086Z",
    "fileset": {
      "name": "access",
      "module": "nginx"
    },
    "source": "/go/src/github.com/elastic/beats/filebeat/module/nginx/access/test/test.log",
    "host": {
      "name": "eac06347dff0"
    },
    "offset": 183,
    "input": {
      "type": "log"
    },
    "prospector": {
      "type": "log"
    }
  },
  {
    "nginx": {
      "access": {
        "body_sent": {
          "bytes": "571"
        },
        "http_version": "1.1",
        "url": "/ocelot",
        "referrer": "-",
        "response_code": "200",
        "remote_ip_list": [
          "10.0.0.2",
          "10.0.0.1",
          "85.181.35.98"
        ],
        "user_agent": {
          "major": "49",
          "os_name": "Mac OS X",
          "name": "Firefox",
          "os_minor": "12",
          "os_major": "10",
          "device": "Other",
          "os": "Mac OS X 10.12",
          "minor": "0"
        },
        "geoip": {
          "region_iso_code": "DE-BE",
          "country_iso_code": "DE",
          "city_name": "Berlin",
          "location": {
            "lat": 52.5167,
            "lon": 13.4
          },
          "continent_name": "Europe",
          "region_name": "Land Berlin"
        },
        "user_name": "-",
        "method": "GET",
        "remote_ip": "85.181.35.98"
      }
    },
    "beat": {
      "hostname": "eac06347dff0",
      "name": "eac06347dff0",
      "version": "6.5.0"
    },
    "@timestamp": "2016-12-07T10:05:07.000Z",
    "read_timestamp": "2018-09-04T17:15:54.086Z",
    "fileset": {
      "name": "access",
      "module": "nginx"
    },
    "source": "/go/src/github.com/elastic/beats/filebeat/module/nginx/access/test/test.log",
    "host": {
      "name": "eac06347dff0"
    },
    "offset": 341,
    "input": {
      "type": "log"
    },
    "prospector": {
      "type": "log"
    }
  },
  {
    "nginx": {
      "access": {
        "body_sent": {
          "bytes": "571"
        },
        "http_version": "1.1",
        "url": "/ocelot",
        "referrer": "-",
        "response_code": "200",
        "remote_ip_list": [
          "85.181.35.98"
        ],
        "user_agent": {
          "major": "49",
          "os_name": "Mac OS X",
          "name": "Firefox",
          "os_minor": "12",
          "os_major": "10",
          "device": "Other",
          "os": "Mac OS X 10.12",
          "minor": "0"
        },
        "geoip": {
          "region_iso_code": "DE-BE",
          "country_iso_code": "DE",
          "city_name": "Berlin",
          "location": {
            "lat": 52.5167,
            "lon": 13.4
          },
          "continent_name": "Europe",
          "region_name": "Land Berlin"
        },
        "user_name": "-",
        "method": "GET",
        "remote_ip": "85.181.35.98"
      }
    },
    "beat": {
      "hostname": "eac06347dff0",
      "name": "eac06347dff0",
      "version": "6.5.0"
    },
    "@timestamp": "2016-12-07T10:05:07.000Z",
    "read_timestamp": "2018-09-04T17:15:54.086Z",
    "fileset": {
      "name": "access",
      "module": "nginx"
    },
    "source": "/go/src/github.com/elastic/beats/filebeat/module/nginx/access/test/test.log",
    "host": {
      "name": "eac06347dff0"
    },
    "offset": 527,
    "input": {
      "type": "log"
    },
    "prospector": {
      "type": "log"
    }
  },
  {
    "nginx": {
      "access": {
        "body_sent": {
          "bytes": "25507"
        },
        "http_version": "1.1",
        "url": "/assets/xxxx?q=100",
        "referrer": "-",
        "response_code": "200",
        "remote_ip_list": [
          "10.5.102.222",
          "199.96.1.1",
          "204.246.1.1",
          "10.2.1.185"
        ],
        "user_agent": {
          "device": "Other",
          "os_name": "Other",
          "os": "Other",
          "name": "Other"
        },
        "geoip": {
          "region_iso_code": "US-IL",
          "country_iso_code": "US",
          "city_name": "Springfield",
          "location": {
            "lat": 39.772,
            "lon": -89.6859
          },
          "continent_name": "North America",
          "region_name": "Illinois"
        },
        "user_name": "-",
        "method": "GET",
        "remote_ip": "199.96.1.1"
      }
    },
    "beat": {
      "hostname": "eac06347dff0",
      "name": "eac06347dff0",
      "version": "6.5.0"
    },
    "@timestamp": "2016-01-22T13:18:29.000Z",
    "read_timestamp": "2018-09-04T17:15:54.087Z",
    "fileset": {
      "name": "access",
      "module": "nginx"
    },
    "source": "/go/src/github.com/elastic/beats/filebeat/module/nginx/access/test/test.log",
    "host": {
      "name": "eac06347dff0"
    },
    "offset": 693,
    "input": {
      "type": "log"
    },
    "prospector": {
      "type": "log"
    }
  },
  {
    "nginx": {
      "access": {
        "body_sent": {
          "bytes": "8571"
        },
        "http_version": "1.1",
        "url": "/test.html",
        "referrer": "-",
        "response_code": "404",
        "remote_ip_list": [
          "2a03:0000:10ff:f00f:0000:0000:0:8000",
          "10.225.192.17",
          "10.2.2.121"
        ],
        "user_agent": {
          "major": "1",
          "os_name": "Other",
          "name": "Facebot",
          "device": "Spider",
          "os": "Other",
          "minor": "0"
        },
        "geoip": {
          "continent_name": "Europe",
          "country_iso_code": "PT",
          "location": {
            "lat": 39.5,
            "lon": -8.0
          }
        },
        "user_name": "-",
        "method": "GET",
        "remote_ip": "2a03:0000:10ff:f00f:0000:0000:0:8000"
      }
    },
    "beat": {
      "hostname": "eac06347dff0",
      "name": "eac06347dff0",
      "version": "6.5.0"
    },
    "@timestamp": "2016-12-30T06:47:09.000Z",
    "read_timestamp": "2018-09-04T17:15:54.087Z",
    "fileset": {
      "name": "access",
      "module": "nginx"
    },
    "source": "/go/src/github.com/elastic/beats/filebeat/module/nginx/access/test/test.log",
    "host": {
      "name": "eac06347dff0"
    },
    "offset": 845,
    "input": {
      "type": "log"
    },
    "prospector": {
      "type": "log"
    }
  },
  {
    "nginx": {
      "access": {
        "body_sent": {
          "bytes": "0"
        },
        "referrer": "-",
        "response_code": "400",
        "remote_ip_list": [
          "127.0.0.1"
        ],
        "user_agent": {
          "device": "Other",
          "os_name": "Other",
          "os": "Other",
          "name": "Other"
        },
        "user_name": "-",
        "remote_ip": "127.0.0.1"
      }
    },
    "beat": {
      "hostname": "eac06347dff0",
      "name": "eac06347dff0",
      "version": "6.5.0"
    },
    "@timestamp": "2018-04-12T07:48:40.000Z",
    "read_timestamp": "2018-09-04T17:15:54.087Z",
    "fileset": {
      "name": "access",
      "module": "nginx"
    },
    "source": "/go/src/github.com/elastic/beats/filebeat/module/nginx/access/test/test.log",
    "host": {
      "name": "eac06347dff0"
    },
    "offset": 1085,
    "input": {
      "type": "log"
    },
    "prospector": {
      "type": "log"
    }
  }
]
@ruflin
Copy link
Member

ruflin commented Sep 4, 2018

Didn't check it in detail yet but I assume it's related to different geoip coordiantes. @jsoriano is working on a similar issue.

@kvch
Copy link
Contributor Author

kvch commented Sep 4, 2018

The coordinates are the same. Every field seems to be similar except for the format of the events.

@ruflin
Copy link
Member

ruflin commented Sep 5, 2018

The thing that is missing in the event is region_iso_code. Can you remove all your local elasticsearch containters and run it again? After that it should work.

@jsoriano
Copy link
Member

jsoriano commented Sep 5, 2018

This is the related issue #8204

@ruflin ruflin added the Team:Integrations Label for the Integrations team label Nov 27, 2018
@andrewvc andrewvc mentioned this issue Dec 12, 2018
Closed
@alvarolobato
Copy link

This is related to an update of the geoip database. Let's close this for now (Nicolas says)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug Filebeat Filebeat module Team:Integrations Label for the Integrations team :Testing
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
4 participants
@jsoriano @ruflin @kvch @alvarolobato