Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Document how to configure ingest node to have geoip #2793

Merged
merged 2 commits into from
Oct 19, 2016

Conversation

dedemorton
Copy link
Contributor

@monicasarbu Here's my first draft. I probably need to give this another good read through, but I wanted to get it out to you for a first review. I've added a step to define mappings in the index template to get the location field to work in the visualization (without the mappings, the latitude and longitude get indexed as floats). (I haven't really worked much with mappings, so I hope it's OK.) If there is someway to take the latitude and longitude and index it directly into the client_location field (which is already a geo_point) then that would be better, but I'm not sure how to do that using the options that are available in the ingest geoIP processor plugin.

@monicasarbu monicasarbu added the in progress Pull request is currently in progress. label Oct 18, 2016
@monicasarbu
Copy link
Contributor

monicasarbu commented Oct 18, 2016

I have created a PR #2795 to make a smooth integration between Beats and the Ingest GeoIP Processor Plugin. Here are the steps that the user should follow in case he/she needs to add GeoIP information for the client_ip:

  1. Install Ingest GeoIP Processor Plugin: https://www.elastic.co/guide/en/elasticsearch/plugins/master/ingest-geoip.html
  2. Define a GeoIP processor in a pipeline by using Console from Kibana:
PUT _ingest/pipeline/geoip-info
{
  "description" : "Add geoip info",
  "processors" : [
    {
      "geoip" : {
        "field" : "client_ip",
        "target_field": "client_geoip",
        "properties": ["location"],
        "ignore_failure" : true
      }
    }
  ]
}

This will add the client_geoip.location field of type geo_point to the event.
3. Start Packetbeat (feature available only in 5.0.0-GA)
4. Import Kibana dashboard for Packetbeat or create a Tile map using client_geoip.location as Geohash.

Things that needs to be done (hopefully part of this PR):

City Database. Using this plugin provides you with greater flexibility and
control over the information that's returned by the geoIP lookup.

//REVIEWERS: trying to put some words around why the plugin is better, but I'm not sure if this is completely true. ^^
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The benefit is that you don't need to install the geoip database on all the Beats, and you can install it only on your Elasticsearch.

{plugindoc}/using-ingest-geoip.html[Using the Geoip Processor in a Pipeline] for
more options.

3. Add the pipeline to Elasticsearch. For example:
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

to Ingest Node instead of ES

[source,shell]
-------------------------------------------------------------------------------
curl -XPUT 'http://localhost:9200/_ingest/pipeline/geoip-info' [email protected]
-------------------------------------------------------------------------------
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

or another easier solution would be to use Console from Kibana instead of writing it to a file and then loading it usign curl.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I prefer your streamlined approach. I was trying to be consistent with how we describe using ingest node elsewhere, but I think it's OK for users to see a couple of different ways to define pipelines and to know that the syntax is a bit easier if you use the console in Kibana.

pipeline: geoip-info
-------------------------------------------------------------------------------

5. Before you start Packetbeat, edit the Packetbeat index template,
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this step is not needed, as I changed the template to accommodate client_geoip.location.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Awesome! That will be so much easier for the users.

@monicasarbu
Copy link
Contributor

@dedemorton The documentation looks good to me. Please update it with the latest changes I wrote above.

@dedemorton dedemorton removed the in progress Pull request is currently in progress. label Oct 19, 2016
@dedemorton
Copy link
Contributor Author

@monicasarbu I've resolved the comments from the review. I decided that I wouldn't cover the Kibana steps in too much detail, but I kept info about refreshing the index field list because I ran into this problem when I tested the steps.

@dedemorton
Copy link
Contributor Author

This PR resolves #1671, which is also tracked in #2482

@monicasarbu monicasarbu merged commit 3c4d343 into elastic:master Oct 19, 2016
@monicasarbu monicasarbu deleted the beats_issue#1671 branch October 19, 2016 07:26
dedemorton added a commit to dedemorton/beats that referenced this pull request Oct 19, 2016
* Document how to configure ingest node to have geoip

* Resolve comments from review
tsg pushed a commit that referenced this pull request Oct 19, 2016
* Document how to configure ingest node to have geoip

* Resolve comments from review
leweafan pushed a commit to leweafan/beats that referenced this pull request Apr 28, 2023
…lastic#2807)

* Document how to configure ingest node to have geoip

* Resolve comments from review
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants