Skip to content

Latest commit

 

History

History
311 lines (252 loc) · 12.4 KB

README.md

File metadata and controls

311 lines (252 loc) · 12.4 KB
Raw

locus

locus is library for Erlang/OTP and Elixir that allows you to pinpoint the country, city or ASN of IP addresses using MaxMind GeoIP2.

The free MaxMind databases you choose are loaded on-demand and, if using HTTP, cached on the filesystem and updated automatically.

⚠️ Starting on December 31st, 2019, a license key is now required to download MaxMind GeoLite2 databases.

The existing URLs have been discontinued; you should upgrade locus to version 1.9.0 or higher if you used them.

Usage

1. Configure your license key

Get a free license key from MaxMind if you haven't one already. Once logged in, you'll find the page to generate it on the left menu, under "My License Key".

Then clone the repository, run make shell and declare your key:

application:set_env(locus, license_key, "YOUR_LICENSE_KEY").
2. Start the database loader
ok = locus:start_loader(country, 'GeoLite2-Country').
% You can also use a HTTP URL or a local path, e.g. "/usr/share/GeoIP/GeoLite2-City.mmdb"
3. Wait for the database to load (optional)
{ok, _DatabaseVersion} = locus:await_loader(country). % or `{error, Reason}'
4. Look up IP addresses
% > locus:lookup(country, "93.184.216.34").
% > locus:lookup(country, "2606:2800:220:1:248:1893:25c8:1946").

{ok,#{prefix => {{93,184,216,0},31},
      <<"continent">> =>
          #{<<"code">> => <<"NA">>,
            <<"geoname_id">> => 6255149,
            <<"names">> =>
                #{<<"de">> => <<"Nordamerika">>,
                  <<"en">> => <<"North America">>,
                  <<"es">> => <<"Norteamérica"/utf8>>,
                  <<"fr">> => <<"Amérique du Nord"/utf8>>,
                  <<"ja">> => <<"北アメリカ"/utf8>>,
                  <<"pt-BR">> => <<"América do Norte"/utf8>>,
                  <<"ru">> => <<"Северная Америка"/utf8>>,
                  <<"zh-CN">> => <<"北美洲"/utf8>>}},
      <<"country">> =>
          #{<<"geoname_id">> => 6252001,
            <<"iso_code">> => <<"US">>,
            <<"names">> =>
                #{<<"de">> => <<"USA">>,
                  <<"en">> => <<"United States">>,
                  <<"es">> => <<"Estados Unidos">>,
                  <<"fr">> => <<"États-Unis"/utf8>>,
                  <<"ja">> => <<"アメリカ合衆国"/utf8>>,
                  <<"pt-BR">> => <<"Estados Unidos">>,
                  <<"ru">> => <<"США"/utf8>>,
                  <<"zh-CN">> => <<"美国"/utf8>>}},
      <<"registered_country">> =>
          #{<<"geoname_id">> => 6252001,
            <<"iso_code">> => <<"US">>,
            <<"names">> =>
                #{<<"de">> => <<"USA">>,
                  <<"en">> => <<"United States">>,
                  <<"es">> => <<"Estados Unidos">>,
                  <<"fr">> => <<"États-Unis"/utf8>>,
                  <<"ja">> => <<"アメリカ合衆国"/utf8>>,
                  <<"pt-BR">> => <<"Estados Unidos">>,
                  <<"ru">> => <<"США"/utf8>>,
                  <<"zh-CN">> => <<"美国"/utf8>>}}}}

Documentation

  1. Supported File Formats
  2. Database Types and Loading
  3. Database Validation
  4. MaxMind sources / HTTP URLs: Downloading and Updating
  5. MaxMind sources / HTTP URLs: Caching
  6. Filesystem URLs: Loading and Updating
  7. Logging
  8. Event Subscriptions
  9. API Reference
  10. Tested Setup
  11. License
  12. Alternative Libraries (Erlang)
  13. Alternative Libraries (Elixir)
Supported File Formats
  • gzip-compressed tarballs (.tar.gz, .tgz)
  • plain tarballs (.tar)
  • MMDB files (.mmdb)
  • gzip-compressed MMDB files (.mmdb.gz)

For tarball files, the first file to be found within it with an .mmdb extension is the one that's chosen for loading.

The implementation of MaxMind DB format is mostly complete.

Database Types and Loading
  • The free GeoLite2 Country, City and ASN databases were all successfully tested; presumably locus can deal with any MaxMind DB 2.x database that maps IP address prefixes to arbitrary data, but no commercial databases have yet been tested
  • The databases are loaded into memory (mostly) as is; reference counted binaries are shared with the application callers using ETS tables, and the original binary search tree is used to lookup addresses. The data for each entry is decoded on the fly upon successful lookups.
Database Validation

Databases, local or remote, can have their compatibility validated through the locus:analyze/1 function after they've been loaded (see function reference.)

Alternatively, they can also be checked from the command line by use of the locus CLI utility:

  1. Run make cli to build the script, named locus, which will be deployed to the current directory.

  2. Run analysis:

    ./locus analyze GeoLite2-City.mmdb
# Loading database from "GeoLite2-City.mmdb"...
# Database version {{2019,11,6},{11,58,0}} successfully loaded
# Analyzing database for flaws...
# Database is wholesome.

The script will exit with code 1 in case of failure, and 0 otherwise. Run ./locus analyze --help for a description of supported options and arguments.

MaxMind sources / HTTP URLs: Downloading and Updating
  • The downloaded database files, when compressed, are inflated in memory
  • The last-modified response header, if present, is used to condition subsequent download attempts (using if-modified-since request headers) in order to save bandwidth
  • The downloaded databases are cached on the filesystem in order to more quickly achieve readiness on future launches of the database loader
  • Until a MaxMind or HTTP URL database loader achieves readiness, download attempts are made every minute; once readiness is achieved (either from cache or network), this interval increases to every 6 hours. These can be tweaked using the pre_readiness_update_period and post_readiness_update_period loader settings (in milliseconds.)
  • When downloading from a MaxMind edition or HTTPS URL, the remote certificate will be authenticated against a list of known Certificate Authorities and connection negotiation will fail in case of an expired certificate, mismatched hostname, self-signed certificate or unknown certificate authority. These checks can be disabled by specifying the insecure loader option.
MaxMind sources / HTTP URLs: Caching
  • Caching is a best effort; the system falls back to relying exclusively on the network if needed
  • A caching directory named locus_erlang is created under the 'user_cache' basedir
  • Cached databases are named after the MaxMind database edition name, or alternatively after the SHA256 hash of their source URL
  • Modification time of the databases is extracted from last-modified response header (when present) and used to condition downloads on subsequent boots and save bandwidth
  • Caching can be disabled by specifying the no_cache option when running :start_loader
Filesystem URLs: Loading and Updating
  • The loaded databases, when compressed, are inflated in memory
  • Until a filesystem database loader achieves readiness, load attempts are made every 5 seconds; once readiness is achieved, this interval increases to every 30 seconds and load attempts are dismissed as long as the database file modification timestamp keeps unchanged
Logging
  • Five logging levels are supported: debug, info, warning, error and none
  • The chosen backend on OTP 21.1+ is logger if lager is either missing or it hasn't removed logger's default handler; for all other scenarios, error_logger is picked instead
  • The default log level is error; it can be changed in the application's env config
  • To tweak the log level in runtime, use locus_logger:set_loglevel/1
Event Subscriptions
  • Any number of event subscribers can be attached to a database loader by specifying the {event_subscriber, Subscriber} option when starting the database
  • A Subscriber can be either a module implementing the locus_event_subscriber behaviour or an arbitrary pid()
  • The format and content of reported events can be consulted in detail on the locus_event_subscriber module documentation; most key steps in the loader pipeline are reported (download started, download succeeded, download failed, caching succeeded, loading failed, etc.)
API Reference

The API reference can be found on HexDocs.

Tested setup
  • Erlang/OTP 19 or newer
  • rebar3
License

MIT License

Copyright (c) 2017-2020 Guilherme Andrade

Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions:

The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.

THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.

locus is an independent project and has not been authorized, sponsored, or otherwise approved by MaxMind.

Alternative Libraries (Erlang)
  • egeoip: IP Geolocation module, currently supporting the MaxMind GeoLite City Database
  • geodata2: Application for working with MaxMind geoip2 (.mmdb) databases
  • geoip: Returns the location of an IP address; based on the ipinfodb.com web service
  • geolite2data: Periodically fetches the free MaxMind GeoLite2 databases
  • ip2location-erlang: Uses IP2Location geolocation database
Alternative Libraries (Elixir)
  • asn: IP-to-AS-to-ASname lookup
  • freegeoip: Simple wrapper for freegeoip.net HTTP API
  • freegeoipx: API Client for freegeoip.net
  • geoip: Lookup the geo location for a given IP address, hostname or Plug.Conn instance
  • geolix: MaxMind GeoIP2 database reader/decoder
  • plug_geoip2: Adds geo location to a Plug connection based upon the client IP address by using MaxMind's GeoIP2 database
  • tz_world: Resolve timezones from a location efficiently using PostGIS and Ecto

Generated by EDoc