IP2Location Fluentd Filter Plugin

fluent-plugin-ip2location is a Fluentd filter plugin that enriches Fluentd events with IP geolocation data from an IP2Location BIN database.

It reads an IP address from a configured record field, queries the IP2Location Ruby library against an IP2Location BIN database, and writes selected lookup fields back into the event record.

Features

• Reads IP address from a top-level or nested record field, for example ip, remote_addr, or $.client.ip.
• Supports nested output fields, for example ip2location or $.enrichment.ip2location.
• Supports IP2Location geolocation fields including country, region, city, latitude, longitude, ISP, domain, ZIP code, time zone, usage type, ASN and AS name.
• Skips invalid IP addresses and private/local IP addresses by default.
• Supports configurable field selection.
• Supports an in-memory lookup cache.

Installation

Install it into the Ruby environment used by Fluentd:

fluent-gem install fluent-plugin-ip2location

Or add it to your Gemfile:

gem "fluent-plugin-ip2location"

Then run:

bundle install

IP2Location database

This plugin requires an IP2Location BIN database file. For DB26, use a file such as:

IPV6-COUNTRY-REGION-CITY-LATITUDE-LONGITUDE-ZIPCODE-TIMEZONE-ISP-DOMAIN-NETSPEED-AREACODE-WEATHER-MOBILE-ELEVATION-USAGETYPE-ADDRESSTYPE-CATEGORY-DISTRICT-ASN.BIN

Place the BIN file somewhere readable by the Fluentd process, for example:

/opt/ip2location/IPV6-COUNTRY-REGION-CITY-LATITUDE-LONGITUDE-ZIPCODE-TIMEZONE-ISP-DOMAIN-NETSPEED-AREACODE-WEATHER-MOBILE-ELEVATION-USAGETYPE-ADDRESSTYPE-CATEGORY-DISTRICT-ASN.BIN

Fluentd configuration

<filter app.**>
  @type ip2location
  database /opt/ip2location/IPV6-COUNTRY-REGION-CITY-LATITUDE-LONGITUDE-ZIPCODE-TIMEZONE-ISP-DOMAIN-NETSPEED-AREACODE-WEATHER-MOBILE-ELEVATION-USAGETYPE-ADDRESSTYPE-CATEGORY-DISTRICT-ASN.BIN
  ip_field $.client.ip
  output_field $.ip2location
  fields country_short,country_long,region,city,latitude,longitude,isp,domain,zipcode,timezone,usagetype,asn,as
  skip_private_ip true
  skip_invalid_ip true
  cache_size 10000
  on_error warn
</filter>

Input record

{
  "client": {
    "ip": "8.8.8.8"
  },
  "message": "hello"
}

Output record

{
  "client": {
    "ip": "8.8.8.8"
  },
  "message": "hello",
  "ip2location": {
    "country_short": "US",
    "country_long": "United States of America",
    "region": "California",
    "city": "Mountain View",
    "latitude": 37.40599,
    "longitude": -122.078514,
    "isp": "Google LLC",
    "zipcode": "94043",
    "timezone": "-08:00",
    "asn": "15169",
    "as": "Google LLC"
  }
}

Configuration parameters

Parameter	Required	Default	Description
database	Yes	-	Absolute path to the IP2Location BIN database file.
ip_field	Yes	-	Record field containing the IP address. Supports Fluentd record accessor syntax, for example ip, remote_addr, or $.client.ip.
output_field	No	ip2location	Destination field for enrichment output. Supports record accessor syntax. Ignored when merge_record true.
fields	No	all	Comma-separated list of IP2Location fields to include. Use all to include every supported field.
merge_record	No	false	When true, writes enrichment fields into the root record instead of a nested object.
prefix	No	ip2location_	Prefix used when merge_record true.
skip_invalid_ip	No	true	When true, invalid IP values are ignored. When false, invalid IP values raise an error.
skip_private_ip	No	true	When true, private, loopback and link-local IPs are not looked up.
include_unknown	No	false	When false, empty values and - values returned by IP2Location are omitted.
cache_size	No	10000	Maximum number of IP lookup results kept in memory. Set 0 to disable caching.
on_error	No	warn	Lookup error behavior. One of ignore, warn, or raise.

Supported fields

country_short,country_long,region,city,isp,latitude,longitude,domain,zipcode,timezone,netspeed,iddcode,areacode,weatherstationcode,weatherstationname,mcc,mnc,mobilebrand,elevation,usagetype,addresstype,category,district,asn,as,as_domain,as_usagetype,as_cidr

Merge fields into the root record

<filter app.**>
  @type ip2location
  database /opt/ip2location/IPV6-COUNTRY-REGION-CITY-LATITUDE-LONGITUDE-ZIPCODE-TIMEZONE-ISP-DOMAIN-NETSPEED-AREACODE-WEATHER-MOBILE-ELEVATION-USAGETYPE-ADDRESSTYPE-CATEGORY-DISTRICT-ASN.BIN
  ip_field remote_addr
  merge_record true
  prefix geoip_
  fields country_short,country_long,city,asn,as
</filter>

Output example:

{
  "remote_addr": "8.8.8.8",
  "geoip_country_short": "US",
  "geoip_country_long": "United States of America",
  "geoip_city": "Mountain View",
  "geoip_asn": "15169",
  "geoip_as": "Google LLC"
}

Development

Install dependencies:

bundle install

Run tests:

bundle exec rake

Build the gem locally:

gem build fluent-plugin-ip2location.gemspec

Install the local gem:

gem install fluent-plugin-ip2location-0.1.0.gem

Run the example configuration from the project directory:

bundle exec fluentd -c example/fluent.conf -p ./lib

Update the database path in example/fluent.conf before running it.

License

LICENSE.txt