Fluentd output plugin

Fluentd is a data collector for unified logging layer, it can be configured with the Loki output plugin, provided in this folder, to ship logs to Loki.

See docs/client/fluentd/README.md for detailed information.

Development

After checking out the repo, run bin/setup to install dependencies. Then, run bin/test to run the tests. You can also run bin/console for an interactive prompt that will allow you to experiment.

To install this gem onto your local machine, run ruby -S bundle exec rake install. To release a new version, update the version number in fluent-plugin-grafana-loki.gemspec, and then run ruby -S bundle exec rake release, which will create a git tag for the version, push git commits and tags, and push the .gem file to rubygems.org.

To create the gem: ruby -S gem build fluent-plugin-grafana-loki.gemspec

Useful additions:

ruby -S gem install rubocop

Testing

Start Loki using:

docker run -it -p 3100:3100 grafana/loki:latest

Verify that Loki accept and stores logs:

curl -H "Content-Type: application/json" -XPOST -s "http://localhost:3100/loki/api/v1/push" --data-raw "{\"streams\": [{\"stream\": {\"job\": \"test\"}, \"values\": [[\"$(date +%s)000000000\", \"fizzbuzz\"]]}]}"
curl "http://localhost:3100/loki/api/v1/query_range" --data-urlencode 'query={job="test"}' --data-urlencode 'step=300' | jq .data.result

The expected output is:

[
  {
    "stream": {
      "job": "test"
    },
    "values": [
      [
        "1588337198000000000",
        "fizzbuzz"
      ]
    ]
  }
]

Start and send test logs with Fluentd using:

LOKI_URL=http://{{ IP }}:3100 make fluentd-test

Verify that syslogs are being feeded into Loki:

curl "http://localhost:3100/loki/api/v1/query_range" --data-urlencode 'query={job="fluentd"}' --data-urlencode 'step=300' | jq .data.result

The expected output is:

[
  {
    "stream": {
      "job": "fluentd"
    },
    "values": [
      [
        "1588336950379591919",
        "log=\"May  1 14:42:30 ibuprofen avahi-daemon[859]: New relevant interface vethb503225.IPv6 for mDNS.\""
      ],
      ...
    ]
  }
]

Build and publish gem

To build and publish a gem to rubygems you first need to update the version in the fluent-plugin-grafana-loki.gemspec file. Then update the VERSION variable in the Makefile to match the new version number. Create a PR with the changes against the main branch und run make fluentd-plugin-push from the root of the project once the PR has been merged.

  • Copyright(c) 2018- Grafana Labs
  • License
    • Apache License, Version 2.0