README.md 1.87 KB
Newer Older
Klaus Wölfel's avatar
Klaus Wölfel committed
1
fluent-plugin-bin
Klaus Wölfel's avatar
Klaus Wölfel committed
2 3 4 5
====================

[Fluentd][] input plugin for binary files based on in_tail.

6 7
Usage Example
=============
Klaus Wölfel's avatar
Klaus Wölfel committed
8

9 10
This exmaple shows how to install and configure fluent-plugin-bin together
with fluent-plugin wendelin output plugin to ingest binary files into
Klaus Wölfel's avatar
Klaus Wölfel committed
11
[Wendelin][] on Ubuntu 14.04.4 LTS.
12

Klaus Wölfel's avatar
Klaus Wölfel committed
13 14 15 16 17 18 19 20 21 22 23 24
    sudo apt-get install supervisor
    sudo service supervisor restart
    sudo apt-get install python-software-properties
    sudo apt-add-repository ppa:brightbox/ruby-ng
    sudo apt-get update
    sudo apt-get install ruby2.0 ruby2.0-⁠⁠dev make
    sudo gem install fluentd --no-ri --no-rdoc
    sudo mkdir -p /etc/fluent/plugin
    cd /etc/fluent/plugin
    sudo wget https://lab.nexedi.com/nexedi/fluent-plugin-bin/raw/master/lib/fluent/plugin/in_bin.rb
    sudo wget https://lab.nexedi.cn/nexedi/fluent-plugin-wendelin/raw/master/lib/fluent/plugin/out_wendelin.rb
    sudo wget https://lab.nexedi.cn/nexedi/fluent-plugin-wendelin/raw/master/lib/fluent/plugin/wendelin_client.rb
25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50

Create /etc/fluent/fluentd.conf:

    <source>
      @type bin
      format none
      path /path/to/files/*.ext
      pos_file /mic/upload.pos
      enable_watch_timer false
      read_from_head true
      tag my_prefix.*
    </source>
    
    <match my_prefix.path.to.files.*.ext>
      @type wendelin
      @id wendelin_out

      streamtool_uri https://wendelin_url/portal_ingestion_policies/my_policy
      user      my_user_name
      password  my_password

      buffer_type       memory
      flush_interval    1s
      disable_retry_limit    true
    </match>

Klaus Wölfel's avatar
Klaus Wölfel committed
51
Create /etc/supervisor/conf.d/fluentd.conf:
Klaus Wölfel's avatar
Klaus Wölfel committed
52

Klaus Wölfel's avatar
Klaus Wölfel committed
53 54 55 56 57 58 59
    [program:fluentd]
    command=/usr/local/bin/fluentd
    autostart=true
    autorestart=true
    stderr_logfile=/var/log/fluentd.log
    stdout_logfile=/var/log/fluentd.log

Klaus Wölfel's avatar
Klaus Wölfel committed
60
Start fluentd:
Klaus Wölfel's avatar
Klaus Wölfel committed
61

Klaus Wölfel's avatar
Klaus Wölfel committed
62 63
    sudo supervisorctl start fluentd

64
[Fluentd]:  http://fluentd.org
Klaus Wölfel's avatar
Klaus Wölfel committed
65
[Wendelin]: http://www.wendelin.io