Commit 27068fc6 authored by Ivan Tyagov's avatar Ivan Tyagov

Add an example of MSGPack transmission to Wendelin from a file.

parent 2b1c718b
......@@ -8,42 +8,44 @@
port 8888
</source>
# output tag=sensor.** to Wendelin, this will match all HTTP request like so
# This will match all HTTP request like so
# ivan@debian: ~$ curl -X POST -d 'json={"foo1":"bar1"}' http://localhost:8888/default_http_json
# then send to Wendelin and append as strings at Wendelin side
<match default_http_json.**>
@type wendelin
@type wendelin
@id wendelin_out
streamtool_uri <Wendelin_URL>/erp5/portal_ingestion_policies/default_http_json
# TODO ^^^ do not check peer's certificate
user <your_wendelin_user>
password <your_wendelin_password>
user <your_wendelin_user>
password <your_wendelin_password>
use_keep_alive true
buffer_type memory
flush_interval 5s
use_keep_alive true
buffer_type memory
flush_interval 5s
</match>
# tail existing file and send its contents to Wendelin
<source>
@type tail
path test.log
pos_file test.log.pos
tag fluentd-1
@type tail
path test.log
pos_file test.log.pos
tag test_sensor.test_product
<parse>
@type none
</parse>
</source>
<match fluentd-1>
@type wendelin
@id wendelin_out
streamtool_uri <Wendelin_URL>/erp5/portal_ingestion_policies/default_http_json
# This will send and streo at Wendelin side in a MsgPack format
  • @Tyagov Please note that it is not the ingestion policy which decides how to store incoming data in wendelin (e.g. when sending to portal_ingestion_policies/default then data can be stored in msgpack or in unpacked string).

    The script defined in ingestion policy only decides how to parse the tag to find a data ingestion or data supply

    It is the Data Operation Script which is defined in data ingestion / data supply which decides how it wants to store data.

    Therefore the current comment is a bit misleading.

  • @klaus , yes sorry for this.

    Here's what I plan to fix as current state (pure Wendelin side) is too confusing:

    Overall goal: 1 "default" ingestion policy which works without any changes needed Wendelin side right now. Based on outcome of storage tests it will either be MSgPack or "append as string" - but let us see (doing it now)

  • @Tyagov in this case, I suggest that this work is done using merge request so it can be discussed before being comitted to master.

    My current thinking is that there can be one "default" data ingestion policy, but there must be several data supplies preinstalled and in the case of fluentd there must not be a single default data supply, because the choice depends how you acquire data (log file, execute command, ...), the type of data (time series or not) and the original data format (json, binary, ...)

Please register or sign in to reply
<match test_sensor.test_product>
@type wendelin
@id fluentd_out
streamtool_uri <Wendelin_URL>/erp5/portal_ingestion_policies/default
# TODO ^^^ do not check peer's certificate
user <your_wendelin_user>
password <your_wendelin_password>
user <your_wendelin_user>
password <your_wendelin_password>
use_keep_alive true
buffer_type memory
flush_interval 5s
use_keep_alive true
buffer_type memory
flush_interval 5s
</match>
Markdown is supported
0% or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment