in_exec json streams does not work as expected

[serbaut]

ruby-yajl reads 8k from the input stream before parsing so input is delayed until the buffer is full (which can take forever if no new input is generated).

This can be reproduced with the following config:

<source>
  type exec
  format tsv
  command while sleep 1; do date; done
  keys tsv_date
  tag all
</source>

<source>
  type exec
  format json
  command while sleep 1; do echo '{"json_date": "'$(date)'"}'; done
  tag all
</source>

<match all>
  type stdout
</match>

tsv_date is emitted each time but json_date is not emitted until the buffer is full (~5 minutes in this case).

I have created a workaround serbaut@c99bcf4 that parses line by line but in the end ruby-yajl has to be fixed so is parses one json document at a time.

[repeatedly]

You are right. We wrote this note for out_exec_filter but we forgot to add same note to in_exec.
I will update the article later.

http://docs.fluentd.org/articles/out_exec_filter

[repeatedly]

I have already added json note to in_exec article.

[tagomoris]

It's documentation issue which is already solved.

[matt-deboer]

Could you clarify the resolution for this? I'm still unclear how to resolve this for the streaming json example in v 0.14.13 (other than to modify the fluentd core as suggested above)...