Make stackdriver agent work for detecting Json log

[yanweiguo]

For a json log emitted by user, for example,

{
  "log": "hello",
  "foo": "bar"
}

In fluentd event flows, before operator output configuration, we convert it to

{
  "log": "{\"log\": \"hello\", \"foo\": \"bar\"}",
  "foo": "bar",
  "kubernetes": {
      # kubernetes metadata
  }
  "stream": "stderr", # or "stdout" or "varlog"
  ...
}

For stderr/stdout, kubernetes_metadata plugin does the json parsing, which is a responsibility shouldn't belong to kubernetes_metadata. (This feature was removed since version kubernetes_metadata 2.1.0). For varlog, we parse the json in tail.

Then operator configure the destination backend and other extra process. This works fine for Elasticsearch, but it doesn't work for Stackdriver. Stackdriver will only try to parse log field as json if there're only the following fields: log(or message), time, severity, docker, kubernetes.

Proposed Changes

• Do not parse json before operator output configuration. For example, the log entry before operator configuration flows is:

{
  "log": "{\"log\": \"hello\", \"foo\": \"bar\"}",
  "kubernetes": {
      # kubernetes metadata
  }
  "stream": "stderr", # or "stdout" or "varlog"
  ...
}

• Let the operator to decide how they want to deal with the log field, which can be a json payload:
  • For ElasticSearch, use parser to parse the json.
  • For Stackdriver, turn on detect_json parameter in google_cloud plugin.

[rootfs]

what is this for?

[yanweiguo]

To indicate any random key-value pair in json log emitted by the app can be collected/processed.

[mdemirhan]

/lgtm
/approve

[vaikas]

/lgtm
/approve

[google-prow-robot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: mdemirhan, vaikas-google, yanweiguo

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details

Needs approval from an approver in each of these files:

• OWNERS [vaikas-google]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment