Elasticsearch partial bulk failures

[binarylogic]

I would like to think about partial failures when ingesting data into Elasticsearch and if we even want to handle this scenario. Options include:

1. Don't do anything.
2. Collect individual failed items and retry. This will probably require error code matching.
3. Provide an option to dead-letter the items.

This is not urgent but I wanted to get it up for discussion and posterity.

[lukesteensen]

I spent some time on this as part of #139, and I think to do it right will require some rework of the util HttpSink. Right now, we don't have a good way to parse the body of an http response and inspect it in our retry logic. One option would be to introduce some kind of response mapper we pass in. Another would be to strip down the inner sink, just using it as a service and pulling up things like timeouts and retries to the level of the outer sinks. The latter would mesh well with making those things configurable per-sink, so that might be the most promising route to explore.

[binarylogic]

Another would be to strip down the inner sink, just using it as a service and pulling up things like timeouts and retries to the level of the outer sinks.

I like that and agree

To add more context, this is definitely a later version feature and I don't think we should get cute with it. The other issue is that the response body can be very large depending on how much data was flushed, which can also cause performance and memory issues. I think a good first step is to check the root level "errors" key and then just retry the entire request. To prevent duplicate data, users can generate their own deterministic IDs which ES will reject on a duplicate requests.