add options for boltdb_index_client to avoid dead loop

[mizeng]

As part of solution for grafana/loki#685, the purpose is to avoid opening boltdb fall into dead loop.

[bboreham]

It seems that your issue is about accessing Boltdb from more than one process.
The Boltdb README says: "multiple processes cannot open the same database at the same time".

So I am not clear why a timeout will help.

[mizeng]

@bboreham The first failing is like you said, But what I'm proposed is to solve the next more incoming requests hang issue.
If one request in the process hang on BoltDB for any reason, it should not block any incoming requests rather than keep hanging. Customers need to get a fast failed instead of hanging. I think that's why this boltdb client has timeout parameter.
Using default parameter anyway looks like improper for fast failed.

Quote from the boltdb description in code for this parameter:

// Timeout is the amount of time to wait to obtain a file lock.
// When set to zero it will wait indefinitely. This option is only
// available on Darwin and Linux.

Also Boltdb README says: "To prevent an indefinite wait you can pass a timeout option to the Open() function:".

[bboreham]

So your aim is to fail every request from then on?

[mizeng]

yes. Fast fail looks better than indefinite wait. What do you think?

[bboreham]

Now I understand the intention, I don't think this value is worth adding as an option - just hard-code it and put a comment explaining the expected outcome.

[mizeng]

fair enough.

updated. could you plz help review and merge again?

[csmarchbanks]

Thanks!