Allow pickle protocol to be overridden

[jakirkham]

This provides the option to override the pickle protocol used. The default behavior is still to use the HIGHEST_PROTOCOL. However this provides users an easy way to customize the pickle protocol through the distributed.yaml config.

cc @mrocklin @quasiben

[jakirkham]

Just placed this at the global level as it is not really worker or scheduler specific. Not picky on where it lives though.

[mrocklin]

Interesting. I'm curious, what was the motivation?

[jakirkham]

Issue ( #4011 ) encouraged me to put it in a PR. Albeit that issue is discussing a different solution. Could also be useful for debugging/helping users.

[mrocklin]

Thanks for the speedy response to that issue. If we were to go this approach (which seems sensible regardless) I would then ask the question

what should we make the default? 4 or 5?

I suspect that your answer is likely "5, because it's faster" while I probably tend towards "4, because it's robust". I'm curious, did we ever speed test pickle protocol 5 on a real-world workload? Would something like x + x.T be made considerably faster by it? Given the cost around Python version mismatches it might be worth verifying that the performance benefit is felt in real-world situations

[jakirkham]

Closing since this seems to be addressed by PR ( #4019 ).

[mrocklin]

Woot. Thanks for the early effort @jakirkham