Allow users to customize Client Connection

[nmengin]

Signed-off-by: nmengin nicolas@containo.us

- What I did

Adding an option to allow users to give DialOptions to the ConnectionBroker and raft.Transport gRPC client connection.

- How I did it

By creating an option into the node.Node Configuration.

- How to test it

I modified the test to add DialOption.

- Description for the changelog

Adding an entry into the Node configuration to allow users to customize the DialOptions for the ConnectionBroker and raft.Transport gRPC client connection.

Fixes #2733

[nmengin]

I guess the CI is down because of the problem described in #2728 .

[vdemeester]

cc @MagnusS @mefyl @dperny @anshulpundir

[olljanat]

@nmengin is this one still valid? Can you rebase?

[nmengin]

Hello @olljanat , yes the PR is still valid.
PR rebased and tested locally.

I hope the CI will validate it 😄

[codecov]

Codecov Report

❗ No coverage uploaded for pull request base (master@08d6e4a). Click here to learn what that means.
The diff coverage is 95.83%.

@@            Coverage Diff            @@
##             master    #2734   +/-   ##
=========================================
  Coverage          ?   62.02%           
=========================================
  Files             ?      137           
  Lines             ?    22136           
  Branches          ?        0           
=========================================
  Hits              ?    13729           
  Misses            ?     6930           
  Partials          ?     1477

[olljanat]

@dperny @wk8 PTAL

[nmengin]

Hello @olljanat , @dperny , @wk8

The CI failed since I rebased my branch. I haven't reproduced the problem locally.
Do you have any idea of the problem?

I can help to analyze if necessary.

[olljanat]

I think that it is flaky test #2559

[nmengin]

Ok @olljanat , thank you for the response.

Do you think it's possible to relaunch the CI or do you prefer I make a push?

[thaJeztah]

I kicked the build

[nmengin]

Thank you @thaJeztah .

For information, do yo think this PR might be merged soon or there is a design problem with it?

[olljanat]

@nmengin did I understood right that you actually want give user possibility to workaround #2733 ? If that is case I guess that this will not get approval from maintainers.

Related issues:

• Unhealthy cluster: grpc: received message larger than max (4246471 vs. XXXX) moby#37941
• docker service ls | Error response from daemon: rpc error: code = ResourceExhausted desc = grpc: received message larger than max (5097134 vs. 4194304) moby#37997

and afaik most technical discussion are done on #2774

[nmengin]

Hello @olljanat .

Unfortunately the workaround you proposed did not work.
Indeed, when I create a huge number of Swarmkit objects, the nodes send the delta between the old configuration and the new one to the managers.
But the broker refuses to receive these messages because the maximum size for the messages it can received is the default one (4194304 byte) and the nodes send bigger messages.

I guess this PR will not be merged, so I close it.
But I just opened another (smaller) one to fix the initial issue #2869, don't hesitate to take a look if you want.