Excessive ChannelUpdateReceived messages

[n1bor]

This is more of a general scaling issue that a specific Eclair one.

I have metrics on and I get about 1 channel update per second. There are 25k channels and each has 2 sides to this is about 2-4 updates per channel per day. This seems reasonable number.

However the router gets many of these from every peer we are connected to! So since I have about 150 peers this means that we receive about 100 times every update. So getting over 100 updates a second.

The current algorithm of just batching into 1 min chucks does not seem to be smart enough to scale - especially as this is an O(2) issue in that if the network doubles in size you will end up with 2x more peers each sending you 2x more data so 4x more...

Just a place for people to make any comments..

19:37:57.133 INFO   fr.acinq.eclair.metrics - type=METER, name=eventstream.ChannelUpdateReceived, count=157676, mean_rate=1.0015846067128484, m1=0.875834680682182, m5=2.463041605474114, m15=2.607360069071711, rate_unit=events/second
...
19:37:57.135 INFO   fr.acinq.eclair.metrics - type=METER, name=router.updates.in, count=18514908, mean_rate=117.60911848352433, m1=28.57854678346341, m5=103.17746415287304, m15=219.90509295098488, rate_unit=events/second
...
19:37:57.135 INFO   fr.acinq.eclair.metrics - type=GAUGE, name=stats.localchannels, value=261

[pm47]

This is a spec-related issue and will be addressed by bitcoin-like inventory messages.

[n1bor]

@pm47 - not sure inv process would be totally optimal.
As once we have the full view of the network should be able to tell most nodes not to tell us about updates from most channels - as we know we are getting updates from nodes that are nearer.

Say we have 1,000 peers we are connected to. And there are 100,000 channels on the network. We would compute which were the nearest 10 peers to each of the 100,000 channels - and then tell the other 990 nodes not to send us any updates on that channel - as we know we will get the updates from one of the 10 nearer peers anyway.
This would reduce the update traffic by a factor of 100 and scale much better with network size.
Obviously when a new channel came on line we would initially get any updates from all 1000 nodes - but would then after the initial set inform 990 nodes that we do not want any more updates on the new channel.
Would require each node to maintain for each peer a list of channels they ARE sending updates to (as will be smaller list than the channels they are not).