netann: channel status manager by cfromknecht · Pull Request #2411 · lightningnetwork/lnd

cfromknecht · 2019-01-04T03:35:48Z

In this PR, we introduce a new subsystem called the netann.ChanStatusManager responsible for managing the announcement of channel updates which toggle a channel's disabled bit. Most of the logic has been moved into a new netann package, which offers greater unit testability of the subsystem's behavior.

The netann.ChanStatusManager exposes two methods:

func RequestEnable(outpoint wire.OutPoint) error {}
func RequestDisable(outpoint wire.OutPoint) error {}

Together, these allow other subsystems to request a toggle in a particular direction. The netann.ChanStatusManager then handles the task of dropping duplicate requests, and also ensuring that it's network behavior follows a well-defined state machine.

State Machine Description

At any point, the netann.ChanStatusManager categorizes known channels into three distinct ChanStatus states:

type ChanStatus uint8

const (
    // ChanStatusActive indicates that the channel's last announcement has
    // the disabled bit cleared.
    ChanStatusActive ChanStatus = iota

    // ChanStatusPendingInactive indicates that the channel's last
    // announcement has the disabled bit cleared, but that the channel was
    // detected in an inactive state. Channels in this state will have a
    // disabling announcement sent after the ChanInactiveTimeout expires
    // from the time of the first detection--unless the channel is explicitly
    // reenabled before the disabling occurs.
    ChanStatusPendingInactive

    // ChanStatusInactive indicates that the channel's last announcement has
    // the disabled bit set.
    ChanStatusInactive
)

Channels will always start in either ChanStatusActive or ChanStatusInactive on startup or after we detect a new outpoint (perhaps for a newly created channel) by examining the last channel update we have on disk.

One key distinction from the existing design is that the netann.ChanStatusManager only uses long polling to detect inactive channels, meaning that channels can ONLY be reenabled via an explict call to RequestEnable for an outpoint. This allows us to use a more accurate metric for enabling channels: connection duration.

Once a channel has been detected as inactive within the switch, presumably because of a disconnection, this kicks off a timed progression from ChanStatusActive -> ChanStatusPendingInactive -> ChanStatusInactive.

If RequestEnable is not received before the progression terminates, a new announcement setting the disable bit will be broadcast to the network.

Otherwise, any call to RequestEnable before the progression finishes will cancel the disable from being sent, and leaves the channel in the its still-enabled state on the network. This allows the netann.ChanStatusManager to tolerate short-lived reconnections, without causing the node to spam the network with channel updates.

Using the exposed configurations, users can tune this to a specific threshold, e.g. require ~95% uptime over a 20 minute interval, in order for the channel to remain enabled.

Supersedes #2080

Depends on:

halseth · 2019-01-08T14:35:02Z

Apart from the addition of tests, I'm struggling to see what this adds that is not already covered by #2080. How is this logic different than taking what we already have in #2080, making sure we only send updates if they alter the disable bit already in the DB?

cfromknecht · 2019-01-09T21:21:13Z

Apart from the addition of tests, I'm struggling to see what this adds that is not already covered by #2080.

To me this is the most crucial part, as we don't have much testing on the server or insight into the existing behavior apart from the integration tests.

Apart from that, the primary difference is that the state machine only performs the passive enable -> disable transition, while the current one will perform the transition in both directions. My hunch is that this is the primary thing leading to the instability we see on the network, and would probably have strange interactions with #2080 if left unchanged.

Another difference is that all state changes are serialized via the ChanStatusManagers main event loop, and are kept in sync with the on-disk state. Right now external commands can modify the persistent view of the graph separately from watchChannelStatus's view of each channel's current state.

With those things in mind, the goals in my mind are to reduce the state space, consolidate the logic into one unit, and provide a way of throughly testing it. It's also a good chance to work on our server bloat :)

cfromknecht · 2019-01-12T01:38:20Z

Testing framework has been significantly reworked, and additional state machine tests for the ChanStatusManager have been finished. The rework now brings just over 80% coverage to the netann package :D

cfromknecht · 2019-02-01T00:26:18Z

@halseth @wpaulino rebased on master and squashed prior fixups. also addressed @wpaulino's recent comments, PTAL

wpaulino · 2019-02-01T20:56:25Z

@cfromknecht looks like the latest version introduced a test failure:

    --- FAIL: TestLightningNetworkDaemon/update_channel_policy (7.19s)
        lnd_test.go:75: Failed: (update channel policy): exited with error: 
            *errors.errorString unable to send payment: received payment error: unable to find a path to destination
            /home/travis/gopath/src/github.com/lightningnetwork/lnd/lnd_test.go:1489 (0xbe2523)
            	testUpdateChannelPolicy: t.Fatalf("unable to send payment: %v", err)
            /home/travis/gopath/src/github.com/lightningnetwork/lnd/lnd_test.go:100 (0xbdac66)
            	(*harnessTest).RunTestCase: testCase.test(net, h)
            /home/travis/gopath/src/github.com/lightningnetwork/lnd/lnd_test.go:13282 (0xc39f74)
            	TestLightningNetworkDaemon.func4: ht.RunTestCase(testCase, lndHarness)
            /home/travis/.gimme/versions/go1.11.5.linux.amd64/src/testing/testing.go:827 (0x4f8caf)
            	tRunner: fn(t)
            /home/travis/.gimme/versions/go1.11.5.linux.amd64/src/runtime/asm_amd64.s:1333 (0x45dd91)
            	goexit: BYTE	$0x90	// NOP

cfromknecht · 2019-02-02T00:54:06Z

@wpaulino indeed still looking into it

Roasbeef

Running latest version on the faucet now, will observe le logs to see if anything peculiar happens over the next day or two.

halseth

Mostly just nits on code and commit structure at this point, looks really good 👮

halseth · 2019-02-13T21:19:07Z

It just bloats the exposed config, and seems like all three could be derived from just setting the longest one, even during integration tests.

cfromknecht · 2019-02-14T07:24:23Z

@wpaulino @halseth @Roasbeef comments addressed, commits restructured, and additional unit tests added. See this force-push for the majority of the changes, ptal

wpaulino

Clean code with excellent test coverage. LGTM 💥

Exposes the three parameters that dictate the behavior of the channel status manager: * --chan-enable-timeout * --chan-disable-timeout * --chan-status-sample-interval

Found that his can sometimes cause a panic with a negative waitgroup counter.

This commit hooks up the new netann.ChanStatusManager, replacing the prior method which used the watchChannelStatus goroutine.

cfromknecht · 2019-02-15T01:18:08Z

@wpaulino comments addressed, @halseth @Roasbeef ptal

Roasbeef

LGTM 🦄

I think we'll be saving the entire network a good bit of bandwidth once this lands in 0.6 ;)

Roasbeef · 2019-02-19T01:00:17Z

+	// in the map. If for some reason the channel isn't closed, the state
+	// will be repopulated on subsequent calls to RequestEnable or
+	// RequestDisable via a db lookup, or on startup.
+	delete(m.chanStates, outpoint)


👍

I think I ran into this issue in a prior diff of this PR. It appears to be fixed now.

Roasbeef · 2019-02-19T01:03:07Z

+	for _, c := range allChannels {
+		// We'll skip any private channels, as they aren't used for
+		// routing within the network by other nodes.
+		if c.ChannelFlags&lnwire.FFAnnounceChannel == 0 {


Not a blocker, but this could have been hidden behind this new interface. With that path, then the ChanStatusManager doesn't need to know how to interpret the bits of the channel flag, or what an advertised channel even is.

Roasbeef · 2019-02-19T01:09:27Z

 	// channel updates for channels going inactive.
-	eve, err := net.NewNode("Eve", []string{"--chan-disable-timeout=2s"})
+	eve, err := net.NewNode("Eve", []string{
+		"--minbackoff=10s",


I have a feeling that we'll need to adjust these values due to Travis flakiness in the future...

Roasbeef · 2019-02-19T01:15:12Z

 			// wait for close tx conf.
 			p.finalizeChanClosure(chanCloser)
+
+		// The channel reannounce delay has elapsed, broadcast the


Nice, I think this'll probably make the biggest difference in our current handling, so we don't immediately enable and span the network due to a flappy peer.

cfromknecht force-pushed the chan-status-manager branch from 91c8911 to 6b2de75 Compare January 4, 2019 03:47

This was referenced Jan 4, 2019

netann: move NodeSigner out of main package #2416

Merged

server: configurable min backoff #2417

Merged

cfromknecht force-pushed the chan-status-manager branch 3 times, most recently from 87828fd to 29f32c8 Compare January 5, 2019 00:27

cfromknecht added channels gossip labels Jan 5, 2019

cfromknecht force-pushed the chan-status-manager branch 2 times, most recently from 56edc99 to 0d5e1e4 Compare January 5, 2019 01:35

Roasbeef requested review from halseth and wpaulino January 8, 2019 01:29

cfromknecht added the blocked label Jan 8, 2019

cfromknecht force-pushed the chan-status-manager branch from c5b1c6d to 5b5c09b Compare January 9, 2019 21:28

cfromknecht mentioned this pull request Jan 9, 2019

netann: Add NANN subsystem logger #2445

Merged

cfromknecht force-pushed the chan-status-manager branch from 5b5c09b to 10a0050 Compare January 10, 2019 01:03

cfromknecht mentioned this pull request Jan 11, 2019

htlcswitch: stricter HasActiveLink #2452

Merged

cfromknecht force-pushed the chan-status-manager branch 9 times, most recently from 245b180 to 405081e Compare January 12, 2019 01:10

cfromknecht removed the blocked label Jan 15, 2019

cfromknecht force-pushed the chan-status-manager branch from 89b5452 to b76ba69 Compare February 1, 2019 00:24

cfromknecht force-pushed the chan-status-manager branch from b76ba69 to 2df928e Compare February 1, 2019 03:03

cfromknecht force-pushed the chan-status-manager branch from 2df928e to a3c2820 Compare February 7, 2019 09:49

cfromknecht commented Feb 7, 2019

View reviewed changes

Comment thread netann/common.go Outdated

Roasbeef reviewed Feb 9, 2019

View reviewed changes

Comment thread netann/common.go Outdated

halseth suggested changes Feb 13, 2019

View reviewed changes

cfromknecht force-pushed the chan-status-manager branch 3 times, most recently from 960c216 to d213d6d Compare February 14, 2019 06:55

wpaulino approved these changes Feb 15, 2019

View reviewed changes

Comment thread netann/channel_update.go Outdated

Comment thread netann/channel_update.go Outdated

Comment thread netann/chan_status_manager.go Outdated

cfromknecht added 10 commits February 14, 2019 17:12

netann/channel_update: exposes channel update helpers moved from server

1a7fd6e

netann/channel_update_test: adds unit tests UpdateDisableFlag

a0daf6f

netann/interface: defines chan status mgr interfaces

559255e

netann/channel_state: defines ChanStatus and per-channel state

b405c87

netann/chan_status_manager: adds ChanStatusManager

53797b7

netann/chan_status_manager_test: adds ChanStatusManager unit tests

cd46719

config+server+lnd_test: expose chan status manager config options

4686db4

Exposes the three parameters that dictate the behavior of the channel status manager: * --chan-enable-timeout * --chan-disable-timeout * --chan-status-sample-interval

lnd_test: set valid chan status config flags

ef41fc6

test_utils: waitgroup manually spawned goroutines

8b185e6

Found that his can sometimes cause a panic with a negative waitgroup counter.

peer+server+test_utlils: use new ChanStatusManager

de28217

This commit hooks up the new netann.ChanStatusManager, replacing the prior method which used the watchChannelStatus goroutine.

cfromknecht force-pushed the chan-status-manager branch from d213d6d to de28217 Compare February 15, 2019 01:17

Roasbeef approved these changes Feb 19, 2019

View reviewed changes

Roasbeef merged commit c8b5e1f into lightningnetwork:master Feb 19, 2019

cfromknecht deleted the chan-status-manager branch February 19, 2019 20:43

cfromknecht mentioned this pull request Feb 19, 2019

peer: send reenable updates after 20m #2080

Closed

Conversation

cfromknecht commented Jan 4, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

State Machine Description

Uh oh!

halseth commented Jan 8, 2019

Uh oh!

cfromknecht commented Jan 9, 2019

Uh oh!

cfromknecht commented Jan 12, 2019

Uh oh!

cfromknecht commented Feb 1, 2019

Uh oh!

wpaulino commented Feb 1, 2019

Uh oh!

cfromknecht commented Feb 2, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Roasbeef left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

halseth left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

halseth Feb 13, 2019

Choose a reason for hiding this comment

Uh oh!

Uh oh!

cfromknecht commented Feb 14, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

wpaulino left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

cfromknecht commented Feb 15, 2019

Uh oh!

Roasbeef left a comment

Choose a reason for hiding this comment

Uh oh!

Roasbeef Feb 19, 2019

Choose a reason for hiding this comment

Uh oh!

Roasbeef Feb 19, 2019

Choose a reason for hiding this comment

Uh oh!

Roasbeef Feb 19, 2019

Choose a reason for hiding this comment

Uh oh!

Roasbeef Feb 19, 2019

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

cfromknecht commented Jan 4, 2019 •

edited

Loading

cfromknecht commented Feb 2, 2019 •

edited

Loading

cfromknecht commented Feb 14, 2019 •

edited

Loading