routing: add support for pegged hops in findPaths

[Bluetegu]

The PR addresses #2282

Edited (08-02-2019):

From the CLI queryroutes description:

A series of nodes and channels the route must pass through can optionally
be provided. The node id is mandatory. The channel leading to this node
can optionally be pegged too, otherwise the channel id should be set to 0.
The resulting route is composed of segments connecting between the pegged
nodes. Each segment does not have loops, but the entire path may. The
route selection assures that no edge will be transversed twice at the
same direction.
The alternate routes are stitched from N-1 shortest segment paths and
one alternate segment path.

Below is an example of a query using routing pegs:

lncli queryroutes --pegs='{"hop_pegs": [
            {
                "node_id": "026a52eadd42eabb70e9b0481af7001d299f7d7b41030977bb37a2faac7bbe0012",
                "chan_id": "2043992116101120"
            }
        ]
    }' "034a15c028970d4070c60f532018cfdb943b584ba267c83fe333ef1ad70b10feaf" 333

Note the single quote used to encapsulate the pegs.

Below is the documentation from the unit test added for this feature.
Feedback appreciated.

// TestPeggedRouting tests route selection with pegged
// nodes and channels. Pegging restrict routing to pass through the pegged
// node (and optionally through a channel) while searching for a route
// between source and destination.
// The resulting route is composed of segments connecting between the pegged
// nodes. Each segment does not have loops, but the entire path may. The
// route selection assures that no edge will be transversed twice at the
// same direction.
// The alternate routes are stitched from N-1 shortest segment paths and
// one alternate segment path.
//
//   ┌──────────┐        20         ┌──────────┐
//   │    E     │───────────────────│    D     │
//   └──────────┘        6          └──────────┘
//        |                           |      |
//        |                           |      | d/e
//        | 10                        | 50   | 20
//        | 3                         | 4    | 5
//        |                           |      |
//   ┌──────────┐        60         ┌──────────┐
//   │    B     │───────────────────│    C     │
//   └──────────┘        2          └──────────┘
//        |
//        |
//        | 30
//        | 1
//        |         ┌──────────┐
//        └─────────│     A    │
//                  └──────────┘
//
// The setup is build from symmetric channels whose base fee is
// marked and the fee rate is set to 0. The channel id between
// each node is marked below the rate.
//
// Test 1:
// Find routes from origin (A) to final destination (E) through D.
// Channel 5 disabled.
// A -> *D -> *E
//
// The expected routes are:
// r1: A -> B -> E -> D -> E                    total fees: 50
// r2: A -> B -> C -> D -> E  			total fees: 130
//
// Note that in r1, E is visited twice, before and after D. The
// pegging does not enforce order, i.e. it does not imply that D
// will be reached before E, rather that after D is reached, E
// will be reached.
//
// The following routes should not be returned:
// r1: A -> B -> E -> D -> E -> D -> E          total fees: 90
// r2: A -> B -> E -> D -> C -> B -> E
// reason: Edges cannot be transversed twice in the same direction.
// The following route is not returned:
// r1: A -> B -> C -> D -> C -> B -> E		total fees: 230
// Eventhough no edge is transversed twice at the same direction
// due to the route stitching selection. This route is similar to
// a route stiched from the 2nd best A -> D with the 2nd best
// D -> E.
//
// Test 2:
// Find routes from origin (A) back to itself through (C) and (E).
// Channel 5 disabled.
// A -> *C -> *E -> *A
//
// The expected routes are:
// r1: A -> B -> C -> B -> E -> B -> A          total fees: 170
// r2: A -> B -> C -> D -> E -> B -> A		total fees: 170
//
// Note that both routes have the same total fees.
//
// Test 3:
// Find routes back to origin (A) passign through node C through channel 4
// Channel 5 disabled.
// A -> (4)C -> *A
//
// The expected routes are:
// r1: A -> B -> E -> D -> C -> B -> A             total fees: 170
// r1: A -> B -> E -> D -> C -> D -> E -> B -> A   total fees: 190
// r3: A -> B -> C -> D -> C -> B -> A             total fees: 250
//
// Test 4:
// Find routes back to origin (A) passing through node D through
// channel 6 and then node C through channel 4
// Channel 5 disabled.
// A -> (6)D -> (4)C -> *A
//
// The expected routes are:
// A -> B -> E -> (6)D -> (4)C -> B -> A
// A -> B -> E -> (6)D -> (4)C -> D -> E -> B -> A
// A -> B -> C -> D -> E -> (6)D -> (4)C -> B -> A
//
// Test 5:
// Find routes back to origin (A) passing through node D through
// channel 6 and then node C through channel 4
// Channel 5 enabled.
// A -> (6)D -> (4)C -> (2)B -> A
//
// The expected routes are:
// A -> B -> E -> (6)D -> (4)C -> (2)B -> A
// A -> B -> C -> (5)D -> E -> (6)D -> (4)C -> (2)B -> A
// A -> B -> C -> (4)D -> E -> (6)D -> (4)C -> (2)B -> A
//

[joostjager]

I think this point is too late to check max hops, because the search has already finished. It would be better to already check HopLimit when extending the bestNode in the main loop above. So don't extend any paths that are already HopLimit hops. That way, the algorithm will continue the search with more costly paths that may stay below the limit. Similar to how the fee limit is checked in processEdge.

This is a pre-existing problem, but now that you want to solve it, I think it is better to do it without findKPaths.

[Bluetegu]

Hi @joostjager. I think it is not possible to check the hop limit during dijkstra, unless you extend it a non trivial way. The problem is that you may already have dismissed the 'more costly fee' alternate path before reaching the node that is identified as going beyond the HopLimit. Please take a look for example at the added test scenario that checks the bug fix. I may be wrong and there is a clever way to do that. I opened a specific PR to address the HopLimit problem PR#2380. Since this PR also needs the same breakup of code I merged code of PR#2380 in here too. I suggest we discuss how to best address the HopLimit problem there so we can fix that bug as soon as possible. Thanks!

[joostjager]

Yes, I know that Dijkstra is not returning the optimal solution for this problem. It finds the lowest cost path to every node without taking into account the hop limit. But cutting off too long paths and continuing the search at least increases the chance to find a solution, as opposed to just returning an error.

I can see that trying to fix the limitation by running k-shortest paths is another way to address it, but it isn't optimal either. So the 'bug' still isn't fixed. I think it is computationally more intensive too.

Also, running into the hop limit is not something that commonly happens.

[Bluetegu]

But cutting off too long paths and continuing the search at least increases the chance to find a solution, as opposed to just returning an error.

I'm not sure I follow. Are you saying that PR#2380 does not fix the bug? It does fix the simple scenario I added in TestNewRoutePathTooLong to check that the bug was fixed. I am not aware of a graph in which the fix will not find a valid path, if such exists, but I haven't proved mathematically that such a graph doesn't exists either. Let me know if you have such a graph in mind.

I can see that trying to fix the limitation by running k-shortest paths is another way to address it, but it isn't optimal either. So the 'bug' still isn't fixed. I think it is computationally more intensive too.

What do you mean by it isn't optimal? What do you mean the bug is not fixed?

Also, running into the hop limit is not something that commonly happens.

Exactly. That is why the additional findKpaths is run only if the shortest path found has too much hops. No additional computations were added to findPath and it keeps the Dijkstra computational complexity and code intact. The hop limits checks were added to the findKPaths stage, and not to the Dijkstra findPath stage.

I understand the fix may look strange. I do not have a better fix than that. Cutting too long paths in Dijkstra does not find valid paths in all graphs. If there is a better algorithm, I'll be more than happy to implement it, if my help would be useful.

[joostjager]

Moved the discussion to #2380

[joostjager]

What is the real-life use case for pegging nodes? I can imagine that it is useful to control the outgoing channel from self in the context of "channel balancing while paying", but does it matter which intermediate nodes are used?

[Bluetegu]

Pegging of both nodes and channels is working. The error in the integration is in the data loss protection, and I believe is unrelated to this code.

[Crypt-iQ]

Since findPaths is going to be deprecated (mentioned here: #2497) by 0.7 release, most of the code changes here will be obsolete. Pegging hops would not be compatible with the new queryroutes using any-to-any queries since one cannot stitch together routes in the same way one can stitch together paths. In my mind, pegging hops requires multiple iterations of Djikstra's and might require a new pathfinding algorithm. I can't think of any way we could pass in a "restriction" that makes findPath go through a hop without heavily modifying the algorithm itself.

[Roasbeef]

Closing this for now as we've gone with an alternative approach (edge white/black listing).