link changes or nexthop deletion may lead to cache desync

[KanjiMonster]

The kernel does not notify always if nexthops or routes are removed:

• deleting a nexthop with id will silently remove all routes with this nexthop as destination
• a link losing carrier (or set down) will silently remove all nexthops with id via this link, which may then trigger silent removal or routes

This means that when a link with a route with a nexthop with id goes down, all libnl sees is the link update that loses carrier, but none of the nexthop or route deletions.

I'm not sure how to properly handle this, though my first idea would be:

• when we get a link update about a link losing carrier:

  • delete all nhs via that link

• when we get a nh deletion:

  • delete all routes via that nh (matched on nh_id)

To make this work properly, we would need some way to send emulate RTM_DELNEXTHOP / RTM_DELROUTE events.

Unfortunately while removing the objects from the cache is "easy" as we can get to the appropriate cache via nl_cache_mngt_require_safe(), there is no easy way to notify about these changes, as there is currently no way to get from a cache to the associated manager.

My idea there would be to extend struct nl_cache_assoc with a link to the "owning" manager, so the is a way to get to the manager and then add a event_input() variant that directly takes a nl_object.

[c-wicklein]

@KanjiMonster I wonder how emulated RTM_DELNEXTHOP / RTM_DELROUTE events would interact with #224.

On my test system, when disconnecting an interface wan0, the kernel removes both a default route via wan0 and a host route via wan0, but (per this issue), libnl only removes the default route and retains the host route. Except there's a catch. The test system started with two default routes, one via wan0 and another via wan1. Related to #224, the route/route cache starts with a default route via wan1 but not the second default route via wan0. When wan0 is disconnected, the default route via wan1 is removed from the route/route cache.

[c-wicklein]

I'm thinking to address this problem at the application level and hopefully avoid any reentrancy issues. I'm using libev to watch each cache manager's Netlink socket and call nl_cache_mngr_data_ready as they become ready to read. I would have the link cache manager's update callback conditionally blow up the route cach manager after problematic link changes and allocate a new one.

[KanjiMonster]

No need to blow up the cache, you can use nl_cache_resync()* to detect the routes that were silently added/removed/modified.

This is more or less what I came up with (bisdn/basebox@e09db4e).

*_v2() is currently in main, but not in a release.

[KanjiMonster]

@KanjiMonster I wonder how emulated RTM_DELNEXTHOP / RTM_DELROUTE events would interact with #224.

On my test system, when disconnecting an interface wan0, the kernel removes both a default route via wan0 and a host route via wan0, but (per this issue), libnl only removes the default route and retains the host route. Except there's a catch. The test system started with two default routes, one via wan0 and another via wan1. Related to #224, the route/route cache starts with a default route via wan1 but not the second default route via wan0. When wan0 is disconnected, the default route via wan1 is removed from the route/route cache.

The issue in #224 is a bit more complicated. You can have multiple routes for a destination with different nexthops, but you can also update a route to use a different nexthop. Both would trigger a RTM_NEWROUTE notification from the kernel, and you need to look at the message flags to tell them apart (NLM_F_CREATE vs NLM_F_REPLACE).

As in #244 is mentioned, libnl / the cache currently does not do that, so both look the same to it, and the messages are treated like an update. This needs to be fixed/extended like network-manager does.

[c-wicklein]

No need to blow up the cache, you can use nl_cache_resync()* to detect the routes that were silently added/removed/modified.

Thanks! I noticed the similar function nl_cache_refill yesterday, but I didn't jump on it because I'd need to switch from using automatically provided sockets to ones I allocate myself, and there's a note in the API documentation for nl_cache_mngr_alloc regarding user-provided sockets advising not to "use the socket for anything else besides receiving netlink notifications." I'm not sure what that means exactly. I assume that sending a message which elicits a response from the kernel is more than "receiving notifications." If so, is this only a concern for arbitrary user code and not libnl's own cache functions? There's also a note that "if the flag NL_AUTO_PROVIDE is specified, any cache added to the manager will automatically be made available to other users using nl_cache_mngt_provide." I'm not sure what are the implications of a cache no longer being made available to other modules.

[c-wicklein]

Well, I've started allocating a socket for the routing domain, using it to initialize a cache manager for the routing domain, and I'm creating caches for both routes and links. As a first pass, in the update callback for changes to the link cache, without regard for what actually happened, I'm calling nl_cache_refill for the route cache. I'm using libnl 3.11.0 built with the patch attached to #224 (include interface index in hashed route attributes.)

It tentatively looks like libnl's route cache now remains correct (or at least is correct after an update occurs and the corresponding cache update callback completes.)

However, I've noticed that calls to nl_cache_refill sometimes fail. Calling nl_cache_refill on the route cache once for every update to the link cache, in a sequence of 26 link changes, 8 nl_cache_refill calls succeed, 6 fail on -NLE_BUSY, and 13 fail on -NLE_MSGTYPE_NOSUPPORT.

Considering the specific circumstances of each link update, I might be able to eliminate some nl_cache_refill calls, and if that doesn't resolve at least the -NLE_BUSY failures, I imagine that I could schedule another attempt. The -NLE_MSGTYPE_NOSUPPORT failures are a bit surprising, though. I'm going to review link change circumstances and see if they relate to which nl_cache_refill calls succeed or fail in any obvious ways.

[KanjiMonster]

Well, I've started allocating a socket for the routing domain, using it to initialize a cache manager for the routing domain, and I'm creating caches for both routes and links. As a first pass, in the update callback for changes to the link cache, without regard for what actually happened, I'm calling nl_cache_refill for the route cache. I'm using libnl 3.11.0 built with the patch attached to #224 (include interface index in hashed route attributes.)

It tentatively looks like libnl's route cache now remains correct (or at least is correct after an update occurs and the corresponding cache update callback completes.)

However, I've noticed that calls to nl_cache_refill sometimes fail. Calling nl_cache_refill on the route cache once for every update to the link cache, in a sequence of 26 link changes, 8 nl_cache_refill calls succeed, 6 fail on -NLE_BUSY, and 13 fail on -NLE_MSGTYPE_NOSUPPORT.

Considering the specific circumstances of each link update, I might be able to eliminate some nl_cache_refill calls, and if that doesn't resolve at least the -NLE_BUSY failures, I imagine that I could schedule another attempt. The -NLE_MSGTYPE_NOSUPPORT failures are a bit surprising, though. I'm going to review link change circumstances and see if they relate to which nl_cache_refill calls succeed or fail in any obvious ways.

Sounds you are using the same netlink socket for requests and notifications. You need to have two distinct netlink sockets, one where the notifications are received, and one for sending requests and reading the replies to them.

If you do it on the same socket, then e.g. a notification can get in-between the (update) request and reply, confusing libnl (this is likely the source for -NLE_MSGTYPE_NOSUPPORT).

[c-wicklein]

Thanks! Using distinct sockets, the route/link update callback successfully calls nl_cache_refill for the route/route cache on every route/link update.

Unfortunately, it appears that there's a timing consideration. Disconnecting an interface results in my route/link update callback being called after an affected next hop is flagged linkdown but before it's flagged dead. Re-connecting the interface results in my route/link update callback being called after the linkdown flag is cleared but perhaps a second or more before the dead flag is cleared. After disconnection, the affected next hop is simply flagged linkdown in the route/route cache, and after reconnection, it remains flagged dead.

I'm kicking around the idea of deferring route/route updates for several seconds after route/link updates, but first I'll see if I can understand how any other applications handle this successfully.

[c-wicklein]

I ended up extending an otherwise event-driven (libnl cache manager callback and user input) application to schedule a three-second repeating timer for "housekeeping" checks. Beginning with any route/link update and ending 15 seconds later, each time the three-second timer expires, I note the contents of the route/route cache (SHA of an NL_DUMP_DETAILS route/route cache dump) , refill it, and again note its contents. If anything changes, I call the same functions which would normally handle a route/route callback from libnl, with the caveat that since I don't know what actually changed with routes, I pass a null pointer as obj.