key error on network debug info

[chaporgin]

This fixes KeyError on specific network configuration when running cloud-init on "network" stage. See the comments for the output.

Proposed Commit Message

summary: no more than 70 characters

A description of what the change being made is and why it is being
made, if the summary line is insufficient.  The blank line above is
required. This should be wrapped at 72 characters, but otherwise has
no particular length requirements.

If you need to write multiple paragraphs, feel free.

LP: #NNNNNNN (replace with the appropriate bug reference or remove
this line entirely if there is no associated bug)

Additional Context

Test Steps

Checklist:

• My code follows the process laid out in the documentation
• I have updated or added any unit tests accordingly
• I have updated or added any documentation accordingly

[chaporgin]

 ~ # ip -o route list
default via 192.168.1.1 dev eth0
192.168.1.0/24 dev eth0 proto kernel scope link src 192.168.1.21
~ # ip --oneline -6 -o route list table all
default dev eth1 table 8 metric 1024 pref medium
::1 dev lo proto kernel metric 256 pref medium
2a02:6b8:c02:901:0:f804:0:25 dev eth1 proto kernel metric 256 pref medium
2a02:6b8::/32 dev eth1 metric 1024 pref medium
2a0d:d6c0::/32 dev eth1 metric 1024 pref medium
fe80::/64 dev eth1 proto kernel metric 256 pref medium
fe80::/64 dev eth0 proto kernel metric 256 pref medium
local ::1 dev lo table local proto kernel metric 0 pref medium
local 2a02:6b8:c02:901:0:f804:0:25 dev eth1 table local proto kernel metric 0 pref medium
local fe80::d20d:1cff:fede:db82 dev eth0 table local proto kernel metric 0 pref medium
local fe80::d21d:1cff:fede:db82 dev eth1 table local proto kernel metric 0 pref medium
ff00::/8 dev eth1 table local metric 256 pref medium
ff00::/8 dev eth0 table local metric 256 pref medium
 ~ # cloud-init init
Cloud-init v. 20.3-2-g371b392c-0ubuntu1~18.04.1 running 'init' at Sun, 03 Jan 2021 15:24:15 +0000. Up 3897.48 seconds.
2021-01-03 15:24:15,517 - util.py[WARNING]: failed stage init
failed run of stage init
------------------------------------------------------------
Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/cloudinit/cmd/main.py", line 653, in status_wrapper
    ret = functor(name, args)
  File "/usr/lib/python3/dist-packages/cloudinit/cmd/main.py", line 284, in main_init
    sys.stderr.write("%s\n" % (netinfo.debug_info()))
  File "/usr/lib/python3/dist-packages/cloudinit/netinfo.py", line 495, in debug_info
    route_lines = route_pformat().splitlines()
  File "/usr/lib/python3/dist-packages/cloudinit/netinfo.py", line 479, in route_pformat
    r['gateway'], r['iface'], r['flags']])
KeyError: 'gateway'

.... applied the patch ...

~ # cloud-init init
Cloud-init v. 20.3-2-g371b392c-0ubuntu1~18.04.1 running 'init' at Sun, 03 Jan 2021 16:21:30 +0000. Up 7332.45 seconds.
ci-info: +++++++++++++++++++++++++++++++++++++++++Net device info+++++++++++++++++++++++++++++++++++++++++
ci-info: +--------+------+----------------------------------+---------------+--------+-------------------+
ci-info: | Device |  Up  |             Address              |      Mask     | Scope  |     Hw-Address    |
ci-info: +--------+------+----------------------------------+---------------+--------+-------------------+
ci-info: |  eth0  | True |           192.168.1.21           | 255.255.255.0 | global | d0:0d:1c:de:db:82 |
ci-info: |  eth0  | True |   fe80::d20d:1cff:fede:db82/64   |       .       |  link  | d0:0d:1c:de:db:82 |
ci-info: |  eth1  | True | 2a02:6b8:c02:901:0:f804:0:25/128 |       .       | global | d0:1d:1c:de:db:82 |
ci-info: |  eth1  | True |   fe80::d21d:1cff:fede:db82/64   |       .       |  link  | d0:1d:1c:de:db:82 |
ci-info: |   lo   | True |            127.0.0.1             |   255.0.0.0   |  host  |         .         |
ci-info: |   lo   | True |             ::1/128              |       .       |  host  |         .         |
ci-info: +--------+------+----------------------------------+---------------+--------+-------------------+
ci-info: +++++++++++++++++++++++++++++Route IPv4 info+++++++++++++++++++++++++++++
ci-info: +-------+-------------+-------------+---------------+-----------+-------+
ci-info: | Route | Destination |   Gateway   |    Genmask    | Interface | Flags |
ci-info: +-------+-------------+-------------+---------------+-----------+-------+
ci-info: |   0   |   0.0.0.0   | 192.168.1.1 |    0.0.0.0    |    eth0   |   UG  |
ci-info: |   1   | 192.168.1.0 |   0.0.0.0   | 255.255.255.0 |    eth0   |   U   |
ci-info: +-------+-------------+-------------+---------------+-----------+-------+
ci-info: +++++++++++++++++++++++++++Route IPv6 info++++++++++++++++++++++++++++
ci-info: +-------+------------------------------+---------+-----------+-------+
ci-info: | Route |         Destination          | Gateway | Interface | Flags |
ci-info: +-------+------------------------------+---------+-----------+-------+
ci-info: |   0   |             ::/0             |         |    eth1   |   UG  |
ci-info: |   2   | 2a02:6b8:c02:901:0:f804:0:25 |    ::   |    eth1   |   U   |
ci-info: |   3   |        2a02:6b8::/32         |    ::   |    eth1   |   U   |
ci-info: |   4   |        2a0d:d6c0::/32        |    ::   |    eth1   |   U   |
ci-info: |   5   |          fe80::/64           |    ::   |    eth1   |   U   |
ci-info: |   6   |          fe80::/64           |    ::   |    eth0   |   U   |
ci-info: |   8   |            local             |    ::   |    eth1   |   U   |
ci-info: |   9   |            local             |    ::   |    eth0   |   U   |
ci-info: |   10  |            local             |    ::   |    eth1   |   U   |
ci-info: |   11  |           ff00::/8           |    ::   |    eth1   |   U   |
ci-info: |   12  |           ff00::/8           |    ::   |    eth0   |   U   |
ci-info: +-------+------------------------------+---------+-----------+-------+
.... 
exits with 0

[chaporgin]

Awaits #747

[raharper]

Thanks for submitting a fix. Some questions, comments and suggestions:

default dev eth1 table 8 metric 1024 pref medium

How did you get a default route through a table?

The real issue here is that netinfo.py does not handle reading from tables. This patch you're providing omits the gateway field altogether, rather than specifying '::' if the entry does not have a gateway.

I'd like to see:

1. A try: except around the the table_v6.add_row() on line 476 catching KeyError as err and LOG that as a warning; indicating that the output may be missing some of the routing info
2. Use r.get('gateway', '::') when fetching the value for gateway; if missing it will default to the no-gateway value.
3. Add a unittest to cloudinit/tests/test_netinfo.py ; you can provide the 3 input files (route_v4, route_v6, and formatted table output), see tests/data/netinfo/ for examples , like sample-iproute-out-v{4,6} route-formatted-output. You can duplicate test_route_iproute_pformat test, changing the function name, docstring, and the names of the input variables and add new ones loading the resource output files you'll place in the tests/data/netinfo dir.

If you're up for it, it would be great to have support for looking up route info in tables other than default, like your example here if we see 'table' in the tokens of a route line, then we'd want to run and collect the output of:

ip route list table 8

and include that in the output table cloud-init prints.

[TheRealFalcon]

@raharper You beat me to this one. 😉

Isn't the issue actually line 306 . If we moved that line above the if statement, I think it'd parse out correctly.

[raharper]

@raharper You beat me to this one. wink

Isn't the issue actually line 306 . If we moved that line above the if statement, I think it'd parse out correctly.

That would work, but we'd not catch any other missing fields for entry in the table. I was hoping to log that we've got incomplete info.

[raharper]

2\. Use r.get('gateway', '::') when fetching the value for gateway; if missing it will default to the no-gateway value.

Actually, I prefer indicating that we don't know the value, similar to what you have; it's not undefined, rather it's 'unknown' or 'not present', or 'missing'. Something to that effect.

Ideally we parse the table N rules to see if there is a gateway (there almost certainly is for a default gateway, v6 always needs a via, though it's possible that SLAAC setups with router advertisements aren't yet received so we would want to keep 'unknown' or whatever we choose in the case that we don't know yet.

[github-actions]

Hello! Thank you for this proposed change to cloud-init. This pull request is now marked as stale as it has not seen any activity in 14 days. If no activity occurs within the next 7 days, this pull request will automatically close.

If you are waiting for code review and you are seeing this message, apologies! Please reply, tagging mitechie, and he will ensure that someone takes a look soon.

(If the pull request is closed, please do feel free to reopen it if you wish to continue working on it.)