issue with double mac addresses on azure/ib0 devices

[ubuntu-server-builder]

This bug was originally filed in Launchpad as LP: #1874544

Launchpad details

affected_projects = []
assignee = None
assignee_name = None
date_closed = 2021-11-01T19:44:05.750161+00:00
date_created = 2020-04-23T20:44:03.187152+00:00
date_fix_committed = 2021-11-01T19:44:05.750161+00:00
date_fix_released = 2021-11-01T19:44:05.750161+00:00
id = 1874544
importance = high
is_complete = True
lp_url = https://bugs.launchpad.net/cloud-init/+bug/1874544
milestone = None
owner = rharding
owner_name = Richard Harding
private = False
status = fix_released
submitter = rharding
submitter_name = Richard Harding
tags = ['id-5e3c4c9ce38502288bd97d44']
duplicates = []

Launchpad user Richard Harding(rharding) wrote on 2020-04-23T20:44:03.187152+00:00

Salesforce: https://canonical.my.salesforce.com/5003z000022ssHs

Reproducer:
After launching the HC44rs instance with a standard Bionic image, install the rdma-core package, and reboot.

Error:
The following error message will appear on the console and ib0 and eth1 will both be unconfigured:
https://pastebin.canonical.com/p/VfkNGc58qd/

[ubuntu-server-builder]

Launchpad user Richard Harding(rharding) wrote on 2020-04-23T20:44:03.187152+00:00

[ubuntu-server-builder]

Launchpad user Dan Watkins(oddbloke) wrote on 2020-04-23T20:56:58.671683+00:00

The traceback (from the Canonical-internal pastebin) is:

Traceback (most recent call last):
File "/usr/lib/python3/dist-packages/cloudinit/cmd/main.py", line 653, in status_wrapper
ret = functor(name, args)
File "/usr/lib/python3/dist-packages/cloudinit/cmd/main.py", line 362, in main_init
init.apply_network_config(bring_up=bool(mode != sources.DSMODE_LOCAL))
File "/usr/lib/python3/dist-packages/cloudinit/stages.py", line 701, in apply_network_config
net.wait_for_physdevs(netcfg)
File "/usr/lib/python3/dist-packages/cloudinit/net/init.py", line 519, in wait_for_physdevs
present_macs = get_interfaces_by_mac().keys()
File "/usr/lib/python3/dist-packages/cloudinit/net/init.py", line 803, in get_interfaces_by_mac
return get_interfaces_by_mac_on_linux()
File "/usr/lib/python3/dist-packages/cloudinit/net/init.py", line 842, in get_interfaces_by_mac_on_linux
(name, ret[mac], mac))
RuntimeError: duplicate mac found! both 'eth1' and 'ib0' have mac '00:15:5d:33:ff:13'

[ubuntu-server-builder]

Launchpad user Ryan Harper(raharper) wrote on 2020-04-23T21:27:32.763110+00:00

we've seen this before and my open questions are:

1. is there an Infiniband standard about which fields of the IB physical address (which is much bigger than MAC) that correspond to ethernet over IB mac ? we have one implementation in tree, but it's not clear if that's the correct value to be using or not

2. is the IB physical address created by the platform? do they intentially embed the same MAC of another nic in the IB interface ?

I'd really like to understand why this happens in a VM which they're SRIOV passing in these devices.

That said, we've discussed in the past to start warning there's a duplicate mac in cloud-init and then just see what happens.

[ubuntu-server-builder]

Launchpad user Joseph Salisbury(jsalisbury) wrote on 2020-05-11T18:02:59.928229+00:00

I was able to get an answer to question 1 in comment #3:

In Linux implementation, the link address for IPoIB has 20 bytes:
1st 4 bytes: the QP number and flags for IPoIB device
next 8 bytes: the subnet prefix for the Infiniband device
next 8 bytes: the device port ID for the Infiniband device

The last 8 bytes should be stable for an IPoIB device. But I'm not aware of any Infiniband standard that defines this format.

For question 2:
As far as I know we don’t create IB MAC addresses on the platform side and I don’t think they follow the MAC of another NIC/port. I am waiting for additional responses for this question. I'll post them here when I get them.

[ubuntu-server-builder]

Launchpad user Joseph Salisbury(jsalisbury) wrote on 2020-05-11T20:49:58.475172+00:00

When Hyper-V surfaces the VF via SR-IOV, it uses the same MAC address as the synthetic vNIC. In the Ethernet case, it combines the VF and the synthetic vNIC so you only see one interface in the VM, this also allows them to fallback to synthetic path if the VF stops working. In the IB case, hyper-v doesn’t support combining the VF and the synthetic vNIC thus there are two interfaces in the VM with the same MAC address. This has been the case from day 1 for IB SR-IOV.

So, “Is the IB physical address created by the platform?” – yes, when Hyper-V creates the VF, it copies the MAC address from the synthetic vNIC and assigns it to the VF.

“Do they intentionally embed the same MAC of another nic in the IB interface?” – yes, this is by hyper-v design.

[ubuntu-server-builder]

Launchpad user Ryan Harper(raharper) wrote on 2020-05-12T16:50:17.486791+00:00

When Hyper-V surfaces the VF via SR-IOV, it uses the same MAC address
as the synthetic vNIC. In the Ethernet case, it combines the VF and
the synthetic vNIC so you only see one interface in the VM, this also
allows them to fallback to synthetic path if the VF stops working. In
the IB case, hyper-v doesn’t support combining the VF and the
synthetic vNIC thus there are two interfaces in the VM with the same
MAC address. This has been the case from day 1 for IB SR-IOV.

Thanks for the backgound. It seems like duplicating the MAC was
designed for ethernet around the kernel "auto bonding" to the SRIOV
device. Since HyperV lacks support for "auto bonding" the IB device
this design is suboptimal; OSes generally don't expect duplicate MACs
outside of bonds. You're likely to run into other software which is
going to complain/not work due to this. Something to think on long
term.

So, “Is the IB physical address created by the platform?” – yes,
when Hyper-V creates the VF, it copies the MAC address from the
synthetic vNIC and assigns it to the VF.

Do you know if the VF for the IB device MAC is user-space modifiable?

Also, do you know of IMDS output includes network configuration about
the VF/IB device?

“Do they intentionally embed the same MAC of another nic in the IB
interface?” – yes, this is by hyper-v design.

I'm wondering if cloud-init should look at a different portion of the
ib device 'address'. Since we have existing code to look at this
location of the ib address, it seems at least, by convention that we
look at this location, however the platform (HyperV) has duplicated
the MAC for some reason that other platforms do not.

I don't think cloud-init is looking at the wrong location; rather
a HyperV/Azure platform implementation detail is leaking into the VM.
I think we'd take a path where we ignore these duplicate MACs for
IB devices.

Lastly, do we have have instructions on where/how to launch such
an instance on Azure to reproduce?

[ubuntu-server-builder]

Launchpad user Joseph Salisbury(jsalisbury) wrote on 2020-05-22T20:15:33.229468+00:00

The VF’s IB GID is derived from the MAC address. I don’t know the implication of modifying the MAC address even if it’s modifiable.

Can you elaborate on what the IMDS output is?

Mellanox is looking into creating a driver for us so that we can expose the VF using DDA instead of using the vSwitch. This will eliminate the duplicate MAC issue for sure, but there is no estimate on when this would be released.

There is info on how to reproduce this issue in the Salesforce case.

[ubuntu-server-builder]

Launchpad user Ryan Harper(raharper) wrote on 2020-05-22T21:47:30.059897+00:00

The VF’s IB GID is derived from the MAC address. I don’t know the
implication of modifying the MAC address even if it’s modifiable.

I'm not sure either, but was hoping that we could detect this for IB
and change it.

Can you elaborate on what the IMDS output is?

Azure's Instance MetaData Service which includes network
configuration information (among other things)

https://docs.microsoft.com/en-us/azure/virtual-machines/linux/instance-metadata-service

https://docs.microsoft.com/en-us/azure/virtual-machines/linux/instance-metadata-service#network-metadata

{
"interface": [
{
"ipv4": {
"ipAddress": [
{
"privateIpAddress": "10.1.0.4",
"publicIpAddress": "X.X.X.X"
}
],
"subnet": [
{
"address": "10.1.0.0",
"prefix": "24"
}
]
},
"ipv6": {
"ipAddress": []
},
"macAddress": "000D3AF806EC"
}
]
}

Mellanox is looking into creating a driver for us so that we can expose the
VF using DDA instead of using the vSwitch. This will eliminate the duplicate
MAC issue for sure, but there is no estimate on when this would be released.

OK, ideally another MAC would be allocated to the ib interface instead of
duplicating the hvnet ethernet MAC.

There is info on how to reproduce this issue in the Salesforce case.

OK

[ubuntu-server-builder]

Launchpad user Joseph Salisbury(jsalisbury) wrote on 2020-06-30T18:33:01.882638+00:00

Are there suggestions on the next steps for this bug?

[ubuntu-server-builder]

Launchpad user Anh Vo (MSFT)(vtqanh) wrote on 2020-09-04T12:36:28.016158+00:00

@ryan Harper: IMDS does not return information about the infiniband interface. It does return information on the Vnic that the infiniband interface is associated with (eth1 in this case).

OSes generally don't expect duplicate MACs
outside of bonds. You're likely to run into other software which is
going to complain/not work due to this. Something to think on long
term.
I agree it's not an ideal SRIOV implementation by Hyper-V, but in practice it doesn't really have any issue. The "MAC address" of the IB interface isn't used for anything because IB isn't routable. There are also not a lot of software that leverage that deal with RDMA and IB. I suspect most of the ones that do will not care about Mac address of IB.

I am curious as to how cloud-init got the 6 bytes IB MAC address, because in my VM I am seeing a 20 bytes IPoIB mac address from the ip link show command.

This is what it looked like in my test deployment:
root@anhtest-10769:/home/azlinux# ip link show
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN mode DEFAULT group default qlen 1000
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP mode DEFAULT group default qlen 1000
link/ether 00:0d:3a:c6:6e:29 brd ff:ff:ff:ff:ff:ff
3: eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP mode DEFAULT group default qlen 1000
link/ether 00:15:5d:33:ff:14 brd ff:ff:ff:ff:ff:ff
5: ib0: <BROADCAST,MULTICAST> mtu 4092 qdisc noop state DOWN mode DEFAULT group default qlen 256
link/infiniband 20:00:09:28:fe:80:00:00:00:00:00:00:00:15:5d:ff:fd:33:ff:14 brd 00:ff:ff:ff:ff:12:40:1b:80:0b:00:00:00:00:00:00:ff:ff:ff:ff

This is the network portion of the IMDS output:
"network": {
"interface": [
{
"ipv4": {
"ipAddress": [
{
"privateIpAddress": "10.0.0.5",
"publicIpAddress": "52.175.238.35"
}
],
"subnet": [
{
"address": "10.0.0.0",
"prefix": "24"
}
]
},
"ipv6": {
"ipAddress": []
},
"macAddress": "000D3AC66E29"
},
{
"ipv4": {
"ipAddress": [],
"subnet": []
},
"ipv6": {
"ipAddress": []
},
"macAddress": "00155D33FF14"
}
]
}

1. How is cloud-init getting the Mac address for ib0? Should it use the full 20 byte address when comparing?
2. It should be safe to ignore the Mac address, if cloud-init can determine that the interface is backed by an infiniband device

[ubuntu-server-builder]

Launchpad user Ryan Harper(raharper) wrote on 2020-09-04T15:22:58.320453+00:00

1. How is cloud-init getting the Mac address for ib0?
   Should it use the full 20 byte address when comparing?

cloudinit/net/init.py:get_ib_interface_hwaddr()

"""Returns the string value of an Infiniband interface's hardware
address. If ethernet_format is True, an Ethernet MAC-style 6 byte
representation of the address will be returned.
"""

In early comments I asked if there was a systematic way of extracting
the portion of an IB HWAddr which corresponds to the Ethernet over IB
layer which can run and I have not found an answer to that yet.

The origin of the Ethernet vs IB style mac comes from here:

e7b0e5f

OpenStack ironic references Infiniband interfaces via a 6 byte 'MAC
address' formed from bytes 13-15 and 18-20 of interface's hardware
address. This address is used as the ethernet_mac_address of Infiniband
links in network_data.json in configdrives generated by OpenStack nova.
We can use this address to map links in network_data.json to their
corresponding interface names.

When generating interface configuration files, we need to use the
interface's full hardware address as the HWADDR, rather than the 6 byte
MAC address provided by network_data.json.

This change allows IB interfaces to be referenced in this dual mode - by
MAC address and hardware address, depending on the context.

It appears that networking-wise, use of the full IB mac is required, however
configuration metadata tends to be unaware of IB vs. Ethernet over IB where
the subsection of the HWAddr is used as an Ethernet MAC.

2. It should be safe to ignore the Mac address, if cloud-init can determine that the interface is backed by an infiniband device

On Azure, yes. Elsewhere I don't think we can say that.

Moving forward, it may be best to have cloud-init ignore duplicate macs;
possibly scoping that to specific platforms. In Azure where it's a design
point it may be better to ignore duplicate macs rather than suggest that
there's something wrong with the networking devices.

[ubuntu-server-builder]

Launchpad user Chad Smith(chad.smith) wrote on 2020-09-23T19:59:48.291042+00:00

Part if this issue (applying network definitions to eth0 (non-IB) only) is in flight by Johnsonshi
in this PR #549 which whitelists the non-IB interface driver as the netplan config emitted based on reading Azure IMDS.

We will still need to sort device renaming login in cloudinit/net/init.py to make sure we don't rename IB interfaces too

[ubuntu-server-builder]

Launchpad user Chad Smith(chad.smith) wrote on 2020-09-24T16:48:17.637942+00:00

Part 1 of this issue (networking on non-IB interfaces) landed today in an upstream commit
4316490