dmidecode triggers system reboot on Inforce 6640

[ubuntu-server-builder]

This bug was originally filed in Launchpad as LP: #1858615

Launchpad details

affected_projects = ['dmidecode (Ubuntu)', 'dmidecode (Ubuntu Xenial)', 'dmidecode (Ubuntu Bionic)', 'dmidecode (Ubuntu Eoan)', 'dmidecode (Ubuntu Focal)', 'dmidecode (Debian)']
assignee = None
assignee_name = None
date_closed = 2020-01-09T00:26:28.132896+00:00
date_created = 2020-01-07T08:29:52.548932+00:00
date_fix_committed = None
date_fix_released = None
id = 1858615
importance = undecided
is_complete = True
lp_url = https://bugs.launchpad.net/cloud-init/+bug/1858615
milestone = None
owner = ethan.hsieh
owner_name = ethan.hsieh
private = False
status = invalid
submitter = ethan.hsieh
submitter_name = ethan.hsieh
tags = ['verification-done', 'verification-done-bionic', 'verification-done-eoan', 'verification-done-xenial']
duplicates = []

Launchpad user ethan.hsieh(ethan.hsieh) wrote on 2020-01-07T08:29:52.548932+00:00

[Impact]
Running 'sudo dmidecode' on non-UEFI ARM systems can cause them to crash/reboot. cloud-init apparently runs dmidecode as root, so it breaks any cloud-init based installation.

[Test Case]
sudo dmidecode

[Fix]
Upstream has the following fix:

commit e12ec26e19e02281d3e7258c3aabb88a5cf5ec1d
Author: Jean Delvare jdelvare@suse.de
Date: Mon Aug 26 14:20:15 2019 +0200

dmidecode: Only scan /dev/mem for entry point on x86

[Regression Risk]
In Ubuntu, dmidecode only builds on amd64, arm64, armhf & i386.
The fix is to disable code on !x86, so the regression risk is restricted to ARM platforms, where we know /dev/mem trolling is bad news.

[ubuntu-server-builder]

Launchpad user Ryan Harper(raharper) wrote on 2020-01-09T00:26:27.114717+00:00

Thanks for filing a bug. I've added a dmidecode task to track the issue with the tool. It may also affect the kernel package, and possibly firmware (though that's not something that Ubuntu provides). Cloud-init and any other tool may invoke this package and it should not reboot the system; but there's no issue with cloud-init or other callers of the tool As such, I'm marking the cloud-init task as invalid.

[ubuntu-server-builder]

Launchpad user ethan.hsieh(ethan.hsieh) wrote on 2020-01-14T10:05:56.754418+00:00

@ryan
Not all aarch64 platforms can support demidecode well.
How about revert the patch? or create a black list?

[ubuntu-server-builder]

Launchpad user Dan Watkins(oddbloke) wrote on 2020-01-14T16:13:44.388841+00:00

Ethan, cloud-init isn't the right level to handle hardware issues with DMI (which is why we've added the dmidecode task). I've pinged kernel folks to see if we can get eyes on this issue.

[ubuntu-server-builder]

Launchpad user Colin Ian King(colin-king) wrote on 2020-01-14T17:32:13.042855+00:00

Does the kernel expose the DMI tables via the sysfs following sysfs file: /sys/firmware/dmi/tables/DMI ?

If so, can you do the following:

sudo cat /sys/firmware/dmi/tables/DMI > dmi.raw

and attach it to the bug report. Also a dump of the kernel dmesg log after it boots may be useful to see if it's a broken firmware DMI table or a kernel issue.

[ubuntu-server-builder]

Launchpad user Colin Ian King(colin-king) wrote on 2020-01-14T17:33:34.626804+00:00

Oh, stupid me, I've just read the info in comment #1

[ubuntu-server-builder]

Launchpad user ethan.hsieh(ethan.hsieh) wrote on 2020-01-15T01:58:35.322325+00:00

@dan
Got it. Thanks~

[ubuntu-server-builder]

Launchpad user Colin Ian King(colin-king) wrote on 2020-01-27T11:31:07.891071+00:00

Hi, can you provide me instructions on how to get and install the image for this board? I'd like to reproduce this issue and get a suitable fix for this.

[ubuntu-server-builder]

Launchpad user Colin Ian King(colin-king) wrote on 2020-01-27T15:48:08.025090+00:00

So, dmidecode directly mmap's to /dev/mem and does some probing based on the belief that the system is a x86 architecture even on arm architectures.

openat(AT_FDCWD, "/dev/mem", O_RDONLY) = 3
fstat(3, {st_mode=S_IFCHR|0640, st_rdev=makedev(0x1, 0x1), ...}) = 0
mmap(NULL, 65536, PROT_READ, MAP_SHARED, 3, 0xf0000) = 0x7f9f6fd000

etc

So that's kind of intrusive and as root one can read any sort of physical addresses in /dev/mem that may cause breakage.

[ubuntu-server-builder]

Launchpad user Colin Ian King(colin-king) wrote on 2020-01-27T16:23:02.602146+00:00

dmidocode.c directly accesses memory and assumes it's an x86 without any checking that the arch is x86.. Randomly scanning arbitrary hunks of memory on non-x86 as root will lead to all sorts of woe:

memory_scan:
if (!(opt.flags & FLAG_QUIET))
printf("Scanning %s for entry point.\n", opt.devmem);
/* Fallback to memory scan (x86, x86_64) */
if ((buf = mem_chunk(0xF0000, 0x10000, opt.devmem)) == NULL)
{
ret = 1;
goto exit_free;
}

It probably needs wrapping with:

#if defined(x86_64) || defined(__x86_64) ||
defined(i386) || defined(__i386)

...

#endif

Anyhow, I don't think this is a kernel specific issue. I can trigger this with various kernels - we just don't protect users with CAP_SYS_ADMIN rights doing crazy probing on /dev/mem.

[ubuntu-server-builder]

Launchpad user Colin Ian King(colin-king) wrote on 2020-01-27T17:07:06.561285+00:00

I guess the next question is why dmidecode being run as root is required on a cloud init? What happens when arches don't have DMI data?

[ubuntu-server-builder]

Launchpad user Colin Ian King(colin-king) wrote on 2020-01-27T17:10:21.104440+00:00

Upstream has a fix like the one I was hinting at in comment #9, I'll SRU this fix.

commit e12ec26e19e02281d3e7258c3aabb88a5cf5ec1d
Author: Jean Delvare jdelvare@suse.de
Date: Mon Aug 26 14:20:15 2019 +0200

dmidecode: Only scan /dev/mem for entry point on x86

x86 is the only architecture which can have a DMI entry point scanned
from /dev/mem. Do not attempt it on other architectures, because not
only it can't work, but it can even cause the system to reboot.

This fixes support request #109697:
https://savannah.nongnu.org/support/?109697

[ubuntu-server-builder]

Launchpad user dann frazier(dannf) wrote on 2020-01-27T17:12:25.707384+00:00

We carry this patch in focal - do we just need to backport it to bionic?

http://git.savannah.nongnu.org/cgit/dmidecode.git/commit/?id=e12ec26e19e02281d3e7258c3aabb88a5cf5ec1d

I uploaded a test fix to ppa:dannf/test. Could someone w/ hw access verify that?

[ubuntu-server-builder]

Launchpad user Colin Ian King(colin-king) wrote on 2020-01-27T17:22:17.499074+00:00

It needs backporting to eoan, disco bionic, I was just about to upload a fix to my ppa so I could get it sponsored. Do you want to take it from here Dann?

[ubuntu-server-builder]

Launchpad user dann frazier(dannf) wrote on 2020-01-27T17:30:14.064347+00:00

Sure Colin, I'll take it from here - thanks for your analysis so far. As a next step, I'll wait for Ethan - or someone else w/ hw access - to verify the PPA build in Comment #12.