[Bug Report] 使用PVM Deployment，在aws C8i linux2023机器上内核无法启动

[kangcp]

Summary

按照PVM Deployment，更新内核，重启后，机器无法正常启动

Environment

• CubeSandbox version / commit: v0.2.0

• Host OS and kernel version: 6.1.166-197.305.amzn2023.x86_64
  机器详细信息：
  NAME="Amazon Linux"
  VERSION="2023"
  ID="amzn"
  ID_LIKE="fedora"
  VERSION_ID="2023"
  PLATFORM_ID="platform:al2023"
  PRETTY_NAME="Amazon Linux 2023.11.20260413"

• KVM info (modinfo kvm): /lib/modules/6.1.166-197.305.amzn2023.x86_64/kernel/arch/x86/kvm/kvm.ko

• Deployment mode: single-node

• Relevant component:

Steps to Reproduce

1. 下载kernel-6.6.69_cube.pvm.host..rpm和 kernel-headers-6.6.69_cube.pvm.host..rpm

2. 直接安装会失败：rpm -ivh kernel-6.6.69_cube.pvm.host..rpm， 错误信息如下：
   Verifying... ################################# [100%]
   Preparing... ################################# [100%]
   package kernel-1:6.1.166-197.305.amzn2023.x86_64 (which is newer than kernel-6.6.69_cube.pvm.host.005.x_gb85200d80fa2-1.x86_64) is already installed
   [root@s113v112 ~]# rpm -ivh kernel-headers-6.6.69_cube.pvm.host..rpm
   error: Failed dependencies:
   kernel-headers is obsoleted by kernel-headers-6.6.69_cube.pvm.host.005.x_gb85200d80fa2-1.x86_64

3. 修改安装方式（如下方式可以安装成功）：
   dnf install --allowerasing ./kernel-6.6.69_cube.pvm.host..rpm
   dnf install --allowerasing ./kernel-headers-6.6.69_cube.pvm.host..rpm

4. 设置PVM内核为默认启动方式（aws上稍微麻烦点，过程如下）：

第 1 步：生成模块依赖

KVER=$(ls /lib/modules/ | grep cube)
sudo depmod ${KVER}

第 2 步：验证 modules.dep 已生成

ls -la /lib/modules/${KVER}/modules.dep

第 3 步：生成 initramfs

sudo dracut --force /boot/initramfs-${KVER}.img ${KVER}

第 4 步：验证 initramfs 已生成

ls -la /boot/initramfs-*cube*

第 5 步：添加 GRUB 条目 + 设为默认

sudo grubby --add-kernel=/boot/vmlinuz-${KVER} \
  --initrd=/boot/initramfs-${KVER}.img \
  --title="PVM Host Kernel 6.6.69" \
  --copy-default
sudo grubby --set-default=/boot/vmlinuz-${KVER}
# 验证
sudo grubby --default-kernel
grubby --info=ALL | grep -E "^kernel|^index|^title"

5. 配置所需的内核引导参数
   bash <(curl -fsSL
   https://raw.githubusercontent.com/TencentCloud/CubeSandbox/master/deploy/pvm/grub/host_grub_config.sh)

6. 重启

Expected Behavior

重启之后，设置成功

Actual Behavior

机器无法启动

Additional Context

[lkml-likexu]

@kangcp Thank you for your feedback. If you can use a console (such as VNC) to observe what happens to the PVM kernel during the boot process—for example, whether it crashes or enters the GRUB interface—it would be very helpful for our future support efforts.

By the way, host_grub_config.sh isn't suitable for UEFI-based guests.

[kangcp]

@kangcp Thank you for your feedback. If you can use a console (such as VNC) to observe what happens to the PVM kernel during the boot process—for example, whether it crashes or enters the GRUB interface—it would be very helpful for our future support efforts.谢谢你的反馈。如果您可以使用控制台（例如VNC）来观察PVM内核在引导过程中发生的情况（例如，它是崩溃还是进入GRUB界面），这将对我们未来的支持工作非常有帮助。

By the way, host_grub_config.sh isn't suitable for UEFI-based guests.顺便说一下，不适合欧足联的客人。

谢谢，For UEFI-based guests.除了文档中的host_grub_config.sh，还有其他选择吗？

[lkml-likexu]

@kangcp Please make sure your Linux guest is booting in UEFI mode, or try running it on a Tencent Cloud CVM (CubeSandbox for PVM has been tested successfully on this platform), or give the following rough-and-ready script a try:

#!/bin/bash

# Re-exec under bash if the script was invoked through /bin/sh. This keeps
# [[ ]] and pipefail working even on systems where /bin/sh is dash.
if [ -z "${BASH_VERSION:-}" ]; then
    if command -v bash >/dev/null 2>&1; then
        exec bash "$0" "$@"
    else
        echo "ERROR: this script requires bash, but bash was not found in PATH" >&2
        exit 1
    fi
fi

set -euo pipefail

GRUB_CMDLINE_LINUX_APPEND="quiet elevator=noop console=ttyS0,115200 console=tty0 vconsole.keymap=us \
crashkernel=1800M-64G:256M,64G-128G:512M,128G-486G:768M,486G-972G:1024M,972G-:2048M \
vconsole.font=latarcyrheb-sun16 net.ifnames=0 biosdevname=0 intel_idle.max_cstate=1 intel_pstate=disable cgroup.memory=nokmem transparent_hugepage=never \
ipv6.disable=1 systemd.unified_cgroup_hierarchy=1 module.sig_enforce=1 \
clearcpuid=27,28,54,57,104,107,118,120,122,131,152,158,193,196,198,199,200,201,214,215,225,241,249,250,254,289,292,295,297,299,302,306,307,309,311,312,317,321,322,323,389,416,418,425,513,514,517,518,520,521,522,523,524,526,534,537,539,540,580 \
clocksource=tsc pti=off no5lvl mitigations=on spec_store_bypass_disable=prctl retbleed=off \
kvm.nx_huge_pages=never tsc=reliable kmem_cache.max_num=16000"

[[ $EUID -eq 0 ]] || {
    echo "please run as root" >&2
    exit 1
}

cp -a /etc/default/grub /etc/default/grub.bak.$(date +%s)

# Detect firmware mode: UEFI if /sys/firmware/efi exists, otherwise Legacy BIOS.
is_uefi=0
if [[ -d /sys/firmware/efi ]]; then
    is_uefi=1
    echo "Firmware mode: UEFI"
else
    echo "Firmware mode: Legacy BIOS"
fi

# Locate the EFI grub.cfg under /boot/efi/EFI/<distro>/grub.cfg.
# On RHEL / CentOS / TencentOS / Rocky / AlmaLinux with UEFI, the active
# grub config lives in the EFI System Partition rather than /boot/grub2.
find_efi_grub_cfg() {
    local efi_dir="/boot/efi/EFI"
    [[ -d "$efi_dir" ]] || return 1

    # Prefer well-known distro directories in a sensible order.
    local candidate
    for candidate in tencentos redhat centos rocky almalinux openEuler anolis fedora ubuntu debian kylin; do
        if [[ -f "$efi_dir/$candidate/grub.cfg" ]]; then
            echo "$efi_dir/$candidate/grub.cfg"
            return 0
        fi
    done

    # Fallback: pick the first grub.cfg found under /boot/efi/EFI/*.
    local cfg
    cfg=$(find "$efi_dir" -maxdepth 2 -name grub.cfg -type f 2>/dev/null | head -n1)
    if [[ -n "$cfg" ]]; then
        echo "$cfg"
        return 0
    fi
    return 1
}

# Read the existing GRUB_CMDLINE_LINUX value from /etc/default/grub,
# or treat it as empty if it is not present.
existing_cmdline=""
if grep -qE '^GRUB_CMDLINE_LINUX=' /etc/default/grub; then
    existing_cmdline=$(grep -E '^GRUB_CMDLINE_LINUX=' /etc/default/grub | head -n1 \
        | sed -E 's/^GRUB_CMDLINE_LINUX=//; s/^"(.*)"$/\1/; s/^'\''(.*)'\''$/\1/')
fi

# Merge the appended parameters into the existing cmdline. De-duplicate by
# key, and let appended parameters override existing parameters with the same key.
merged_cmdline="$existing_cmdline"
for param in $GRUB_CMDLINE_LINUX_APPEND; do
    key="${param%%=*}"
    # Drop existing parameters with the same key, in either key or key=value form.
    merged_cmdline=$(printf '%s\n' "$merged_cmdline" | tr -s ' ' '\n' \
        | awk -v key="$key" 'NF { split($0, parts, "="); if (parts[1] != key) print }' \
        | tr '\n' ' ')
    merged_cmdline="${merged_cmdline% } $param"
done
# Normalize extra whitespace.
merged_cmdline=$(echo "$merged_cmdline" | tr -s ' ' | sed -E 's/^ +//; s/ +$//')

if grep -qE '^GRUB_CMDLINE_LINUX=' /etc/default/grub; then
    sed -i "/^GRUB_CMDLINE_LINUX=/c\\GRUB_CMDLINE_LINUX=\"$merged_cmdline\"" /etc/default/grub
else
    echo "GRUB_CMDLINE_LINUX=\"$merged_cmdline\"" >> /etc/default/grub
fi

# Regenerate the grub configuration. When running under UEFI on RHEL-family
# distributions, the active config is /boot/efi/EFI/<distro>/grub.cfg, not
# /boot/grub2/grub.cfg; Debian/Ubuntu's update-grub handles both modes itself.
regen_grub_cfg() {
    local target="$1"
    # Back up existing grub.cfg if present (best-effort).
    if [[ -f "$target" ]]; then
        cp -a "$target" "${target}.bak.$(date +%s)" || true
    fi
    mkdir -p "$(dirname "$target")"
    if command -v grub2-mkconfig >/dev/null 2>&1; then
        grub2-mkconfig -o "$target"
    else
        grub-mkconfig -o "$target"
    fi
}

if command -v update-grub >/dev/null 2>&1; then
    # Ubuntu / Debian: update-grub internally detects BIOS vs UEFI and writes
    # to the correct location (/boot/grub/grub.cfg or the EFI partition).
    update-grub
elif command -v grub2-mkconfig >/dev/null 2>&1 || command -v grub-mkconfig >/dev/null 2>&1; then
    # RHEL / CentOS / TencentOS / Rocky / AlmaLinux / openEuler / Kylin ...
    if [[ $is_uefi -eq 1 ]]; then
        efi_cfg=$(find_efi_grub_cfg || true)
        if [[ -n "${efi_cfg:-}" ]]; then
            echo "Detected UEFI grub config: $efi_cfg"
            regen_grub_cfg "$efi_cfg"
        else
            echo "WARNING: UEFI detected but no grub.cfg found under /boot/efi/EFI/*." >&2
            echo "Falling back to /boot/grub2/grub.cfg (may not take effect on UEFI)." >&2
            if [[ -d /boot/grub2 ]]; then
                regen_grub_cfg /boot/grub2/grub.cfg
            else
                regen_grub_cfg /boot/grub/grub.cfg
            fi
        fi
    else
        # Legacy BIOS
        if [[ -d /boot/grub2 ]]; then
            regen_grub_cfg /boot/grub2/grub.cfg
        else
            regen_grub_cfg /boot/grub/grub.cfg
        fi
    fi
else
    echo "ERROR: no grub config tool found (update-grub / grub2-mkconfig / grub-mkconfig)" >&2
    exit 1
fi

[kangcp]

@lkml-likexu 您好，我装好PVM内核后，然后执行上面您这边脚本，依然无法重启，但是我拿到了日志，需要麻烦您这边帮忙看看哈

i-0d2297f14f0ebd5ca.log

[lkml-likexu]

@kangcp According to the logs, the /EFI/amzn/grubenv directory on the EFI partition is missing, causing GRUB to fail and drop into the rescue shell. I’m not sure what customizations amzn has made to GRUB, but please verify that you can switch to other non-default kernels using the same GRUB configuration. For the pvm kernel, the GRUB changes are minimal (only requiring pti=off no5lvl); in your environment, you may also need selinux=0 audit=0 (in a test environment).

Or you may try nested KVM on aws, https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/amazon-ec2-nested-virtualization.html