UEFI Support on CloudStack #3638

pavanaravapalli · 2019-10-16T12:30:15Z

Description

To Enable UEFI Support for Guest VM's deployed on Hypervisors[ KVM, VMware] in Cloud Stack.

Known things.

UEFI Capability for existing hosts [Vmware, KVM] is not supported, this marked for future enhancement.
KVM Host has dependancy on two pre-check conditions 1. OVMF package 2. uefi.properties configured properly or not.
Cloud Admin is responsible for updating newly added KVM host(s) with relevant uefi.properties. If mismatch in the config params of uefi.properties , UEFI deploy VM will throw libvirt exception.
For normal template selected and deployed with UEFI [legacy, secure] options , There won’t be any Exception thrown by Cloud Stack. VM will be in Running state in Cloud Stack, where as actually it’s ended up booting issue’s on Host due to wrong template/OS deployed to UEFI firmware. Also vice versa also can be happend. This is known behaviour, and uefi enable intelligence can be addressed in the next enhancement when it’s planned.
Cloud Admin is responsible to prepare Guest VM template which enabled with UEFI support.
Make sure to deploy a VM manually on Hypervisor and verify UEFI enabled template , prior to on-boarding to CloudStack.

Refer below doc for more information.
https://cwiki.apache.org/confluence/display/CLOUDSTACK/Enable+UEFI+booting+for+Instance

Types of changes

Breaking change (fix or feature that would cause existing functionality to change)
New feature (non-breaking change which adds functionality)
Bug fix (non-breaking change which fixes an issue)
Enhancement (improves an existing feature and functionality)
Cleanup (Code refactoring and cleanup, that may add test cases)

Screenshots (if appropriate):

KVM Host configuration

Install OVMF package by following instruction from the link here
Create uefi.properties file at location /etc/cloudstack/agent/uefi.properties
insert loader details with respective to the ovmf package would like to integrate. for example

How Has This Been Tested?

Hypervisor	UEFI Legacy	UEFI Secure	Manual Dev Testing Find outs [Linux & Windows ] with Uefi Legacy & Secure templates
KVM RHEL / CentOS (7.x)	Yes	Hypervisor Not Supported	Passed
KVM RHEL/ CentOS (8.x)	Yes	Yes	Passed
Vmware ESXI(5.5)	Yes	Hypervisor Not Supported	Passed
VMware ESXI(6.5)	Yes	Yes	Passed

KVM

Prepared KVM [ RHEL 8 ] host UEFI enabled by installing OVMF package and configuring libvirt with nvram params
Installed cloud stack agent rpm on KVM host.
Created & updated uefi.properties configuration file with UEFI params as describe above in "KVM Host Configuration".
Added KVM Host to a Cluster in Cloud Stack.
Prepared & registered 'Windows Server 2016' & 'Cent OS 7.x' [KVM] templates on to CloudStack. i.e UEFI enabled.
Deployed a VM [ boot options as UEFI -> Secure , UEFI -> Legacy ]with Windows Server 2016
template and verified the UEFI boot information.
Deployed a VM [ boot options as UEFI -> Secure , UEFI -> Legacy ]with Cent OS 7.x template and verified the UEFI boot information.

Note: brctl-utils package is deprecated in RHEL 8. We have force installed epel7 rpm in RHEL 8 and configured KVM agent.

VM Booting with UEFI firmware

Windows Server 2016 VM Secure boot enabled

Windows Server 2016 VM Secure boot disabled

Cent OS 7 Linux : Secure boot enabled

VMware

Added VMware ESXi[6.5] Host in Cloud Stack Vmware Cluster
Prepared & registered 'Windows Server 2016' [Vmware] templates on to CloudStack. i.e UEFI
enabled.
Deployed a VM [ boot options as UEFI -> Secure , UEFI -> Legacy ]with Windows Server 2016
template and verified the UEFI boot information.

Windows Server 2106 VM Secure boot disabled

Windows Server 2106 VM Secure boot enabled

UEFI Enabled Host

Guest VM Uefi Boot Details

...ns/hypervisors/vmware/src/main/java/com/cloud/hypervisor/vmware/resource/VmwareResource.java

nathanejohnson · 2019-10-16T15:08:51Z

@pavanaravapalli first off glad to see this PR! it doesn't seem to compile just now, I marked in the review where it's complaining.

Here is a log of

mvn install -Dnoredist

compile.log

nvazquez

Thanks @pavanaravapalli. I left some code review comments. Can you also add documentation? For example for KVM it seems to need some properties set in order to work properly

api/src/main/java/org/apache/cloudstack/api/command/user/vm/DeployVMCmd.java

api/src/main/java/org/apache/cloudstack/api/response/HostResponse.java

...ypervisors/kvm/src/main/java/com/cloud/hypervisor/kvm/resource/LibvirtComputingResource.java

nvazquez · 2019-10-16T23:40:08Z

Hi @nathanejohnson are you planning to test this PR?

nathanejohnson · 2019-10-17T02:59:30Z

@nvazquez I'm going to test on kvm. would be good to get a volunteer for vmware as well

nvazquez · 2019-10-17T13:07:26Z

@nathanejohnson sure, I can manually test it on Vmware once the PR is complete
@pavanaravapalli please let us know when this is complete, afaik it is still in progress right?

pavanaravapalli · 2019-10-18T09:20:16Z

@nvazquez I am working on it. I will address your review comments given in the PR#3643. I have created another PR#3643 for the same branch due to some forking issue.

@nathanejohnson @nvazquez
Please refer the PR#3643 for future reference.
I am closing this PR

ustcweizhou · 2019-10-18T09:28:05Z

@pavanaravapalli you do not need to create new pull request.
you can push your new commits (or force-push to overwrite existing commit) to your github.

pavanaravapalli · 2019-11-18T06:43:27Z

Since PR#3643's forked branch been obsoleted, re-opening this PR.

pavanaravapalli · 2019-11-20T13:44:06Z

Thanks @pavanaravapalli. I left some code review comments. Can you also add documentation? For example for KVM it seems to need some properties set in order to work properly

Updated Description with KVM related uefi.properties information.
Also refer to the Design doc for more information.

pavanaravapalli · 2019-11-22T06:47:05Z

@nvazquez @nathanejohnson
PR is in good shape now, Can you test functionality and provide review comments.

yadvr · 2019-12-07T22:25:30Z

@blueorangutan package

blueorangutan · 2019-12-07T22:26:27Z

@rhtyd a Jenkins job has been kicked to build packages. I'll keep you posted as I make progress.

blueorangutan · 2019-12-07T23:10:58Z

Packaging result: ✖centos6 ✖centos7 ✖debian. JID-435

...on/src/main/java/org/apache/cloudstack/engine/cloud/entity/api/VirtualMachineEntityImpl.java

pavanaravapalli · 2019-12-17T14:38:48Z

@sureshanaparti, review comments addressed as suggested

sureshanaparti

Checked the changes for UEFI support. Code changes LGTM.

server/src/main/java/com/cloud/agent/manager/allocator/impl/FirstFitAllocator.java

server/src/main/java/com/cloud/deploy/FirstFitPlanner.java

server/src/main/java/com/cloud/hypervisor/HypervisorGuruBase.java

server/src/main/java/com/cloud/server/ManagementServerImpl.java

api/src/main/java/org/apache/cloudstack/api/command/user/vm/DeployVMCmd.java

pavanaravapalli · 2020-03-13T17:16:35Z

@pavanaravapalli I don't think you had anymore functional changes to make so it is fine by me to leave thos fixes for a new PR after this one.

Ok.

blueorangutan · 2020-03-13T19:54:42Z

Trillian test result (tid-1241)
Environment: kvm-centos7 (x2), Advanced Networking with Mgmt server 7
Total time taken: 33201 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr3638-t1241-kvm-centos7.zip
Smoke tests completed. 83 look OK, 0 have error(s)
Only failed tests results shown below:

Test	Result	Time (s)	Test File

andrijapanicsb · 2020-03-13T19:56:12Z

3 x Approvals, regression tests went fine.

Merging

yadvr · 2020-03-14T09:31:17Z

engine/orchestration/src/main/java/com/cloud/vm/VirtualMachineManagerImpl.java

                }

                final VirtualMachineProfileImpl vmProfile = new VirtualMachineProfileImpl(vm, template, offering, owner, params);
+                s_logger.info(" Uefi params " + "UefiFlag: " + params.get(VirtualMachineProfile.Param.UefiFlag)


Could this potentially cause any exception? Should this be only logged when custom params are passed; otherwise it may always log as null/null confuse admin/users cc @pavanaravapalli @DaanHoogland @andrijapanicsb

yadvr · 2020-03-14T09:31:46Z

api/src/main/java/org/apache/cloudstack/api/command/user/vm/DeployVMCmd.java

                }
            }
        }
+        if(getBootType() != null){ // export to get


@pavanaravapalli lint issue - add space between if (...

yadvr · 2020-03-14T09:32:27Z

api/src/main/java/org/apache/cloudstack/api/command/user/vm/DeployVMCmd.java

            }
        }
+        if(getBootType() != null){ // export to get
+            if(getBootType() == ApiConstants.BootType.UEFI) {


@pavanaravapalli cc @DaanHoogland @andrijapanicsb
There is no null check, write it as someEnum.equals(someVariable or Method()) - this will never throw NPE.

yadvr · 2020-03-14T09:32:50Z

api/src/main/java/org/apache/cloudstack/api/command/user/vm/DeployVMCmd.java

        }
+        if(getBootType() != null){ // export to get
+            if(getBootType() == ApiConstants.BootType.UEFI) {
+                customparameterMap.put(getBootType().toString(), getBootMode().toString());


@pavanaravapalli cc @DaanHoogland @andrijapanicsb
What is one of the methods return null?

yadvr · 2020-03-14T09:33:15Z

api/src/main/java/org/apache/cloudstack/api/response/HostResponse.java

        detailsCopy.remove("username");
        detailsCopy.remove("password");

+        if(detailsCopy.containsKey(Host.HOST_UEFI_ENABLE)) {


@pavanaravapalli space between if (...

yadvr · 2020-03-14T09:34:43Z

...tration/src/main/java/org/apache/cloudstack/engine/cloud/entity/api/VMEntityManagerImpl.java

        vmProfile.setServiceOffering(_serviceOfferingDao.findByIdIncludingRemoved(vm.getId(), vm.getServiceOfferingId()));
+        if (MapUtils.isNotEmpty(vmEntityVO.getDetails()) &&
+                vmEntityVO.getDetails().containsKey(VirtualMachineProfile.Param.UefiFlag.getName()) &&
+                "yes".equalsIgnoreCase(vmEntityVO.getDetails().get(VirtualMachineProfile.Param.UefiFlag.getName())))


@pavanaravapalli
Can you make this simpler - just put in the key in the details only when it's applicable, then you don't need to compare the uefi value at all.

yadvr · 2020-03-14T09:37:42Z

...ypervisors/kvm/src/main/java/com/cloud/hypervisor/kvm/resource/LibvirtComputingResource.java

+            guest.setBootType(GuestDef.BootType.UEFI);
+            guest.setBootMode(GuestDef.BootMode.LEGACY);
+            if (StringUtils.isNotBlank(customParams.get(GuestDef.BootType.UEFI.toString())) && "secure".equalsIgnoreCase(customParams.get(GuestDef.BootType.UEFI.toString()))) {
+                guest.setMachineType("q35");


@pavanaravapalli cc @DaanHoogland @andrijapanicsb Unless there is a requirement of allowing customisation of UEFI (generally people use whatever libvirt gives), you can simplify this by addressing the change in the libvirt xml. See my changes around support EFI for arm64/raspberrypi (for example https://github.com/apache/cloudstack/pull/3644/files#diff-042ba018afca0d690feae46e22f73301R128)

@rhtyd

UEFI Secure boot does not work with default libvirt xml changes and "pc" as machine type.

There are few other mandatory changes to be handled in case of secure boot guest xml, the same have been captured in the design doc. please have a look in the wiki doc shared with this PR description.

DaanHoogland · 2020-03-19T12:32:36Z

@pavanaravapalli please have a look at @rhtyd 's concerns and at #3978 as this PR seems to cause issue. We'll need to fix before release or revert!
cc @davidjumani @rhtyd @andrijapanicsb

DaanHoogland · 2020-03-19T12:33:11Z

cc @nathanejohnson @kiwiflyer , hate to escalate but please see my prior comment.

pavanaravapalli · 2020-03-19T18:05:53Z

@pavanaravapalli please have a look at @rhtyd 's concerns and at #3978 as this PR seems to cause issue. We'll need to fix before release or revert!
cc @davidjumani @rhtyd @andrijapanicsb

@DaanHoogland
I will check for the exception #3978 cause and update.
For addressing @rhtyd concern i need some more time as i am occupied with other activities.

yadvr · 2020-03-21T10:39:02Z

@pavanaravapalli could est. when would you be able to do this, preferably before 4.14 RC1?

pavanaravapalli · 2020-03-22T14:18:11Z

@pavanaravapalli could est. when would you be able to do this, preferably before 4.14 RC1?

@rhtyd it may take time to address all the review comments provided. I am hoping to give fix for NPE #3978 soon. And i need some one to test it as i don't have VMware setup with me present.

DaanHoogland · 2020-03-23T08:03:27Z

@pavanaravapalli I can take care of the test required to validate that #3978 is solved. Please add your update A.S.A.P.

DaanHoogland · 2020-03-23T13:58:12Z

@pavanaravapalli please add your improvements and suggestions as PR to #3983.

pavanaravapalli · 2020-03-23T17:30:59Z

@pavanaravapalli please add your improvements and suggestions as PR to #3983.

Raised PR #3985 for the fix . @DaanHoogland please verify the fix and approve it.

DaanHoogland · 2020-05-18T12:06:16Z

@pavanaravapalli please review #4088 we need to fix this to be able to release.

sureshanaparti · 2020-05-18T13:06:58Z

@pavanaravapalli Can you update the actual UEFI functionality (with boot mode & type) supported for VMware and KVM, in CloudStack, with this PR changes.

pavanaravapalli · 2020-05-18T15:40:33Z

@pavanaravapalli Can you update the actual UEFI functionality (with boot mode & type) supported for VMware and KVM, in CloudStack, with this PR changes.

Updated Description with info.

pavanaravapalli · 2020-05-18T15:44:13Z

@pavanaravapalli please review #4088 we need to fix this to be able to release.

Verified code for #4089 , looks good to me. I have not encountered this issue while testing UEFI changes.

…ypervisor KVM,VMware. enabled boot modes [Legacy,Secure] support for UEFI boot with known caveats. (apache#3638)" This reverts commit d4b537e. Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>

nathanejohnson requested changes Oct 16, 2019

View reviewed changes

...ns/hypervisors/vmware/src/main/java/com/cloud/hypervisor/vmware/resource/VmwareResource.java Outdated Show resolved Hide resolved

nvazquez reviewed Oct 16, 2019

View reviewed changes

pavanaravapalli force-pushed the uefisupport branch from 6f3c77d to dd98b3b Compare October 16, 2019 18:17

svenvogel added type:new-feature component:compute labels Oct 17, 2019

pavanaravapalli closed this Oct 18, 2019

pavanaravapalli reopened this Nov 18, 2019

pavanaravapalli force-pushed the uefisupport branch 2 times, most recently from f69ee48 to 2f8300d Compare November 20, 2019 13:36

pavanaravapalli changed the title ~~WIP: UEFI Support on CloudStack~~ UEFI Support on CloudStack Nov 20, 2019

yadvr added this to the 4.14.0.0 milestone Dec 7, 2019

sureshanaparti reviewed Dec 10, 2019

View reviewed changes

...on/src/main/java/org/apache/cloudstack/engine/cloud/entity/api/VirtualMachineEntityImpl.java Outdated Show resolved Hide resolved

pavanaravapalli force-pushed the uefisupport branch 2 times, most recently from 9758867 to 59bf8d3 Compare December 13, 2019 14:25

pavanaravapalli mentioned this pull request Dec 17, 2019

WIP: UEFI Support on CloudStack #3643

Closed

4 tasks

pavanaravapalli force-pushed the uefisupport branch from 59bf8d3 to ac94684 Compare December 17, 2019 14:19

sureshanaparti approved these changes Dec 17, 2019

View reviewed changes

DaanHoogland mentioned this pull request Mar 13, 2020

fix logs and modularisation leftovers from #3638 #3966

Closed

andrijapanicsb merged commit d4b537e into apache:master Mar 13, 2020

yadvr reviewed Mar 14, 2020

View reviewed changes

DaanHoogland mentioned this pull request Mar 23, 2020

UEFI fixes and comments addressed #3983

Closed

5 tasks

yadvr mentioned this pull request May 19, 2020

Revert: UEFI feature from #3638 and #3985 #4091

Closed

GabrielBrascher mentioned this pull request May 19, 2020

UEFI boot in Legacy mode corrupts VSphere connection. #4088

Closed

slavkap mentioned this pull request Feb 5, 2021

uefi vm creation fail #4238

Closed

Noelantogerorge mentioned this pull request Jun 6, 2023

Existing VM migrated from proxmox to cloudstack not working #7586

Closed

UEFI Support on CloudStack #3638

UEFI Support on CloudStack #3638

Uh oh!

Conversation

pavanaravapalli commented Oct 16, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Types of changes

Screenshots (if appropriate):

How Has This Been Tested?

Uh oh!

Uh oh!

nathanejohnson commented Oct 16, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

nvazquez left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

nvazquez commented Oct 16, 2019

Uh oh!

nathanejohnson commented Oct 17, 2019

Uh oh!

nvazquez commented Oct 17, 2019

Uh oh!

pavanaravapalli commented Oct 18, 2019

Uh oh!

ustcweizhou commented Oct 18, 2019

Uh oh!

pavanaravapalli commented Nov 18, 2019

Uh oh!

pavanaravapalli commented Nov 20, 2019

Uh oh!

pavanaravapalli commented Nov 22, 2019

Uh oh!

yadvr commented Dec 7, 2019

Uh oh!

blueorangutan commented Dec 7, 2019

Uh oh!

blueorangutan commented Dec 7, 2019

Uh oh!

Uh oh!

pavanaravapalli commented Dec 17, 2019

Uh oh!

sureshanaparti left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

pavanaravapalli commented Mar 13, 2020

Uh oh!

blueorangutan commented Mar 13, 2020

Uh oh!

andrijapanicsb commented Mar 13, 2020

Uh oh!

yadvr Mar 14, 2020

Choose a reason for hiding this comment

Uh oh!

yadvr Mar 14, 2020

Choose a reason for hiding this comment

Uh oh!

yadvr Mar 14, 2020

Choose a reason for hiding this comment

Uh oh!

yadvr Mar 14, 2020

Choose a reason for hiding this comment

Uh oh!

yadvr Mar 14, 2020

Choose a reason for hiding this comment

Uh oh!

yadvr Mar 14, 2020

Choose a reason for hiding this comment

Uh oh!

pavanaravapalli commented Oct 16, 2019 •

edited

Loading

nathanejohnson commented Oct 16, 2019 •

edited

Loading

pavanaravapalli Mar 19, 2020 •

edited

Loading

pavanaravapalli commented May 18, 2020 •

edited

Loading