Fix and improve VTL switching loop and state management by sangho2 · Pull Request #545 · microsoft/litebox

sangho2 · 2025-12-09T01:50:42Z

This PR fixes some bugs in the VTL switching loop and improve its reliability and performance.
Specifically, this PR uses assembly code to carefully switch between VTLs while saving/restoring
VTL0 registers. This PR removes some unreliable stack manipulation code we had before.
In addition, this PR makes it clear that we no longer store the VTL1 state in per-core storage.

Note: this PR and #541 should be merged together since #541 suffers from a crash due to
some bugs that this PR is fixing.

wdcui

Thank you for improving the code, Sangho. I left a couple of comments below. Please wait for @jstarks 's review.

litebox_platform_lvbs/src/host/per_cpu_variables.rs

litebox_platform_lvbs/src/mshv/vtl_switch.rs

wdcui

It's not clear to me (1) why we have both per_cpu and ktls and (2) why we load/store vtl0 state from per_cpu via stack. I left some other minor comments.

wdcui · 2026-01-18T23:03:12Z

litebox_platform_lvbs/src/mshv/vtl_switch.rs

+/// `fn_save_state` is the function to save VTL state stored in the stack.
+/// `scratch_off` is the offset of the kernel TLS which holds the scratch space to
+/// save/restore the current stack pointer.
+macro_rules! SAVE_VTL_STATE_ASM {


Code like this should be moved out of mshv at some point?

Do we need to handle xsave here?

xsave and xstore are handled inside vtl_switch_loop_body because we don't particularly need assembly code for it.

wdcui · 2026-01-18T23:06:33Z

litebox_platform_lvbs/src/mshv/vtl_switch.rs

+            stringify!($scratch_off),
+            "]\n", // anchor
+            "sub rsp, ",
+            stringify!($state_size),


Is the rsp value after line 159 the same as after the call?

We save and restore rsp in Lines 156 and 177, respectively, using scrachpad.

wdcui · 2026-01-18T23:09:26Z

litebox_platform_lvbs/src/mshv/vtl_switch.rs

+        "mov rsp, gs:[{kernel_sp_off}]", // reset kernel stack pointer. Hyper-V saves/restores rsp and rip.
+        VTL_RETURN_ASM!({vtl_ret_addr_off}),
+        // *** VTL1 resumes here regardless of the entry reason (VTL switch or intercept) ***
+        SAVE_VTL_STATE_ASM!({save_vtl0_state}, {scratch_off}),


Can we save the vtl0 state to the per cpu storage directly?

As far as I know, using stack as temporary storage to save the current registers is one of the most reliable and efficient ways.

wdcui · 2026-01-18T23:11:33Z

litebox_platform_lvbs/src/mshv/vtl_switch.rs

+    // is due to memory or register access violation, the VTL0 kernel might not have saved
+    // its states), we conservatively save and restore its extended states on every VTL switch.
+    with_per_cpu_variables_mut(|per_cpu_variables| {
+        per_cpu_variables.save_extended_states(HV_VTL_NORMAL);


How do we make sure all the code executed before we get to this point doesn't change the extended state?

It only executes some assembly code and memory copy operations. I agree that some compiler optimization might be able to use SSE though.

litebox_platform_lvbs/src/syscall_entry.rs

sangho2 · 2026-01-19T04:11:52Z

It's not clear to me (1) why we have both per_cpu and ktls and (2) why we load/store vtl0 state from per_cpu via stack. I left some other minor comments.

The main reason I introduced KernelTls is that, assembly code cannot easily access PerCpuVariables because it is protected by RefCell (i.e., this is for Rust code not assembly code). KernelTls is named after ShimTls but at this point this is confusing because we no longer have ShimTls. Let me rename it (PerCpuVariablesAsm or something else).
I think using stack is most reliable and efficient way to save the current CPU registers. Also, as noted above, PerCpuVariables is protected by RefCell so assembly code cannot simply access it.

praveen-pk · 2026-01-21T16:20:48Z

this PR makes it clear that we no longer store the VTL1 state in per-core storage.

Is this being dropped because all VTL1 invocations can start with a fresh stack for now. Should this change be revisited once dynamic TAs are enabled?

litebox_platform_lvbs/src/host/per_cpu_variables.rs

sangho2 · 2026-01-21T17:39:28Z

this PR makes it clear that we no longer store the VTL1 state in per-core storage.

Is this being dropped because all VTL1 invocations can start with a fresh stack for now. Should this change be revisited once dynamic TAs are enabled?

We shouldn't use per-core storage for saving the VTL1 state because, as discussed, we can't ensure that VTL1 is re-entered with the same core. We need a separate global data structure for this.

litebox_platform_lvbs/src/mshv/vtl_switch.rs

sangho2 · 2026-01-22T05:46:23Z

Let me add XSAVE/XRSTORE when I merge #541. doing this in this PR is a bit tricky due to complicated rebase.

This PR adds `run_thread` support for LVBS platform and revise its syscall handling routine based on Linux userland platform's syscall handling. It has suppressed new page table creation for now because it needs to be done by the runner via an upcall (userspace support still exists, but doesn't create a separate one for each TA instance for now). --------- Co-authored-by: Sangho Lee <sanghle@microsoft.com>

tgopinath-microsoft

approved with comments

sangho2 marked this pull request as ready for review December 9, 2025 05:39

wdcui requested review from CvvT and jstarks December 9, 2025 22:18

wdcui reviewed Dec 9, 2025

View reviewed changes

litebox_platform_lvbs/src/host/per_cpu_variables.rs Outdated Show resolved Hide resolved

litebox_platform_lvbs/src/mshv/vtl_switch.rs Outdated Show resolved Hide resolved

sangho2 force-pushed the sanghle/lvbs/ktls branch from bd87679 to a298043 Compare December 18, 2025 22:46

Sangho Lee added 5 commits December 18, 2025 22:47

fix vtl switch loop. use push/pop for better perf

853c118

drop unneeded vtl1_state. revision

0e9a2ce

alignment related improvement

23e54e6

revise macro

321f45c

addressed comments

1cbad25

sangho2 force-pushed the sanghle/lvbs/fix_vtl_switch branch from 898c3d5 to 1cbad25 Compare December 18, 2025 22:47

wdcui reviewed Jan 18, 2026

View reviewed changes

praveen-pk reviewed Jan 21, 2026

View reviewed changes

litebox_platform_lvbs/src/host/per_cpu_variables.rs Show resolved Hide resolved

sangho2 mentioned this pull request Jan 21, 2026

Add run_thread to LVBS platform and revise its syscall handling #566

Merged

CvvT reviewed Jan 21, 2026

View reviewed changes

litebox_platform_lvbs/src/mshv/vtl_switch.rs Outdated Show resolved Hide resolved

litebox_platform_lvbs/src/mshv/vtl_switch.rs Outdated Show resolved Hide resolved

litebox_platform_lvbs/src/mshv/vtl_switch.rs Outdated Show resolved Hide resolved

sangho2 mentioned this pull request Jan 22, 2026

Raise GP fault in VTL0 if VTL1 panics #586

Merged

KernelTls -> PerCpuVariablesAsm

a15fbbb

sangho2 force-pushed the sanghle/lvbs/fix_vtl_switch branch from f7e7331 to 9f80306 Compare January 22, 2026 05:33

simplify vtlstate save/load macros

c85aac5

sangho2 force-pushed the sanghle/lvbs/fix_vtl_switch branch from 9f80306 to c85aac5 Compare January 22, 2026 05:34

sangho2 merged commit dc8b467 into sanghle/lvbs/ktls Jan 22, 2026
4 of 8 checks passed

sangho2 deleted the sanghle/lvbs/fix_vtl_switch branch January 22, 2026 21:48

sangho2 restored the sanghle/lvbs/fix_vtl_switch branch January 23, 2026 03:54

tgopinath-microsoft reviewed Jan 26, 2026

View reviewed changes

Conversation

sangho2 commented Dec 9, 2025

Uh oh!

wdcui left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

wdcui left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

sangho2 commented Jan 19, 2026

Uh oh!

praveen-pk commented Jan 21, 2026

Uh oh!

Uh oh!

sangho2 commented Jan 21, 2026

Uh oh!

Uh oh!

Uh oh!

Uh oh!

sangho2 commented Jan 22, 2026

Uh oh!

Uh oh!

tgopinath-microsoft left a comment

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants