[BUG] "waagent -deprovision+user" does not remove /var/lib/hyperv/, resulting in stale kvp_pool files in a new VM

[dcui]

When preparing a Linux VHD image for Azure, we usually run "sudo waagent -deprovision+user" (see https://learn.microsoft.com/en-us/azure/virtual-machines/generalize) and assume that the VHD won't contain any stale info from the previous VM.

It turns out this assumption is not true because walinuxagent doesn't remove the folder /var/lib/hyperv/, resulting in stale kvp_pool files in a new VM:
/var/lib/hyperv/.kvp_pool_0 /var/lib/hyperv/.kvp_pool_1 /var/lib/hyperv/.kvp_pool_2 /var/lib/hyperv/.kvp_pool_3 /var/lib/hyperv/.kvp_pool_4

Some of the pool files may contain stale and incorrect info, e.g. /var/lib/hyperv/.kvp_pool_3 contains the Windows host name. It looks like under some condition the KVP daemon fails to update the host name in /var/lib/hyperv/.kvp_pool_3 before the VM's first reboot (the root cause is still unclear yet, but it's a different issue); when this happens, a user may be concerned as the host name may be used to implement some kind of load balancing, and an unexpected incorrect host name in the kvp pool file can mislead the load balancing code.

The request is: "waagent -deprovision+user" should remove /var/lib/hyperv/.

Steps to reproduce the behavior:

1. Start a Linux VM of any Linux distro on Azure or a local Hyper-V host, e.g. Ubutnu/RHEL/SLES. Make sure the KVP daemon is running, and you see the files "ls /var/lib/hyperv/.kvp_pool_*".
2. Run "sudo waagent -deprovision+user".
3. "ls /var/lib/hyperv/.kvp_pool_*" still shows the kvp pool files.

This behavior exists in all versions of WALinuxAgent.

[maddieford]

@longlimsft, I see rdma logic in the agent uses these kvp files to get driver info, etc:

WALinuxAgent/azurelinuxagent/pa/rdma/rdma.py

Line 88 in 5fcca0f

driver_info_source = '/var/lib/hyperv/.kvp_pool_0'

Agent team doesn't have expertise on this usage, or the impact of removing this directory during deprovisioning.
Assigning this issue to RDMA team to review.