VMware: template VMs not removed during storage GC/cleanup

[andrijapanicsb]

Current master (testing specifically PR #4250), vSphere 6.7 (but I believe not important, as the same has been observed in earlier VMware versions)

Issue: when all VMs from a specific template are deleted from a particular Primary Storage (Datastore), the template should be also removed from the Primary Storage after the storage GC kicks in - but this never happens, and template-VMs (with UUID-like names) are left on the Primary Storage, consuming space and junk-ing around.

[yadvr]

With @Spaceman1984 to attempt

[Spaceman1984]

I'm taking a look at this

[yadvr]

cc #4307 now

[andrijapanicsb]

this is true for every ACS release in past few years
deploy a VM from a new template, notice the template-VM created (UUID-like name).
destroy all VMs from that template
reduce the storage cleanup period (global settings to i..e. 60sec) and restart mgmt
once it runs, nothing is removed from the primarty storage, though records ARE removed from the template_spool_ref

[Spaceman1984]

I found the code where the problem is in VmwareResource.java.

public Answer execute(DestroyCommand cmd) { ...

if (vmMo != null && vmMo.isTemplate()) {
    if (s_logger.isInfoEnabled()) {
        s_logger.info("Destroy template volume " + vol.getPath());
    }
    vmMo.destroy();
} else {
    if (s_logger.isInfoEnabled()) {
        s_logger.info("Template volume " + vol.getPath() + " is not found, no need to delete.");
    }
}

Volumes will never be deleted by looking at this. (if (vmMo != null && vmMo.isTemplate()) {)

The specific line of code was added with this issue https://issues.apache.org/jira/browse/CLOUDSTACK-8405
I will have to run through this issue to fix it correctly I think the logic for deciding whether to delete a volume or template should be in a different place and not here.

[sureshanaparti]

@Spaceman1984 vmMo's objects can be never be a volume, it is either a VM or Template.

[Spaceman1984]

Ok, @sureshanaparti, it's not a volume, it's a VM, but not a template and it is failing on that line of code. If it is supposed to be a template, then that could be the issue. I will need to do more digging.