Vmware Import Problem

[omurozlu]

ISSUE TYPE

• Bug Report

COMPONENT NAME

Vmware

CLOUDSTACK VERSION

4.19

SUMMARY

Storage is not seen while importing [ 4.18.1.0 Working.]

[sureshanaparti]

@omurozlu Is that storage added as primary storage in CloudStack, cluster-wide or zone-wide?

[omurozlu]

hello @sureshanaparti,

I tried it both ways. (zone + cluster)

[sureshanaparti]

Hi @omurozlu I did quick code check, seems issue with datastore path comparison here

cloudstack/server/src/main/java/org/apache/cloudstack/vm/UnmanagedVMsManagerImpl.java

Line 554 in 45d267c

if (StringUtils.contains(pool.getPath(), dsPath)) {

(updated in #8445)

[omurozlu]

hello @sureshanaparti,
im check api post process.
Do I need a manual process to do this?

[sureshanaparti]

hello @sureshanaparti, im check api post process. Do I need a manual process to do this?

@omurozlu Manual process does the same comparison, while importing root and data disks of the instance. Is root disk of the instance, exists on that storage?

[nvazquez]

Hi @omurozlu - is the storage pool a datastore cluster type? Can you upload the management server logs for the import operation?

[omurozlu]

hello @sureshanaparti,
same zone added

hello @nvazquez ,
datastore cluster type VMFS

management server log this :

2024-02-28 15:48:59,347 DEBUG [c.c.a.ApiServer] (qtp1278254413-352:ctx-5219e9e4 ctx-3aeefedf) (logid:15f58092) CIDRs from which account 'Account [{"accountName":"admin","id":2,"uuid":"683aa0c3-d606-11ee-8b6a-0050568d528d"}]' is allowed to perform API calls: 0.0.0.0/0,::/0
2024-02-28 15:48:59,349 INFO  [o.a.c.a.DynamicRoleBasedAPIAccessChecker] (qtp1278254413-352:ctx-5219e9e4 ctx-3aeefedf) (logid:15f58092) Account [Account [{"accountName":"admin","id":2,"uuid":"683aa0c3-d606-11ee-8b6a-0050568d528d"}]] is Root Admin or Domain Admin, all APIs are allowed.
2024-02-28 15:48:59,350 WARN  [o.a.c.a.ProjectRoleBasedApiAccessChecker] (qtp1278254413-352:ctx-5219e9e4 ctx-3aeefedf) (logid:15f58092) Project is null, ProjectRoleBasedApiAccessChecker only applies to projects, returning API [importUnmanagedInstance] for user [User {"username":"admin","uuid":"683bac0e-d606-11ee-8b6a-0050568d528d"}.] as allowed.
2024-02-28 15:48:59,351 DEBUG [o.a.c.a.StaticRoleBasedAPIAccessChecker] (qtp1278254413-352:ctx-5219e9e4 ctx-3aeefedf) (logid:15f58092) RoleService is enabled. We will use it instead of StaticRoleBasedAPIAccessChecker.
2024-02-28 15:48:59,351 DEBUG [o.a.c.r.ApiRateLimitServiceImpl] (qtp1278254413-352:ctx-5219e9e4 ctx-3aeefedf) (logid:15f58092) API rate limiting is disabled. We will not use ApiRateLimitService.
2024-02-28 15:48:59,353 WARN  [c.c.a.d.ParamGenericValidationWorker] (qtp1278254413-352:ctx-5219e9e4 ctx-3aeefedf) (logid:15f58092) Received unknown parameters for command importUnmanagedInstance. Unknown parameters : temppath importsource hypervisor host zoneid diskpath
2024-02-28 15:48:59,366 INFO  [o.a.c.f.j.i.AsyncJobMonitor] (API-Job-Executor-10:ctx-aa2475ff job-142) (logid:4377266b) Add job-142 into job monitoring
2024-02-28 15:48:59,374 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (qtp1278254413-352:ctx-5219e9e4 ctx-3aeefedf) (logid:15f58092) submit async job-142, details: AsyncJobVO: {id:142, userId: 2, accountId: 2, instanceType: None, instanceId: null, cmd: org.apache.cloudstack.api.command.admin.vm.ImportUnmanagedInstanceCmd, cmdInfo: {"temppath":"","migrateallowed":"true","httpmethod":"GET","ctxAccountId":"2","details[0].memory":"32768","cmdEventType":"VM.IMPORT","hypervisor":"vmware","host":"","ctxStartEventId":"373","ctxDetails":"{\"interface com.cloud.offering.ServiceOffering\":\"04a0bc45-5669-4ef7-8214-ea7bf8e4e87f\",\"interface com.cloud.org.Cluster\":\"07cc5a91-8306-4ae9-9a49-8b89965ea23f\"}","details[0].cpuNumber":"8","details[0].cpuSpeed":"1000","clusterid":"07cc5a91-8306-4ae9-9a49-8b89965ea23f","importsource":"unmanaged","nicnetworklist[0].network":"871e547e-d26a-43c0-9d7b-0c1f33a3edb8","serviceofferingid":"04a0bc45-5669-4ef7-8214-ea7bf8e4e87f","response":"json","ctxUserId":"2","displayname":"testvm","name":"WinTestYeni","zoneid":"3e34a0b7-ba5e-44f1-ab7a-8598549095dc","diskpath":"","nicnetworklist[0].nic":"Network adapter 1","username":""}, cmdVersion: 0, status: IN_PROGRESS, processStatus: 0, resultCode: 0, result: null, initMsid: 3232277003, completeMsid: null, lastUpdated: null, lastPolled: null, created: null, removed: null}
2024-02-28 15:48:59,374 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (API-Job-Executor-10:ctx-aa2475ff job-142) (logid:36097674) Executing AsyncJobVO: {id:142, userId: 2, accountId: 2, instanceType: None, instanceId: null, cmd: org.apache.cloudstack.api.command.admin.vm.ImportUnmanagedInstanceCmd, cmdInfo: {"temppath":"","migrateallowed":"true","httpmethod":"GET","ctxAccountId":"2","details[0].memory":"32768","cmdEventType":"VM.IMPORT","hypervisor":"vmware","host":"","ctxStartEventId":"373","ctxDetails":"{\"interface com.cloud.offering.ServiceOffering\":\"04a0bc45-5669-4ef7-8214-ea7bf8e4e87f\",\"interface com.cloud.org.Cluster\":\"07cc5a91-8306-4ae9-9a49-8b89965ea23f\"}","details[0].cpuNumber":"8","details[0].cpuSpeed":"1000","clusterid":"07cc5a91-8306-4ae9-9a49-8b89965ea23f","importsource":"unmanaged","nicnetworklist[0].network":"871e547e-d26a-43c0-9d7b-0c1f33a3edb8","serviceofferingid":"04a0bc45-5669-4ef7-8214-ea7bf8e4e87f","response":"json","ctxUserId":"2","displayname":"testvm","name":"WinTestYeni","zoneid":"3e34a0b7-ba5e-44f1-ab7a-8598549095dc","diskpath":"","nicnetworklist[0].nic":"Network adapter 1","username":""}, cmdVersion: 0, status: IN_PROGRESS, processStatus: 0, resultCode: 0, result: null, initMsid: 3232277003, completeMsid: null, lastUpdated: null, lastPolled: null, created: null, removed: null}
2024-02-28 15:48:59,375 DEBUG [c.c.a.ApiServlet] (qtp1278254413-352:ctx-5219e9e4 ctx-3aeefedf) (logid:15f58092) ===END===  192.168.161.8 -- GET  name=WinTestYeni&clusterid=07cc5a91-8306-4ae9-9a49-8b89965ea23f&displayname=testvm&zoneid=3e34a0b7-ba5e-44f1-ab7a-8598549095dc&importsource=unmanaged&hypervisor=vmware&host=&username=&diskpath=&temppath=&serviceofferingid=04a0bc45-5669-4ef7-8214-ea7bf8e4e87f&details[0].cpuNumber=8&details[0].cpuSpeed=1000&details[0].memory=32768&migrateallowed=true&nicnetworklist[0].nic=Network+adapter+1&nicnetworklist[0].network=871e547e-d26a-43c0-9d7b-0c1f33a3edb8&command=importUnmanagedInstance&response=json
2024-02-28 15:48:59,378 WARN  [c.c.a.d.ParamGenericValidationWorker] (API-Job-Executor-10:ctx-aa2475ff job-142 ctx-8dae85a2) (logid:36097674) Received unknown parameters for command importUnmanagedInstance. Unknown parameters : temppath importsource hypervisor host zoneid diskpath
2024-02-28 15:48:59,386 DEBUG [c.c.u.AccountManagerImpl] (API-Job-Executor-10:ctx-aa2475ff job-142 ctx-8dae85a2) (logid:36097674) Access granted to Account [{"accountName":"admin","id":2,"uuid":"683aa0c3-d606-11ee-8b6a-0050568d528d"}] to Service offering {"id":37,"name":"Custom","uuid":"04a0bc45-5669-4ef7-8214-ea7bf8e4e87f"}. by AffinityGroupAccessChecker
2024-02-28 15:48:59,388 TRACE [o.a.c.a.c.a.v.ImportUnmanagedInstanceCmd] (API-Job-Executor-10:ctx-aa2475ff job-142 ctx-8dae85a2) (logid:36097674) nic, 'Network adapter 1', goes on net, '871e547e-d26a-43c0-9d7b-0c1f33a3edb8'
2024-02-28 15:48:59,397 DEBUG [c.c.a.ApiServlet] (qtp1278254413-364:ctx-71cf1a66) (logid:5b984267) ===START===  192.168.161.8 -- GET  jobId=36097674-1d79-401d-ba43-130fde33d3c4&command=queryAsyncJobResult&response=json
2024-02-28 15:48:59,397 DEBUG [c.c.a.ApiServlet] (qtp1278254413-364:ctx-71cf1a66) (logid:5b984267) Two factor authentication is already verified for the user 2, so skipping
2024-02-28 15:48:59,402 DEBUG [c.c.a.m.DirectAgentAttache] (DirectAgent-199:ctx-09d81f14) (logid:99c7322f) Seq 1-3508304109721617290: Executing request
2024-02-28 15:48:59,402 DEBUG [c.c.h.v.r.VmwareResource] (DirectAgent-199:ctx-09d81f14) (logid:36097674) Executing resource command GetUnmanagedInstancesCommand: [].
2024-02-28 15:48:59,403 DEBUG [c.c.a.ApiServer] (qtp1278254413-364:ctx-71cf1a66 ctx-f1129f61) (logid:5b984267) CIDRs from which account 'Account [{"accountName":"admin","id":2,"uuid":"683aa0c3-d606-11ee-8b6a-0050568d528d"}]' is allowed to perform API calls: 0.0.0.0/0,::/0
2024-02-28 15:48:59,405 INFO  [o.a.c.a.DynamicRoleBasedAPIAccessChecker] (qtp1278254413-364:ctx-71cf1a66 ctx-f1129f61) (logid:5b984267) Account [Account [{"accountName":"admin","id":2,"uuid":"683aa0c3-d606-11ee-8b6a-0050568d528d"}]] is Root Admin or Domain Admin, all APIs are allowed.
2024-02-28 15:48:59,407 WARN  [o.a.c.a.ProjectRoleBasedApiAccessChecker] (qtp1278254413-364:ctx-71cf1a66 ctx-f1129f61) (logid:5b984267) Project is null, ProjectRoleBasedApiAccessChecker only applies to projects, returning API [queryAsyncJobResult] for user [User {"username":"admin","uuid":"683bac0e-d606-11ee-8b6a-0050568d528d"}.] as allowed.
2024-02-28 15:48:59,407 DEBUG [o.a.c.a.StaticRoleBasedAPIAccessChecker] (qtp1278254413-364:ctx-71cf1a66 ctx-f1129f61) (logid:5b984267) RoleService is enabled. We will use it instead of StaticRoleBasedAPIAccessChecker.
2024-02-28 15:48:59,408 DEBUG [o.a.c.r.ApiRateLimitServiceImpl] (qtp1278254413-364:ctx-71cf1a66 ctx-f1129f61) (logid:5b984267) API rate limiting is disabled. We will not use ApiRateLimitService.
2024-02-28 15:48:59,418 DEBUG [c.c.a.ApiServlet] (qtp1278254413-364:ctx-71cf1a66 ctx-f1129f61) (logid:5b984267) ===END===  192.168.161.8 -- GET  jobId=36097674-1d79-401d-ba43-130fde33d3c4&command=queryAsyncJobResult&response=json
2024-02-28 15:48:59,419 DEBUG [c.c.h.v.m.HostMO] (DirectAgent-199:ctx-09d81f14 esx02.flexapp360.cloud, job-142, cmd: GetUnmanagedInstancesCommand) (logid:36097674) find VM hypervisor name: WinTestYeni on host
2024-02-28 15:48:59,419 INFO  [c.c.h.v.m.HostMO] (DirectAgent-199:ctx-09d81f14 esx02.flexapp360.cloud, job-142, cmd: GetUnmanagedInstancesCommand) (logid:36097674) VM hypervisor name: WinTestYeni not found in host cache
2024-02-28 15:48:59,419 DEBUG [c.c.h.v.m.HostMO] (DirectAgent-199:ctx-09d81f14 esx02.flexapp360.cloud, job-142, cmd: GetUnmanagedInstancesCommand) (logid:36097674) load VM cache on host
2024-02-28 15:48:59,459 DEBUG [c.c.a.ApiServlet] (qtp1278254413-352:ctx-944e5120) (logid:b2962708) ===START===  192.168.161.8 -- GET  clusterid=07cc5a91-8306-4ae9-9a49-8b89965ea23f&page=1&pagesize=10&command=listUnmanagedInstances&response=json
2024-02-28 15:48:59,459 DEBUG [c.c.a.ApiServlet] (qtp1278254413-352:ctx-944e5120) (logid:b2962708) Two factor authentication is already verified for the user 2, so skipping
2024-02-28 15:48:59,460 DEBUG [c.c.a.ApiServlet] (qtp1278254413-1586:ctx-b445e3bf) (logid:e3503aed) ===START===  192.168.161.8 -- GET  listall=true&clusterid=07cc5a91-8306-4ae9-9a49-8b89965ea23f&page=1&pagesize=10&command=listVirtualMachines&response=json
2024-02-28 15:48:59,460 DEBUG [c.c.a.ApiServlet] (qtp1278254413-1586:ctx-b445e3bf) (logid:e3503aed) Two factor authentication is already verified for the user 2, so skipping
2024-02-28 15:48:59,466 DEBUG [c.c.a.ApiServer] (qtp1278254413-352:ctx-944e5120 ctx-3e42fa19) (logid:b2962708) CIDRs from which account 'Account [{"accountName":"admin","id":2,"uuid":"683aa0c3-d606-11ee-8b6a-0050568d528d"}]' is allowed to perform API calls: 0.0.0.0/0,::/0
2024-02-28 15:48:59,466 DEBUG [c.c.a.ApiServer] (qtp1278254413-1586:ctx-b445e3bf ctx-74422909) (logid:e3503aed) CIDRs from which account 'Account [{"accountName":"admin","id":2,"uuid":"683aa0c3-d606-11ee-8b6a-0050568d528d"}]' is allowed to perform API calls: 0.0.0.0/0,::/0
2024-02-28 15:48:59,468 INFO  [o.a.c.a.DynamicRoleBasedAPIAccessChecker] (qtp1278254413-352:ctx-944e5120 ctx-3e42fa19) (logid:b2962708) Account [Account [{"accountName":"admin","id":2,"uuid":"683aa0c3-d606-11ee-8b6a-0050568d528d"}]] is Root Admin or Domain Admin, all APIs are allowed.
2024-02-28 15:48:59,468 INFO  [o.a.c.a.DynamicRoleBasedAPIAccessChecker] (qtp1278254413-1586:ctx-b445e3bf ctx-74422909) (logid:e3503aed) Account [Account [{"accountName":"admin","id":2,"uuid":"683aa0c3-d606-11ee-8b6a-0050568d528d"}]] is Root Admin or Domain Admin, all APIs are allowed.
2024-02-28 15:48:59,470 WARN  [o.a.c.a.ProjectRoleBasedApiAccessChecker] (qtp1278254413-1586:ctx-b445e3bf ctx-74422909) (logid:e3503aed) Project is null, ProjectRoleBasedApiAccessChecker only applies to projects, returning API [listVirtualMachines] for user [User {"username":"admin","uuid":"683bac0e-d606-11ee-8b6a-0050568d528d"}.] as allowed.
2024-02-28 15:48:59,470 WARN  [o.a.c.a.ProjectRoleBasedApiAccessChecker] (qtp1278254413-352:ctx-944e5120 ctx-3e42fa19) (logid:b2962708) Project is null, ProjectRoleBasedApiAccessChecker only applies to projects, returning API [listUnmanagedInstances] for user [User {"username":"admin","uuid":"683bac0e-d606-11ee-8b6a-0050568d528d"}.] as allowed.

api job response

{
    "queryasyncjobresultresponse": {
        "accountid": "683aa0c3-d606-11ee-8b6a-0050568d528d",
        "account": "admin",
        "domainid": "424ddefc-d606-11ee-8b6a-0050568d528d",
        "domainpath": "ROOT",
        "userid": "683bac0e-d606-11ee-8b6a-0050568d528d",
        "cmd": "org.apache.cloudstack.api.command.admin.vm.ImportUnmanagedInstanceCmd",
        "jobstatus": 2,
        "jobprocstatus": 0,
        "jobresultcode": 530,
        "jobresulttype": "object",
        "jobresult": {
            "errorcode": 530,
            "errortext": "Storage pool for disk Hard disk 1(123-2000) with datastore: FA360-DS02-ESX not found in zone ID: 3e34a0b7-ba5e-44f1-ab7a-8598549095dc"
        },
        "created": "2024-02-28T16:26:23+0300",
        "completed": "2024-02-28T16:26:23+0300",
        "jobid": "18707e99-88a3-4a0c-948c-b1937496a6a4"
    }
}

[nvazquez]

Thanks @omurozlu - I have quickly tested imports on NFS cluster and zone wide storage and couldn't hit the issue. It seems related to datastore cluster type, will try to replicate it using datastore cluster

[omurozlu]

4.18 version working but 4.19 not working

[nvazquez]

Hi @omurozlu - I have tested 4.19.0 creating a VM with a ROOT volume on a datastore cluster, unmanaging it and import it back and it succeeds.

I have checked your logs and the images you have shared and I think the problem is that the datastore cluster in which the ROOT volume of the VM is located, is not added to CloudStack.

From the cmk output, CloudStack is looking for datastore FA360-DS02-ESX:

"Storage pool for disk Hard disk 1(123-2000) with datastore: FA360-DS02-ESX not found in zone ID: 3e34a0b7-ba5e-44f1-ab7a-8598549095dc"

However, from the images above, I only see one datastore with name FA360-DS01-ESX.

Can you double check the names on vCenter and CloudStack? In case you have renamed the datastore on vCenter, you should rename it as well on CloudStack for them to match

[omurozlu]

Hello again @nvazquez ,
I now tested again (reset system)





this first tested

==============
and im again reset system

all time give same error

[nvazquez]

Sorry @omurozlu I wrongly tested datastore cluster pools, but I see you are using VMFS pools instead. Can you share what's the output of this SQL query?

select * from storage_pool;

[omurozlu]

@nvazquez,

[nvazquez]

Thanks @omurozlu can you please upload the management server logs around the whole import operation? I want to compare the received unmanaged instance disk's information to the information of the pool in the database