Pods unable to be created on windows nodes

[lizhuqi]

It seems that Windows OS has some issue handling concurrent actions. Concurrent actions sometimes failed to start containers on Windows.

27m Warning FailedCreatePodSandBox pod/[POD_ONE] Failed to create pod sandbox: rpc error: code = Unknown desc = failed to start sandbox container for pod "[POD_ONE]": Error response from daemon: hcsshim::PrepareLayer - failed failed in Win32: The device is not ready. (0x15)
24m Warning FailedMount pod/[POD_ONE] Unable to attach or mount volumes: unmounted volumes=[workspace-volume default-token-xyz12], unattached volumes=[workspace-volume default-token-xyz12l]: timed out waiting for the condition
30m Normal Scheduled pod/[POD_TWO] Successfully assigned cje/[POD_TWO] to [NODE_ID]
29m Warning FailedCreatePodSandBox pod/[POD_TWO] Failed to create pod sandbox: rpc error: code = Unknown desc = failed to start sandbox container for pod "[POD_TWO]": Error response from daemon: hcsshim::PrepareLayer - failed failed in Win32: The device is not ready. (0x15)

[vrapolinario]

Hi @lizhuqi an you please provide more details on the issue you're finding? Such as: Is this a managed K8s (AKS, EKS)? What version of K8s? K8s node config, etc?

[marosset]

Can you include a few more details such as

• Pod spec
• number of containers starting concurrently
• Node side (reserved)
• Any other repro steps.
• OS version

Thanks!

[lizhuqi]

I noticed that cluster which has the issue is still using 10.0.17763.1577. I will ask the owner to upgrade to 10.0.17763.1757 first and then add more details.

[zhiweiv]

This is also reported here microsoft/hcsshim#919.
VM spec is E2_V4(2 cores/16 GB) , we use bare pod as job workloads, there are about 20 concurrent pod creation/termination always.
The pod spec

    resources:
      requests:
        cpu: 50m
        memory: 500Mi
      limits:
        cpu: 1000m
        memory: 4Gi

[immuzz]

Is this on AKS or some other distro or DIY K8s?

[zhiweiv]

For me, it is reproduced in both AKS and AKS-Engine.

This issue has been open for 30 days with no updates.
@immuzz, please provide an update or close this issue.

This issue has been open for 30 days with no updates.
@immuzz, please provide an update or close this issue.

[zhiweiv]

There is a fix microsoft/hcsshim#1091 (comment).
Per microsoft/hcsshim#919 (comment), I guess still need a long time for it to be available in AKS.

[AbelHu]

@zhiweiv, I think that the fix needs to be picked up by moby at first and then we can upgrade the moby version in AKS side.
@immuzz do you know who can speed up taking the new hccshim release with this fix in next moby version?

[zhiweiv]

Hope it can be merged into AKS Windows ContainerD ASAP too. Get Outlook for iOS<https://aka.ms/o0ukef>

…

________________________________ From: Chou Hu ***@***.***> Sent: Monday, August 16, 2021 10:23:29 AM To: microsoft/Windows-Containers ***@***.***> Cc: Zhiwei Liu ***@***.***>; Mention ***@***.***> Subject: Re: [microsoft/Windows-Containers] Pods unable to be created on windows nodes (#109) @zhiweiv<https://github.com/zhiweiv>, I think that the fix needs to be picked up by moby at first and then we can upgrade the moby version in AKS side. @immuzz<https://github.com/immuzz> do you know who can speed up taking the new hccshim release with this fix in next moby version? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub<#109 (comment)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/AFH3LQPHTZH6GWJSLQOVGADT5BZCDANCNFSM43TNATIA>.