Upgrade Fails due to node drain timeout #930
Description
Is this a request for help?: NO
Is this an ISSUE or FEATURE REQUEST? (choose one): ISSUE
What version of aks-engine?: v0.33.2
Orchestrator and version (e.g. Kubernetes, DC/OS, Swarm) Kubernetes (upgrade from 1.13.3 to 1.13.5)
What happened: When doing an upgrade, there is now a drain process due to . However, the timeout for the drain is only 1 minute, and this is nowhere near long enough with persistent Azure disks, which take several minutes to drain from a node because they are dang slow to reschedule on other nodes.
The upgrade fails every time due to this timeout
What you expected to happen:
Drain should not timeout or set an option to increase it.
How to reproduce it (as minimally and precisely as possible):
Create a cluster with a bunch of PVs. Upgrade it.
Anything else we need to know: