SoftLayer instability

[rvagg]

I'm currently bogged down getting all 3 SoftLayer CentOS 6 machines back online, release and 2 x test. I've had to reprovision them but am having all sorts of problems. They don't seem to like being rebooted and my guess is that there's some kind of problem with the kernel that gets installed during a yum upgrade (a kernel we want for security but doesn't seem to be liked by the VM host is my guess). I also got a host upgrade notice on two VMs that I'd just provisioned so I suppose they're still working through their own upgrade process and that could be related.

So @nodejs/build, don't go messing too much with SoftLayer hosts just now, don't reboot anything, absolutely do not touch the CentOS 5 machines there!

[maclover7]

ping @rvagg -- did this ever get resolved?

[rvagg]

been meaning to test it out, I haven't restarted anything on softlayer yet as I don't have the time to spend fixing up a borked restart if it's still a problem! there was an incident report from them today, critical maintenance or something, my guess is that this churn around Meltdown is causing a lot of internal churn for infra providers that is still to settle down

[maclover7]

ping @rvagg -- would like to start working on migrating the setup CentOS ansible scripts to ansible.

[rvagg]

I've been doing plenty of rebooting over the past week an all's fine as far as I can tell

@maclover7 I thought we had centos covered in the ansible scripts already? maybe not CentOS5 but we're waiting for that to die out with Node 4 I think.

[maclover7]

@rvagg Not sure -- setup/centos5 and setup/centos6 still exist, so was assuming migration was complete.

[rvagg]

@maclover7 no, their existence in setup doesn't indicate completeness of migration. It wasn't that well organised! Most of the things in setup/ for standard test machines are redundant. www, raspberry-pi are the two that I still keep updated and have not been migrated. cross-compiler might still be valid for that machine. I don't know about rhel72-linuxonecc or aix61. github-bot I think is still in use. linter I'm not sure about. I think armv7-wheezy might still be the only way to manage those machines. I think the rest are OK to remove, like you've been slowly doing.

[maclover7]

@rvagg that's my bad -- meant to say migration was not complete, since those directories still exist :)

I think the rest are OK to remove, like you've been slowly doing.

Yep, slowly I'm going through and testing all of the playbooks on a Vagrant VM, and then filing PRs to remove as appropriate.

[maclover7]

@rvagg I'm getting a timeout when trying to ssh to test-softlayer-centos6-x64-1 and test-softlayer-centos6-x64-2, going to reopen this for now

[rvagg]

@maclover7 do you have the latest inventory? 50.23.85.250 and 169.61.75.58 are what you should have and they work for me.

[maclover7]

@rvagg Here's what's in master for test-softlayer-centos6-*:

centos6-x64-1: {ip: 50.23.85.251}
centos6-x64-2: {ip: 184.173.120.137}

Maybe this was/is changed in a PR?