Now: Jenkins is in a weird mood

[refack]

Saying something about reserving the machines

[refack]

Ohh the master node is offline:

[gibfahn]

I can bring it back online, but we fundamentally need to clean up that disk (or allocate more space).

[gibfahn]

Oh wait, I can't bring it back online:

[refack]

sad emoji

[gibfahn]

@refack Have you tried the node-build IRC channel? You need someone with the nodejs_build_infra ssh key (who can actually get into the machines).

[refack]

@refack Have you tried the node-build IRC channel? You need someone with the nodejs_build_infra ssh key (who can actually get into the machines).

Yep, and #node-dev (posted here last since I thought maybe someone will get an e-mail)

[rvagg]

on it

[rvagg]

working again ... but we're using up so much disk, I've even gone back from 7 days to 5 days retention and we're still at 95% disk. We're going to have to switch to block storage for this

[gibfahn]

I've even gone back from 7 days to 5 days retention and we're still at 95% disk.

Do we know what's using this up? Is it just loads of copies of Node repos? Jenkins artifacts? Something else?

[refack]

Maybe it's time to revisit #739, gate the big jobs (or maybe just PR) on first passing linux one, it's also gives a very quick feedback.

[joaocgreis]

Found a huge job log file, cctest produced 40Gb of output (stack traces in the beginning). We are back at 50+ Gb free, which is about normal.

[rvagg]

we should move to block storage on DO for this now that we can