retrospective on download outage

[Trott]

For picking up once the current issue is addressed and behind us, a retrospective to figure out how to make it better next time would be a good thing, I think.

Refs: nodejs/node#32683
Refs: nodejs/node#32683

[MattIPv4]

An official status page for NodeJS would likely be helpful in situations like this for a couple of reasons:

• A canonical place for NodeJS folks to communicate & acknowledge the issue
• Make the community aware, reducing the number of issues created in the org & beyond

Created #2265 to look in more detail at the process of creating a status page.

[Trott]

No one has been added to @nodejs/build-infra in literally years. 5 years or something like that? We need to break that choke-point. It's OK if there are only 4 people in the group, but it's got to be 4 highly-active people. Right now, my guess is that we have 1 highly-active person, 2 moderately-active people, and 1 effectively-inactive person.

I would like us to think about who we might trust with this level of access. I'm surprised we haven't added @sam-github to the list, to be honest.

[Trott]

This, of course, also raises the issue of if we are taking a risky approach by relying entirely on volunteers. Assuming the Foundation would be willing to pay to have someone on staff doing the most critical work, should we consider that? (We'll have to carefully define the most critical work.)

[Trott]

On-call schedule? No, wait, hear me out! The idea is that we may not trust (for example) me with the keys to the kingdom, but maybe we trust me with the cell phone numbers of the people who do have those keys. And we trust me to not abuse this and to only notify people in a real crisis. Maybe we can have someone on-call to figure out who is in a not-terrible time zone and try to get them on the phone if we have an outage like this one.

I'm not sold on this being a great idea, but I'm trying to get all the ideas out anyway.

[AshCripps]

I think the crux of the issue with build infra is coverage, we have 1 US, 2 Europe and 1 AUS which gives us decent coverage of time zones - but the European members are often harder to contact. Adding @sam-github would be a good start but still leaves a big void in the middle of the timezones. Obviously this is tricky due to the security concerns around the level of access this role gives.

[MattIPv4]

Having a proper system for an emergency contact makes sense, either through an on-call system of trusted volunteers (build-infra) or a paid position that comes with the responsibility of being okay with being woken up in the middle of the night if the world falls apart.

[sam-github]

@MattIPv4, it may not be clear from the project activity, https://github.com/nodejs/build/graphs/contributors?from=2020-01-08&to=2020-04-06&type=c (is there an equivalent view for comments on issues and PRs? I wish there was, but it wouldn't substantially change that view), but the top 3 folks at that link are developers, who primarily maintain jenkins and CI, and the 4th (Rod) is an expert, but only 1 person.

We don't lack system for making emergency contacts, IMO, we lack people to contact. There has been talk of getting paid infra support in, this might change that conversation.

cloudfare experts are few and far between in the Node.js project, or at least the number of them volunteering to join the build-wg is aprox zero.

[MattIPv4]

Totally makes sense, I intended for my comment to imply that if an on-call system is used, part of that is ensuring there are folks on the other end to call that can actually help.

[MattIPv4]

I should also say that I’m happy to help in any way I can (eg. getting #2265 going), though I would say I’m not an expert on infra.

More than happy to help connect y’all with folks at DO or Cf if you think there’s anything specific we can help with. :)

[rvagg]

Talking about choke points is fine as long as you can prove that something would be different if we added more people, and in this case nothing would be different. Even if I was awake in the middle of the night watching this play out I have no idea what I would have done to "rectify" it. At the moment it looks like a Cloudflare problem that's out of our control.

I know the feeling of impotence is frustrating, but just splashing out more access to more people doesn't necessarily improve the situation. We had this same conversation when we had a disk-full problem ("I can log in to nodejs.org but can't do X"), but even if more people had access, it's not clear that those people would know what to do to fix the problem.

There's (1) access but there's also (2) knowledge of the systems involved and (3) understanding of how it all fits together and how to not break things even further. There's a lot to break and there's a lot a stake.

Let's not try and make this look simpler than it is, you're just going to make it harder for those of us who carry the bulk of the burden because the blame gets focused like a laser -- "but it's so simple, you're at fault because you [haven't done X | did Y]".

[rvagg]

My assessment so far is written up @ #2263 (comment)

Reproducing the most interesting part of that:

What can we do better next time? 🤷‍♂ that graph looks to me like it's mostly out of our control and this is a CF hiccup that they haven't acknowledged on their status page. I'll do some more digging but I'm exhausting all but my most outlandish theories.

This does bring into focus our edge cache invalidation process which hasn't evolved since we've pushed to full CF caching of our downloads. There's a lot of room for improvement there and options for replacing the process with an entirely better setup. It'll just take very careful work.

[Trott]

Talking about choke points is fine as long as you can prove that something would be different if we added more people, and in this case nothing would be different. Even if I was awake in the middle of the night watching this play out I have no idea what I would have done to "rectify" it. At the moment it looks like a Cloudflare problem that's out of our control.

I wouldn't say "nothing would be different". Having someone authoritative saying things every few hours along the lines of "We're aware of the problem and actively looking into it" and/or "Looks like a Cloudflare problem that's out of our control, but we'll provide more information when we have it" is a huge help, even if the outage lasts exactly as long as it would otherwise. These things are squishy, but I'd say that the actual outage was only 20% of the problem. A squishy 80% of the problem was not being able to communicate anything meaningful to end users. That looks bad, increases anxiety, and prompts FUD in places like reddit and Twitter.

I appreciate what you're saying, which (if I'm understanding correctly) is that finding qualified people and (perhaps especially) taking the necessary time and effort to do knowledge transfer is vastly harder than my original comments imply. And then, of course, there's also the general issue of trust. It's not enough for someone to be qualified and trained. They need to be trusted as well.

We've talked about how to improve our bus-factor more than once in the past, so perhaps step 1 is for me to go back through old email threads and meeting notes before saying much more about it.