Range requests interact badly with the CDN

[edsko]

As reported by hvr

[09:32:35]  <hvr>   well, if you grab the last few bytes of 01-index.tar
[09:32:49]  <hvr>   the request that would take 1 sec directly to hackage
[09:32:54]  <hvr>   via the CDN takes >10 secs
[09:33:02]  <hvr>   until the CDN has the new .tar cached
[09:33:26]  <hvr>   probably the CDN needs to grab the full new .tar
[09:33:42]  <hvr>   s/needs to//
[09:33:57]  <hvr>   and that's 300MiB
[09:34:09]  <hvr>   and since the CDN is distributed
[09:34:21]  <hvr>   it does so multiple times for its nodes
[09:34:44]  <edsko> could we somehow push to the CDN?
[09:34:58]  <edsko> whenever the index is updated, tell the CDN "please update the .tar and .gz, now, please"
[09:35:36]  <hvr>   I'm rather thinking of disabling the CDN for the uncompressed .tar
[09:35:53]  <edsko> hm
[09:36:01]  <edsko> with mirror selection as it currently stands
[09:36:09]  <edsko> that would mean that _all_ requests for updates to the central hackage server
[09:36:11]  <edsko> unless it does not respond
[09:36:17]  <edsko> in which case they will go the mirror
[09:36:36]  <edsko> if I do disable the CDN, then I guess we need to put some some of randomization into mirror selection?

[09:50:11]  <hvr>   and it's not deterministic
[09:50:24]  <hvr>   it's possible to reproduce it with curl if you're lucky
[09:50:28]  <hvr>   (I did)
[09:50:38]  <hvr>   ideally shortly after the tarball got updated

[dcoutts]

So this should be much better now because we do range reqs on the tar.gz, and if lots of people are hitting this file then it'll be warm in the cache more often. However it'll still be the case that sometimes a client will request the last 64k or so of the 01-index.tar.gz and the cache will go and fetch the whole 30Mb 01-index.tar.gz before returning the last bit. So that'll still be poor in some cases.