Hash sum mismatches with apt

[badp]

I have the following error when updating my repositories through pac4cli, but not when I instruct apt to go directly through the proxy:

~> sudo apt update
[...]
Get:28 http://nl.archive.ubuntu.com/ubuntu bionic-updates/main i386 Packages [252 kB]                                          
Get:28 http://nl.archive.ubuntu.com/ubuntu bionic-updates/main i386 Packages [252 kB]                                                  
Get:28 http://nl.archive.ubuntu.com/ubuntu bionic-updates/main i386 Packages [252 kB]
Err:28 http://nl.archive.ubuntu.com/ubuntu bionic-updates/main i386 Packages                                                                 
  Hash Sum mismatch
  Hashes of expected file:
   - Filesize:252356 [weak]
   - SHA256:4ac29191784f2fb23fcc15e0c63e31186e715ac5f225ae4f67e8609f670c328d
   - SHA1:06838369a89a798e49f6cd38402da035f4cadce3 [weak]
   - MD5Sum:aa49210f64a49fc87757d47d776d2205 [weak]
  Hashes of received file:
   - SHA256:b9b2bcbdc22af3c84ae97165e56bbc1072c03417a56a3fcbe18cb5a3801252cf
   - SHA1:05529bcf9522ecf66b06f0896e104b847693a157 [weak]
   - MD5Sum:164012ed99f02a2427f3aee21bb4de89 [weak]
   - Filesize:158124 [weak]
  Last modification reported: Fri, 10 Aug 2018 00:59:36 +0000
  Release file created at: Fri, 10 Aug 2018 06:13:35 +0000
Err:31 http://nl.archive.ubuntu.com/ubuntu bionic-updates/main amd64 DEP-11 Metadata                              

Get:32 http://nl.archive.ubuntu.com/ubuntu bionic-updates/main DEP-11 64x64 Icons [53.7 kB]                                                                                      
Get:33 http://nl.archive.ubuntu.com/ubuntu bionic-updates/universe amd64 Packages [158 kB]                            
Get:34 http://nl.archive.ubuntu.com/ubuntu bionic-updates/universe i386 Packages [158 kB]                                                                                        
Hit:24 https://packagecloud.io/slacktechnologies/slack/debian jessie InRelease  
[...]                                                      

W: http://nl.archive.ubuntu.com/ubuntu/dists/bionic-updates/main/binary-i386/by-hash/SHA256/4ac29191784f2fb23fcc15e0c63e31186e715ac5f225ae4f67e8609f670c328d: Automatically disabled Acquire::http::Pipeline-Depth due to incorrect response from server/proxy. (man 5 apt.conf)     

I cannot reproduce the issue from the command line:

~> env HTTP_PROXY='http://webproxy:3128/' wget --quiet http://nl.archive.ubuntu.com/ubuntu/dists/bionic-updates/main/binary-i386/by-hash/SHA256/4ac29191784f2fb23fcc15e0c63e31186e715ac5f225ae4f67e8609f670c328d -O - | wc -c
252356
~> env HTTP_PROXY='http://localhost:3128/' wget --quiet http://nl.archive.ubuntu.com/ubuntu/dists/bionic-updates/main/binary-i386/by-hash/SHA256/4ac29191784f2fb23fcc15e0c63e31186e715ac5f225ae4f67e8609f670c328d -O - | wc -c
252356

The downloaded files are binary-equal.

This doesn't just happen for apt update. apt-upgrade also failed on, say, this package:

Err:2 http://nl.archive.ubuntu.com/ubuntu bionic-proposed/main amd64 console-setup all 1.178ubuntu2.6
  Hash Sum mismatch
  Hashes of expected file:
   - SHA256:0054a78701d67fd1f34227b5718edf3d8e50102696790dcf934dd372988ef201
   - SHA1:b9424f627123511aa5b94705275d1cf635e25e28 [weak]
   - MD5Sum:9795f2526d271aedfff53ecdf6cecee8 [weak]
   - Filesize:104608 [weak]
  Hashes of received file:
   - SHA256:d5cf27902451885a3ff18446015729f3eae72e7a196a926017982170bf8a27c7
   - SHA1:b8cd777ee21157ad3cca0e16336addac01a18963 [weak]
   - MD5Sum:1668125ec62016fcf375c6bdf102039e [weak]
   - Filesize:104608 [weak]
  Last modification reported: Thu, 09 Aug 2018 13:08:18 +0000

I remember having a similar issue with my workplace's squid proxies as well, but the newer proxies don't seem to exhibit this issue. The only resolution at the time, IIRC, was not using the proxies...

[eythian]

I've had success with:

$ cat /etc/apt/apt.conf.d/80http 
Acquire::http::No-Cache true;
Acquire::http::Pipeline-Depth 0;

for this sort of thing in the past.

[tkluck]

This probably has the same root cause as #13 , and I like @eythian's suggestion. Wikipedia says that on http < v2, http pipelining is barely used and often not properly implemented. That's probably also true for the monkey-patched twisted code that pac4cli uses.

In general, pac4cli's http handling is poorly tested because most traffic goes over https (so proxy CONNECT) these days. That also means that I haven't dared put the proxy configuration for Apt in the debian package. (Another reason for that is that it would mean a pac4cli breakage means we can't update pac4cli to fix it.)

[badp]

as an addendum, it seems that the Acquire::http::No-Cache true; directive is unnecessary (I'll amend this comment if reality proves me wrong).

[eythian]

That may well be correct, I remember it being a "good offense is the best defence" type solution to defend against proxy weirdness.

…

On 10 August 2018 17:31:38 CEST, Santi ***@***.***> wrote: as an addendum, it seems that the `Acquire::http::No-Cache true;` directive is unnecessary (I'll amend this comment if reality proves me wrong). -- You are receiving this because you were mentioned. Reply to this email directly or view it on GitHub: #32 (comment)

[tkluck]

-> #37