Fix infinite loop in cache loop detection algorithm

[israelglar]

fixes #6008

I was able to repeatedly have the problem with my current project, but I can't share it's code and I'm not really sure what part of it is causing it so unfortunately I can't create a repeatable environment.

But since I had the issue, after some testing and reading other people's comments I found that the issue was in cache loops removal part of the code.

The already visited digests were being removed from the visited list

[israelglar]

For reference this is a sample of the log I added to the checkLoops function during the infinite loop
The "visited" map for all these digests started empty and keep incrementing. When I stopped the process, all the same digests were still being checked and the len(visited) was 100+ for all of them.

checking loops digest sha256:f65ccfbfec288565c1d414275985547799fde0ed286c85a50bd0ec5faa01d1ac visited 57 links 1
checking loops digest sha256:27e1615212f3c6ea846ed6c412df1361ce97f006ee20bb5aa2483a3b61d5cadd visited 73 links 1
checking loops digest sha256:48b361d46638bfa4eee090c158a750a69c7beec3a62e703e2801125551b1b157 visited 58 links 1
checking loops digest sha256:4e47eb5525df25f94da777993dafa41d9ab2bfa80a89e28f76d42cd46ab082e7 visited 39 links 1
checking loops digest sha256:600b4cdf20cc06a7b5a5cca5f7464296861815519af6d8a14604201b13965ab8 visited 40 links 1
checking loops digest sha256:0788979fc9366e21cd56311511b897a222cf91711481bcd7dc837eac2172d087 visited 41 links 1
checking loops digest sha256:42f0bec3310ddd8a55e8d62817337ca49c55a898c14ab073d07c16dee24d73d4 visited 102 links 1
checking loops digest sha256:e0850a775c17a87060c0cf6efad1020e0cbef5a44ba942bef6add5776598de53 visited 74 links 1
checking loops digest sha256:1e68ed4e3d58a51096a7feea3947f40debf1fd9246ec977eb62ab93c81823ad9 visited 75 links 1
checking loops digest sha256:a0d177b4967a6d99f4ff117defe1c0d23d4e78ca4630febcb948ee9e4520eff3 visited 76 links 1
checking loops digest sha256:00328ce57bbc14b33bd6695bc8eb32cdf2fb5f3a7d89ec14a42825e15d39df60 visited 77 links 1
checking loops digest sha256:d7cdaa5ca0582076c8e772cce739e32c5077cfd24f2ea33f04bb754594989a56 visited 78 links 1
checking loops digest sha256:23c657f2efda7731a3c1990b25f318fa2eb1332208f97ab9cc2a7eac70ab5a76 visited 79 links 1
checking loops digest sha256:af180e4359fc6179dc953abdcbdcaf7c146b53e1bee2b335e50dead11ccefa07 visited 80 links 1
checking loops digest sha256:09895de0407bcb0386733daa14bdb5dfa544505530c634334a05a60f161b71fc visited 81 links 1
checking loops digest sha256:841a05fd378a2c067058585e3691c2a3f5399206fded7a580fdbbc281003168e visited 103 links 1
checking loops digest sha256:c76b405781134be1dab7fe45adfb8c32104805a01de7b863e1004b66d56edf9f visited 63 links 1
checking loops digest sha256:27d719c754aacd492a6dc8a1b76619355abcf5ef473cbec02018d3c57ebbf0d5 visited 64 links 1
checking loops digest sha256:de482c7ed5ca67ae135ef25bf3b13194970a2f3902318f1eda3c64af2a2eb344 visited 104 links 1
checking loops digest sha256:ee62de25ccc2b55d3a0495244b246fb97055b6f1c2697d837b8e94976c03756f visited 65 links 1
checking loops digest sha256:efd96aedf377e20afd95285a7c751a864260bd6a149656a4040c5b7757bdbbb6 visited 66 links 1
checking loops digest sha256:da4d43f295ce92630829272fad6d2e7237c6248e9cd9499e6382d6fa6d758e7b visited 105 links 1
checking loops digest sha256:06de973bb45531d52cdbd483c5e50bcddaa2095f9515e03cfad490061cc9831e visited 106 links 1
checking loops digest sha256:88b54564b232405ab2165996517fece1149259cf1ea262a375db0f66039294d0 visited 107 links 1
checking loops digest sha256:ad3b83575249b68aab9602de378314fc221ab07a9b2ab0bb4a245ec649219f45 visited 108 links 1
checking loops digest sha256:826e27285307a923759de350de081d6218a04f4cff82b20c5ddaa8c60138c066 visited 109 links 6
checking loops digest sha256:30eec89ddd9c342ef28a87f731d6e50ba977baf12d7caa7045a9d56b0e923f03 visited 110 links 1
checking loops digest sha256:b6cb293891dd62748d85aa2e00eb97e267870905edefdfe53a2ea0f3da49e88d visited 111 links 1
checking loops digest sha256:3a8f6d79cd434dc10588606993976b7b2bc038ff4a2481e857ac0168fc29a683 visited 111 links 1
checking loops digest sha256:98144d79af44407273f26589afc01901b7b296deada61a4740b0d404c5043c53 visited 112 links 1
checking loops digest sha256:b1585fdb272b31401eaac5dd46a936c1c09b4861e53e23f12ac72fc077b3c82c visited 113 links 1
checking loops digest sha256:e73cb135243c08ab2c2adc333b150b9237093315f6b38e3361f07caf2bfb4d6b visited 114 links 1
checking loops digest sha256:064c3e311ef63912b0cc91db9681ce2d301c3e76c447febf8faa303de38cc005 visited 110 links 1
checking loops digest sha256:b6cb293891dd62748d85aa2e00eb97e267870905edefdfe53a2ea0f3da49e88d visited 111 links 1
checking loops digest sha256:3a8f6d79cd434dc10588606993976b7b2bc038ff4a2481e857ac0168fc29a683 visited 110 links 1
checking loops digest sha256:98144d79af44407273f26589afc01901b7b296deada61a4740b0d404c5043c53 visited 111 links 1
checking loops digest sha256:b1585fdb272b31401eaac5dd46a936c1c09b4861e53e23f12ac72fc077b3c82c visited 112 links 1
checking loops digest sha256:e73cb135243c08ab2c2adc333b150b9237093315f6b38e3361f07caf2bfb4d6b visited 113 links 1
checking loops digest sha256:e078af3026edb42cc26b32784baa142a79970078f7ac58f8c7b74115f4f7fb60 visited 110 links 1
checking loops digest sha256:1e5ee5e58c8f490ae68e7e91b1575ebefc2bf6c211f302a553ff0c4925e85321 visited 111 links 1
checking loops digest sha256:7f0a22117f8fe0172cf9209ff622b64a51aaeda21d58b5b62685a93dbe2dad25 visited 67 links 1
checking loops digest sha256:71a1c003a2b855d85582c8f6c7648c49d3fe836408a7e1b5d9b222448acb3c1b visited 68 links 1
checking loops digest sha256:27e1615212f3c6ea846ed6c412df1361ce97f006ee20bb5aa2483a3b61d5cadd visited 69 links 1
checking loops digest sha256:e0850a775c17a87060c0cf6efad1020e0cbef5a44ba942bef6add5776598de53 visited 70 links 1
checking loops digest sha256:1e68ed4e3d58a51096a7feea3947f40debf1fd9246ec977eb62ab93c81823ad9 visited 71 links 1
checking loops digest sha256:a0d177b4967a6d99f4ff117defe1c0d23d4e78ca4630febcb948ee9e4520eff3 visited 72 links 1
checking loops digest sha256:00328ce57bbc14b33bd6695bc8eb32cdf2fb5f3a7d89ec14a42825e15d39df60 visited 73 links 1
checking loops digest sha256:d7cdaa5ca0582076c8e772cce739e32c5077cfd24f2ea33f04bb754594989a56 visited 74 links 1
checking loops digest sha256:23c657f2efda7731a3c1990b25f318fa2eb1332208f97ab9cc2a7eac70ab5a76 visited 75 links 1
checking loops digest sha256:af180e4359fc6179dc953abdcbdcaf7c146b53e1bee2b335e50dead11ccefa07 visited 76 links 1
checking loops digest sha256:09895de0407bcb0386733daa14bdb5dfa544505530c634334a05a60f161b71fc visited 77 links 1
checking loops digest sha256:968076be2e38cf897d4d6cea3faca9c037e1a4e3b4b7744fb2533e07751bd30a visited 43 links 1
checking loops digest sha256:c75d3f1f5bcd6914d0331ce5ec17c0db8f2070a2d4285f8e3ff11c6ca19168ff visited 44 links 1
checking loops digest sha256:d6e5a20b30f87216b2c758f5e7a23c437dbc3dfa1ccb177c474de152bb0ef731 visited 45 links 1
checking loops digest sha256:e7866fdc6672f827c76f6124ca3eeaff44aff8b7caf4ee1469b2ab887e7e7875 visited 46 links 2
checking loops digest sha256:c76b405781134be1dab7fe45adfb8c32104805a01de7b863e1004b66d56edf9f visited 47 links 1
checking loops digest sha256:27d719c754aacd492a6dc8a1b76619355abcf5ef473cbec02018d3c57ebbf0d5 visited 48 links 1
checking loops digest sha256:ee62de25ccc2b55d3a0495244b246fb97055b6f1c2697d837b8e94976c03756f visited 49 links 1
checking loops digest sha256:efd96aedf377e20afd95285a7c751a864260bd6a149656a4040c5b7757bdbbb6 visited 50 links 1
checking loops digest sha256:7f0a22117f8fe0172cf9209ff622b64a51aaeda21d58b5b62685a93dbe2dad25 visited 51 links 1
checking loops digest sha256:71a1c003a2b855d85582c8f6c7648c49d3fe836408a7e1b5d9b222448acb3c1b visited 52 links 1
checking loops digest sha256:c6bd343ae0007cdb979de7540f2668fe849d68ff47fa1a650a28f89104f41f1e visited 112 links 1
checking loops digest sha256:30eec89ddd9c342ef28a87f731d6e50ba977baf12d7caa7045a9d56b0e923f03 visited 113 links 1
checking loops digest sha256:b6cb293891dd62748d85aa2e00eb97e267870905edefdfe53a2ea0f3da49e88d visited 114 links 1
checking loops digest sha256:3a8f6d79cd434dc10588606993976b7b2bc038ff4a2481e857ac0168fc29a683 visited 114 links 1
checking loops digest sha256:98144d79af44407273f26589afc01901b7b296deada61a4740b0d404c5043c53 visited 115 links 1
checking loops digest sha256:b1585fdb272b31401eaac5dd46a936c1c09b4861e53e23f12ac72fc077b3c82c visited 116 links 1
checking loops digest sha256:e73cb135243c08ab2c2adc333b150b9237093315f6b38e3361f07caf2bfb4d6b visited 117 links 1
checking loops digest sha256:1e5ee5e58c8f490ae68e7e91b1575ebefc2bf6c211f302a553ff0c4925e85321 visited 110 links 1
checking loops digest sha256:c6bd343ae0007cdb979de7540f2668fe849d68ff47fa1a650a28f89104f41f1e visited 111 links 1
checking loops digest sha256:30eec89ddd9c342ef28a87f731d6e50ba977baf12d7caa7045a9d56b0e923f03 visited 112 links 1
checking loops digest sha256:b6cb293891dd62748d85aa2e00eb97e267870905edefdfe53a2ea0f3da49e88d visited 113 links 1
checking loops digest sha256:3a8f6d79cd434dc10588606993976b7b2bc038ff4a2481e857ac0168fc29a683 visited 113 links 1
checking loops digest sha256:98144d79af44407273f26589afc01901b7b296deada61a4740b0d404c5043c53 visited 114 links 1
checking loops digest sha256:b1585fdb272b31401eaac5dd46a936c1c09b4861e53e23f12ac72fc077b3c82c visited 115 links 1
checking loops digest sha256:e73cb135243c08ab2c2adc333b150b9237093315f6b38e3361f07caf2bfb4d6b visited 116 links 1
checking loops digest sha256:c6bd343ae0007cdb979de7540f2668fe849d68ff47fa1a650a28f89104f41f1e visited 110 links 1
checking loops digest sha256:30eec89ddd9c342ef28a87f731d6e50ba977baf12d7caa7045a9d56b0e923f03 visited 111 links 1
checking loops digest sha256:b6cb293891dd62748d85aa2e00eb97e267870905edefdfe53a2ea0f3da49e88d visited 112 links 1
checking loops digest sha256:3a8f6d79cd434dc10588606993976b7b2bc038ff4a2481e857ac0168fc29a683 visited 112 links 1
checking loops digest sha256:98144d79af44407273f26589afc01901b7b296deada61a4740b0d404c5043c53 visited 113 links 1
checking loops digest sha256:b1585fdb272b31401eaac5dd46a936c1c09b4861e53e23f12ac72fc077b3c82c visited 114 links 1
checking loops digest sha256:e73cb135243c08ab2c2adc333b150b9237093315f6b38e3361f07caf2bfb4d6b visited 115 links 1
checking loops digest sha256:c498f3fe97c0df55ee8dea01a72572059b93f42d235a5e439e9c9a1654d6d4e1 visited 89 links 1
checking loops digest sha256:a4ecdd704d258aa841bb3f9a1e3b0cafc59bd88810e542f8e7a0519809d78fe7 visited 90 links 1
checking loops digest sha256:4a30a219a9d7663fdd35c0a5df49c8d55018f13a0c53e10dd8efe8f7e4cc5d89 visited 91 links 1
checking loops digest sha256:8e6aee9efac8086ebac545d45c63e0d0dfcddd0d77d53e45c04d05cafdd2a8a8 visited 42 links 1
checking loops digest sha256:83151157c10d85af7c84657c71c3e3603d955160f0526fce672481da83a2e090 visited 92 links 1
checking loops digest sha256:5f193b350c8aba4883dedf97367ef3080821470661d0a2e1faf420a300cb5ca8 visited 43 links 1
checking loops digest sha256:caa1aedb2a6ce96b39b9fde1a49e1ebcb431b6da4586da0aef56df9b78221d60 visited 93 links 1
checking loops digest sha256:27e1615212f3c6ea846ed6c412df1361ce97f006ee20bb5aa2483a3b61d5cadd visited 53 links 1
checking loops digest sha256:e0850a775c17a87060c0cf6efad1020e0cbef5a44ba942bef6add5776598de53 visited 54 links 1
checking loops digest sha256:549a2fac47d713cc00f2db498ad6b5574fb03c9293aef6c7ad50a11b394c197d visited 94 links 1
checking loops digest sha256:1e68ed4e3d58a51096a7feea3947f40debf1fd9246ec977eb62ab93c81823ad9 visited 55 links 1
checking loops digest sha256:a0d177b4967a6d99f4ff117defe1c0d23d4e78ca4630febcb948ee9e4520eff3 visited 56 links 1
checking loops digest sha256:d83c7ee736be931d85b78a4a60881ced3ff9a31bb417804e45b1d30de40f94f2 visited 95 links 1
checking loops digest sha256:f1607c19a0f910ca1b8dce18843bc34e46a533c87e3524ea75798949f7a352d5 visited 44 links 1
checking loops digest sha256:48f89b630677c2cbb70e2ba05bf7a3633294e368a45bdc2c7df9d832f9e0c941 visited 96 links 1
checking loops digest sha256:9b15fed64ef16980f625aeed46ab4cd2c498690551d3a2d1e5254d551d7d6ddf visited 45 links 1
checking loops digest sha256:f626051bc94422f26f4b774a2bca105e122df36a2f32f51bd7ee470daa620b0b visited 97 links 1
checking loops digest sha256:62f77e7d6197863ac98d9e0cfa76bea0c8e05379ed5281afbe72f7fc206fe37b visited 46 links 1
checking loops digest sha256:b3a8e0e1f9ab1bfe3a36f231f676f78bb30a519d2b21e6c530c0eee8ebb4a5d0 visited 98 links 1
checking loops digest sha256:e52d08747b9d7a6d04551bb86ee3f7ee6c49f7477c8cd66f77448378cc30b92b visited 47 links 1
checking loops digest sha256:353767b239099863e13ca954e20a66c9d75f777baf239f56e399958de49bf79d visited 99 links 1
checking loops digest sha256:ad21a2b810af49a8b9241e10dfce3a016987441cc93aa72feae47dd017ddf0bb visited 100 links 1
checking loops digest sha256:01299ac65733b5a3d774265fbfe8396b8611e5e3321855dbc541cd301e71fe5e visited 48 links 1
checking loops digest sha256:42f0bec3310ddd8a55e8d62817337ca49c55a898c14ab073d07c16dee24d73d4 visited 101 links 1
checking loops digest sha256:de5872c6bb4494cebd250152ce148cd6231654e4469229f2f993984b3950b422 visited 49 links 1
checking loops digest sha256:841a05fd378a2c067058585e3691c2a3f5399206fded7a580fdbbc281003168e visited 102 links 1
checking loops digest sha256:12e2c8df501501b2bb531e941a737ffa7a2a491e849c5c5841e3b6132291bc35 visited 50 links 1
checking loops digest sha256:de482c7ed5ca67ae135ef25bf3b13194970a2f3902318f1eda3c64af2a2eb344 visited 103 links 1
checking loops digest sha256:2c4cf657337835125bc4258d0e2e546af4185bdb70f64e1b0aa46d1d78017404 visited 51 links 1
checking loops digest sha256:da4d43f295ce92630829272fad6d2e7237c6248e9cd9499e6382d6fa6d758e7b visited 104 links 1
checking loops digest sha256:21ef779311a43f0e067d0f4f600bb5451a8a7e093662086a1fe6a75d27d7892a visited 52 links 1
checking loops digest sha256:06de973bb45531d52cdbd483c5e50bcddaa2095f9515e03cfad490061cc9831e visited 105 links 1
checking loops digest sha256:00328ce57bbc14b33bd6695bc8eb32cdf2fb5f3a7d89ec14a42825e15d39df60 visited 57 links 1
checking loops digest sha256:88b54564b232405ab2165996517fece1149259cf1ea262a375db0f66039294d0 visited 106 links 1
checking loops digest sha256:d7cdaa5ca0582076c8e772cce739e32c5077cfd24f2ea33f04bb754594989a56 visited 58 links 1
checking loops digest sha256:ad3b83575249b68aab9602de378314fc221ab07a9b2ab0bb4a245ec649219f45 visited 107 links 1
checking loops digest sha256:23c657f2efda7731a3c1990b25f318fa2eb1332208f97ab9cc2a7eac70ab5a76 visited 59 links 1
checking loops digest sha256:af180e4359fc6179dc953abdcbdcaf7c146b53e1bee2b335e50dead11ccefa07 visited 60 links 1
checking loops digest sha256:826e27285307a923759de350de081d6218a04f4cff82b20c5ddaa8c60138c066 visited 108 links 6
checking loops digest sha256:64c212df34c66e6fe9fccbfebc8899c10584cfa1669c42a175d65db073b13bc0 visited 53 links 1
checking loops digest sha256:c6bd343ae0007cdb979de7540f2668fe849d68ff47fa1a650a28f89104f41f1e visited 109 links 1
checking loops digest sha256:2af4dd48399a5cf64c23fc7933e11aaf6171d80001b4b1377498ae6056b1acbf visited 54 links 1
checking loops digest sha256:30eec89ddd9c342ef28a87f731d6e50ba977baf12d7caa7045a9d56b0e923f03 visited 110 links 1
checking loops digest sha256:09895de0407bcb0386733daa14bdb5dfa544505530c634334a05a60f161b71fc visited 61 links 1
checking loops digest sha256:3a8f6d79cd434dc10588606993976b7b2bc038ff4a2481e857ac0168fc29a683 visited 111 links 1
checking loops digest sha256:98144d79af44407273f26589afc01901b7b296deada61a4740b0d404c5043c53 visited 112 links 1
checking loops digest sha256:9512d95d00d61bdec03d2b99d6ecc455ee5644ae52d10e7c4a61c93062dc97a3 visited 47 links 1
checking loops digest sha256:b1585fdb272b31401eaac5dd46a936c1c09b4861e53e23f12ac72fc077b3c82c visited 113 links 1
checking loops digest sha256:9556b82499cc0aaf86aee7f0d253e17c61b7ef73d48a295f37d98f08b04ffa7f visited 48 links 1
checking loops digest sha256:e73cb135243c08ab2c2adc333b150b9237093315f6b38e3361f07caf2bfb4d6b visited 114 links 1
checking loops digest sha256:718127812c05853f0bec61582a4a3840b1c844fe11fe1a004b5b7eb8b8b59846 visited 49 links 1
checking loops digest sha256:b6cb293891dd62748d85aa2e00eb97e267870905edefdfe53a2ea0f3da49e88d visited 111 links 1
checking loops digest sha256:392a52e4f77c40bf3321dc2feac356fac2a906a80c961748170af4ce2bce1e6a visited 55 links 1
checking loops digest sha256:30eec89ddd9c342ef28a87f731d6e50ba977baf12d7caa7045a9d56b0e923f03 visited 109 links 1
checking loops digest sha256:3a1dfb05d7257530e6349233688c3e121945c5de50f1273a7620537755d61e45 visited 50 links 1
checking loops digest sha256:c76b405781134be1dab7fe45adfb8c32104805a01de7b863e1004b66d56edf9f visited 51 links 1
checking loops digest sha256:27d719c754aacd492a6dc8a1b76619355abcf5ef473cbec02018d3c57ebbf0d5 visited 52 links 1
checking loops digest sha256:ee62de25ccc2b55d3a0495244b246fb97055b6f1c2697d837b8e94976c03756f visited 53 links 1
checking loops digest sha256:efd96aedf377e20afd95285a7c751a864260bd6a149656a4040c5b7757bdbbb6 visited 54 links 1
checking loops digest sha256:7f0a22117f8fe0172cf9209ff622b64a51aaeda21d58b5b62685a93dbe2dad25 visited 55 links 1
checking loops digest sha256:71a1c003a2b855d85582c8f6c7648c49d3fe836408a7e1b5d9b222448acb3c1b visited 56 links 1
checking loops digest sha256:27e1615212f3c6ea846ed6c412df1361ce97f006ee20bb5aa2483a3b61d5cadd visited 57 links 1
checking loops digest sha256:e0850a775c17a87060c0cf6efad1020e0cbef5a44ba942bef6add5776598de53 visited 58 links 1
checking loops digest sha256:1e68ed4e3d58a51096a7feea3947f40debf1fd9246ec977eb62ab93c81823ad9 visited 59 links 1
checking loops digest sha256:a0d177b4967a6d99f4ff117defe1c0d23d4e78ca4630febcb948ee9e4520eff3 visited 60 links 1
checking loops digest sha256:00328ce57bbc14b33bd6695bc8eb32cdf2fb5f3a7d89ec14a42825e15d39df60 visited 61 links 1
checking loops digest sha256:d7cdaa5ca0582076c8e772cce739e32c5077cfd24f2ea33f04bb754594989a56 visited 62 links 1
checking loops digest sha256:b6cb293891dd62748d85aa2e00eb97e267870905edefdfe53a2ea0f3da49e88d visited 110 links 1
checking loops digest sha256:3a8f6d79cd434dc10588606993976b7b2bc038ff4a2481e857ac0168fc29a683 visited 110 links 1
checking loops digest sha256:98144d79af44407273f26589afc01901b7b296deada61a4740b0d404c5043c53 visited 111 links 1
checking loops digest sha256:b1585fdb272b31401eaac5dd46a936c1c09b4861e53e23f12ac72fc077b3c82c visited 112 links 1
checking loops digest sha256:e73cb135243c08ab2c2adc333b150b9237093315f6b38e3361f07caf2bfb4d6b visited 113 links 1
checking loops digest sha256:064c3e311ef63912b0cc91db9681ce2d301c3e76c447febf8faa303de38cc005 visited 109 links 1
checking loops digest sha256:b6cb293891dd62748d85aa2e00eb97e267870905edefdfe53a2ea0f3da49e88d visited 110 links 1
checking loops digest sha256:3a8f6d79cd434dc10588606993976b7b2bc038ff4a2481e857ac0168fc29a683 visited 109 links 1
checking loops digest sha256:98144d79af44407273f26589afc01901b7b296deada61a4740b0d404c5043c53 visited 110 links 1
checking loops digest sha256:b1585fdb272b31401eaac5dd46a936c1c09b4861e53e23f12ac72fc077b3c82c visited 111 links 1
checking loops digest sha256:e73cb135243c08ab2c2adc333b150b9237093315f6b38e3361f07caf2bfb4d6b visited 112 links 1
checking loops digest sha256:e078af3026edb42cc26b32784baa142a79970078f7ac58f8c7b74115f4f7fb60 visited 109 links 1
checking loops digest sha256:23c657f2efda7731a3c1990b25f318fa2eb1332208f97ab9cc2a7eac70ab5a76 visited 63 links 1
checking loops digest sha256:1e5ee5e58c8f490ae68e7e91b1575ebefc2bf6c211f302a553ff0c4925e85321 visited 110 links 1
checking loops digest sha256:af180e4359fc6179dc953abdcbdcaf7c146b53e1bee2b335e50dead11ccefa07 visited 64 links 1
checking loops digest sha256:c6bd343ae0007cdb979de7540f2668fe849d68ff47fa1a650a28f89104f41f1e visited 111 links 1
checking loops digest sha256:09895de0407bcb0386733daa14bdb5dfa544505530c634334a05a60f161b71fc visited 65 links 1
checking loops digest sha256:30eec89ddd9c342ef28a87f731d6e50ba977baf12d7caa7045a9d56b0e923f03 visited 112 links 1
checking loops digest sha256:b6cb293891dd62748d85aa2e00eb97e267870905edefdfe53a2ea0f3da49e88d visited 113 links 1
checking loops digest sha256:9d693eeee1d1899cbc50b6d45df953d3835acf28ee869879b45565fccc814765 visited 43 links 1
checking loops digest sha256:3a8f6d79cd434dc10588606993976b7b2bc038ff4a2481e857ac0168fc29a683 visited 113 links 1
checking loops digest sha256:08490295488a1189099751ebeddb5992313dd2a831e07a92e66d196ddc261777 visited 44 links 1
checking loops digest sha256:98144d79af44407273f26589afc01901b7b296deada61a4740b0d404c5043c53 visited 114 links 1
checking loops digest sha256:a0eaec5a55dc2f5b2ba523018adc485ff620b9d83509b9f37186a7716e438d21 visited 45 links 1
checking loops digest sha256:b1585fdb272b31401eaac5dd46a936c1c09b4861e53e23f12ac72fc077b3c82c visited 115 links 1
checking loops digest sha256:138d9e809e386a7b800791d1f664f56d1c55f3d1ba411b950862729bc486c5ce visited 46 links 1
checking loops digest sha256:e73cb135243c08ab2c2adc333b150b9237093315f6b38e3361f07caf2bfb4d6b visited 116 links 1
checking loops digest sha256:835d5e8314340ab852a2f979ab4cd53e994dbe38366afb6eed84fe4957b980c8 visited 47 links 1
checking loops digest sha256:1e5ee5e58c8f490ae68e7e91b1575ebefc2bf6c211f302a553ff0c4925e85321 visited 109 links 1
checking loops digest sha256:f65ccfbfec288565c1d414275985547799fde0ed286c85a50bd0ec5faa01d1ac visited 56 links 1
checking loops digest sha256:c6bd343ae0007cdb979de7540f2668fe849d68ff47fa1a650a28f89104f41f1e visited 110 links 1
checking loops digest sha256:c0509a487a18b003ba05e505419ebb63e57a29158073e381f57160b5c5b86426 visited 48 links 1
checking loops digest sha256:30eec89ddd9c342ef28a87f731d6e50ba977baf12d7caa7045a9d56b0e923f03 visited 111 links 1
checking loops digest sha256:48b361d46638bfa4eee090c158a750a69c7beec3a62e703e2801125551b1b157 visited 57 links 1
checking loops digest sha256:b6cb293891dd62748d85aa2e00eb97e267870905edefdfe53a2ea0f3da49e88d visited 112 links 1
checking loops digest sha256:3a8f6d79cd434dc10588606993976b7b2bc038ff4a2481e857ac0168fc29a683 visited 112 links 1
checking loops digest sha256:98144d79af44407273f26589afc01901b7b296deada61a4740b0d404c5043c53 visited 113 links 1
checking loops digest sha256:600b4cdf20cc06a7b5a5cca5f7464296861815519af6d8a14604201b13965ab8 visited 39 links 1
checking loops digest sha256:b1585fdb272b31401eaac5dd46a936c1c09b4861e53e23f12ac72fc077b3c82c visited 114 links 1
checking loops digest sha256:0788979fc9366e21cd56311511b897a222cf91711481bcd7dc837eac2172d087 visited 40 links 1
checking loops digest sha256:114bd151f8fb0c58642d2170da4ae7d7c57977260ac2cc8905306cab6b2acabc visited 49 links 1
checking loops digest sha256:8e6aee9efac8086ebac545d45c63e0d0dfcddd0d77d53e45c04d05cafdd2a8a8 visited 41 links 1
checking loops digest sha256:0a2d643bfd24a028cd236e76575d828424ccffbfa47392bd09d8ca9dc85e2f8d visited 50 links 1
checking loops digest sha256:9a049b03f6fc40bfcf2f136320359257ed4af8513f71aa6fef47f17059bbae23 visited 51 links 1
checking loops digest sha256:f0bc318fb8965cad8d73d578cd03c63b7987dc6a79b906aada091e1b6a13443f visited 52 links 1
checking loops digest sha256:8ae4c23b80d1e7c8ff79e515fe791ebd68190bae842dda7af193db125f700452 visited 53 links 1
checking loops digest sha256:79bf08685d3138f9b109c3546780f056bc954fd69377b84a2cf23622e464897b visited 54 links 1
checking loops digest sha256:6af1f692e9496c6d0b668316eccb93276ae6b6774fa728aac31ff40a38318760 visited 55 links 1
checking loops digest sha256:749fc650cacb0f06547520d53c31505c8156e0a3be07073eddb2ef3ad9e383ba visited 56 links 1
checking loops digest sha256:14063697603e22d600d336bee6cff12c8be93509ce84a0642918d89b2aef1753 visited 57 links 1
checking loops digest sha256:72440a20f54075ac43f51a2cf0dbb2a14366b38a5c01b110ae174abc1cb44238 visited 58 links 1
checking loops digest sha256:82c01ce15b431d420eb6a1febfba7d7a2b69e5bcdcb929cb42cd3e9179d43fc4 visited 59 links 1
checking loops digest sha256:011af72a910ac4acf367eef9e6b761e0980842c30d4e9809840f4141d5163ede visited 60 links 1
checking loops digest sha256:37c20f19f3272b5ccc3a5d80587eb9deb3f4afcf568c4280fb195568da8eb1a2 visited 61 links 1
checking loops digest sha256:396f804443825586c1283a27fdcadf74abb82008bcd9b260a30912a26563f27d visited 62 links 1
checking loops digest sha256:766cb53c753baedac5dc782593e04694b3bae3aed057ac2ff98cc1aef6413137 visited 63 links 1
checking loops digest sha256:5f193b350c8aba4883dedf97367ef3080821470661d0a2e1faf420a300cb5ca8 visited 42 links 1
checking loops digest sha256:f1607c19a0f910ca1b8dce18843bc34e46a533c87e3524ea75798949f7a352d5 visited 43 links 1
checking loops digest sha256:9b15fed64ef16980f625aeed46ab4cd2c498690551d3a2d1e5254d551d7d6ddf visited 44 links 1
checking loops digest sha256:62f77e7d6197863ac98d9e0cfa76bea0c8e05379ed5281afbe72f7fc206fe37b visited 45 links 1
checking loops digest sha256:e52d08747b9d7a6d04551bb86ee3f7ee6c49f7477c8cd66f77448378cc30b92b visited 46 links 1
checking loops digest sha256:01299ac65733b5a3d774265fbfe8396b8611e5e3321855dbc541cd301e71fe5e visited 47 links 1
checking loops digest sha256:de5872c6bb4494cebd250152ce148cd6231654e4469229f2f993984b3950b422 visited 48 links 1
checking loops digest sha256:12e2c8df501501b2bb531e941a737ffa7a2a491e849c5c5841e3b6132291bc35 visited 49 links 1
checking loops digest sha256:2c4cf657337835125bc4258d0e2e546af4185bdb70f64e1b0aa46d1d78017404 visited 50 links 1
checking loops digest sha256:21ef779311a43f0e067d0f4f600bb5451a8a7e093662086a1fe6a75d27d7892a visited 51 links 1
checking loops digest sha256:64c212df34c66e6fe9fccbfebc8899c10584cfa1669c42a175d65db073b13bc0 visited 52 links 1
checking loops digest sha256:2af4dd48399a5cf64c23fc7933e11aaf6171d80001b4b1377498ae6056b1acbf visited 53 links 1
checking loops digest sha256:392a52e4f77c40bf3321dc2feac356fac2a906a80c961748170af4ce2bce1e6a visited 54 links 1
checking loops digest sha256:f65ccfbfec288565c1d414275985547799fde0ed286c85a50bd0ec5faa01d1ac visited 55 links 1
checking loops digest sha256:48b361d46638bfa4eee090c158a750a69c7beec3a62e703e2801125551b1b157 visited 56 links 1
checking loops digest sha256:9f484139a27415ae2e8612bf6c65a8101a18eb5e9b7809e74ca63a45a65f17f4 visited 64 links 1
checking loops digest sha256:1e472b39b105d349bcd069c4a711b44a2fffb8e274714bb07ecfff69a9a7f67b visited 65 links 2
checking loops digest sha256:e52d08747b9d7a6d04551bb86ee3f7ee6c49f7477c8cd66f77448378cc30b92b visited 45 links 1
checking loops digest sha256:0e12831a7047f759733b21f028525039607350b1b1b4fe904595427e72ea0d9b visited 93 links 1
checking loops digest sha256:01299ac65733b5a3d774265fbfe8396b8611e5e3321855dbc541cd301e71fe5e visited 46 links 1
checking loops digest sha256:75c3e223190bf1a1fa2af808d1dfcdffe33727d57eb0028b5a52ad893480eeb9 visited 94 links 1
checking loops digest sha256:de5872c6bb4494cebd250152ce148cd6231654e4469229f2f993984b3950b422 visited 47 links 1
checking loops digest sha256:c498f3fe97c0df55ee8dea01a72572059b93f42d235a5e439e9c9a1654d6d4e1 visited 95 links 1
checking loops digest sha256:12e2c8df501501b2bb531e941a737ffa7a2a491e849c5c5841e3b6132291bc35 visited 48 links 1
checking loops digest sha256:a4ecdd704d258aa841bb3f9a1e3b0cafc59bd88810e542f8e7a0519809d78fe7 visited 96 links 1
checking loops digest sha256:2c4cf657337835125bc4258d0e2e546af4185bdb70f64e1b0aa46d1d78017404 visited 49 links 1
checking loops digest sha256:4a30a219a9d7663fdd35c0a5df49c8d55018f13a0c53e10dd8efe8f7e4cc5d89 visited 97 links 1
checking loops digest sha256:21ef779311a43f0e067d0f4f600bb5451a8a7e093662086a1fe6a75d27d7892a visited 50 links 1
checking loops digest sha256:83151157c10d85af7c84657c71c3e3603d955160f0526fce672481da83a2e090 visited 98 links 1
checking loops digest sha256:64c212df34c66e6fe9fccbfebc8899c10584cfa1669c42a175d65db073b13bc0 visited 51 links 1
checking loops digest sha256:caa1aedb2a6ce96b39b9fde1a49e1ebcb431b6da4586da0aef56df9b78221d60 visited 99 links 1
checking loops digest sha256:2af4dd48399a5cf64c23fc7933e11aaf6171d80001b4b1377498ae6056b1acbf visited 52 links 1
checking loops digest sha256:549a2fac47d713cc00f2db498ad6b5574fb03c9293aef6c7ad50a11b394c197d visited 100 links 1
checking loops digest sha256:392a52e4f77c40bf3321dc2feac356fac2a906a80c961748170af4ce2bce1e6a visited 53 links 1
checking loops digest sha256:d83c7ee736be931d85b78a4a60881ced3ff9a31bb417804e45b1d30de40f94f2 visited 101 links 1
checking loops digest sha256:f65ccfbfec288565c1d414275985547799fde0ed286c85a50bd0ec5faa01d1ac visited 54 links 1
checking loops digest sha256:c75d3f1f5bcd6914d0331ce5ec17c0db8f2070a2d4285f8e3ff11c6ca19168ff visited 66 links 1
checking loops digest sha256:48b361d46638bfa4eee090c158a750a69c7beec3a62e703e2801125551b1b157 visited 55 links 1
checking loops digest sha256:48f89b630677c2cbb70e2ba05bf7a3633294e368a45bdc2c7df9d832f9e0c941 visited 102 links 1
checking loops digest sha256:f626051bc94422f26f4b774a2bca105e122df36a2f32f51bd7ee470daa620b0b visited 103 links 1
checking loops digest sha256:b3a8e0e1f9ab1bfe3a36f231f676f78bb30a519d2b21e6c530c0eee8ebb4a5d0 visited 104 links 1
checking loops digest sha256:8e6aee9efac8086ebac545d45c63e0d0dfcddd0d77d53e45c04d05cafdd2a8a8 visited 39 links 1
checking loops digest sha256:353767b239099863e13ca954e20a66c9d75f777baf239f56e399958de49bf79d visited 105 links 1
checking loops digest sha256:ad21a2b810af49a8b9241e10dfce3a016987441cc93aa72feae47dd017ddf0bb visited 106 links 1
checking loops digest sha256:5f193b350c8aba4883dedf97367ef3080821470661d0a2e1faf420a300cb5ca8 visited 40 links 1

[BtbN]

I don't think that's really the issue.
The function recurses into a path, and wants to check if in that path some entry exists twice. Whenever it returns out of the path, it cleans up the already visited nodes via the defered delete. So at least on first glance, the defer looks correct to me. It wants to detect loops after all, not an entry existing twice.

[BtbN]

How about:

diff --git a/cache/remotecache/v1/utils.go b/cache/remotecache/v1/utils.go
index cf014913e..d2e3de0bf 100644
--- a/cache/remotecache/v1/utils.go
+++ b/cache/remotecache/v1/utils.go
@@ -134,13 +134,21 @@ func (s *normalizeState) removeLoops(ctx context.Context) {
        }

        visited := map[digest.Digest]struct{}{}
+       processed := map[digest.Digest]struct{}{}

        for _, d := range roots {
-               s.checkLoops(ctx, d, visited)
+               s.checkLoops(ctx, d, visited, processed)
        }
 }

-func (s *normalizeState) checkLoops(ctx context.Context, d digest.Digest, visited map[digest.Digest]struct{}) {
+func (s *normalizeState) checkLoops(ctx context.Context, d digest.Digest, visited map[digest.Digest]struct{}, processed map[digest.Digest]struct{}) {
+       if _, ok := processed[d]; ok {
+               return
+       }
+       defer func() {
+               processed[d] = struct{}{}
+       }()
+
        it, ok := s.byKey[d]
        if !ok {
                return
@@ -166,7 +174,7 @@ func (s *normalizeState) checkLoops(ctx context.Context, d digest.Digest, visite
                                }
                                delete(links[l], id)
                        } else {
-                               s.checkLoops(ctx, id, visited)
+                               s.checkLoops(ctx, id, visited, processed)
                        }
                }
        }

That should avoid it ever re-processing the same tree over and over again when reaching it from different ends, hopefully reducing the amount of processing needed.
I'm not set up to test this condition, so not sure how to validate it.

[tonistiigi]

@israelglar Can you see if #6129 fixes the issue for you? This whole code should be removed there.

[israelglar]

@tonistiigi these are the results with the new remote cache. It fixes the issue 🥳

==== Runs with current cache storage ===
Run #1 Duration: 12 seconds (push cache)
Run #2 Duration: 4 seconds (only pull)
Run #3 Duration: 4 seconds (only pull)
Run #4 Duration: 3 seconds (only pull)
Run #5 Duration: 5 seconds (only pull)
Run #6 Duration: 60 seconds + canceled (stuck)


==== Runs with new cache storage ====
Run #1 Duration: 9 seconds (push cache)
Run #2 Duration: 4 seconds (only pull)
Run #3 Duration: 3 seconds (only pull)
Run #4 Duration: 4 seconds (only pull)
Run #5 Duration: 3 seconds (only pull)
Run #6 Duration: 3 seconds (only pull)
Run #7 Duration: 3 seconds (only pull)
Run #8 Duration: 3 seconds (only pull) 
Run #9 Duration: 3 seconds (only pull)
Run #10 Duration: 3 seconds (only pull)