FPM child processes maxed out since v3.3.0 and TYPO3 11LTS + PHP 8.0 + Cloudflare

[DenisMir]

We got a weird problem with the extension that leads to maxed out FPM child processes with the latest v3.3.0 version and TYPO3 11LTS (latest patch version) + PHP 8.0 that we can't find the reason for.

I try do describe what information we collected till now:

• We had v3.2.0 with TYPO3 10LTS + PHP 7.4 and Cloudflare running for over a year without problems
• After we migrated to TYPO3 11LTS + PHP 8.0 and v3.3.0 of the proxycachemanager the problems began
• The PHP slow log shows hundreds of struggling/hanging requests (a lot of them from the Guzzle underlying Curl (version 7.61.1) called by the proxymanager)
• The FPM child processes start adding up pretty fast after a restart of the FPM master process
• Removing v3.3.0 immediately stops the problem from happening
• ReInstalling/Requiring the v3.3.0 starts the problem immediately again (child processes add up to the limit till the server gets unreachable from all of the open requests)
• The server environment hasn't changed except the PHP version

In the end this whole behaviour leads to hundreds of health check failures in cloudflare.

@bmack Have you got any idea which could have changed from v3.2.0 to v3.3.0 that could lead to these problems? Unfortunately trying out v3.2.0 is not possible with TYPO3 11LTS + PHP 8.0 due multiple things not being compatible with PHP 8.0.

Right now I'm out of ideas what the extension could do that leads to these problems since from looking at the code it "just" calls the CF API in the specific provider.

Perhaps something comes to your mind that I could follow/debug through.

[DenisMir]

Hi @bmack,

One followup here. I found the reason for the load of FPM processes.

The cloudflare provider does include something that doesn't make sense:

https://github.com/b13/proxycachemanager/blob/master/Classes/Provider/CloudflareProxyProvider.php#L90

Flush All Caches or Flush Frontend Caches does lead to reaching the else part which purges the whole cache in chunks. For big sites (multiple zones, hundreds of pages, hundreds of image urls) this can add up to thousands of purge requests. (in the end it is cache_tx_proxy entry count divided by chunk size == request count)

The fix is simple. Since we want to clear the whole cache (well the interface method says it flushAllUrls) it is enough to do the:

            foreach ($this->getZones() as $domain => $zoneId) {
                try {
                    $this->getClient($zoneId)->post('purge_cache', ['json' => ['purge_everything' => true]]);
                } catch (TransferException $e) {
                    $this->logger->error('Could not flush URLs for {zone} via POST "purge_cache"', [
                        'urls' => $urls,
                        'zone' => $zoneId,
                        'exception' => $e,
                    ]);
                }
            }

This leads to only requests and does also clear the whole cache for each zone.

Do you see any problems with this?

The difference to 3.2.0 is that 3.3.0 now includes images on the page in the cache table (I can only guess but this one could be the reason for this: cbbdf83) which increases the per page count of urls in the cache table by 10 to 20x. This results in a lot more purge requests compared to 3.2.0 when purging the whole cache by urls. (which got the FPM setup to get down in one Flush All request from the BE)

[DenisMir]

Pull request added.

[DenisMir]

Ok one more followup on this issue here. The fix above didn't completely solve the issue. I added one more "fix" which drops the images from being added to the cache: Communiacs@75ae964

The test is still running in production but for now the problem seems to be gone.

@bmack Is this asset collector call really that expensive? Or is it just the iteration for the images (+ adding it to the cache table) which costs that much? The asset collector can't be the problem because it is already filled with the images right? So the iteration must be the problem. In our case it is about 50 iterations (images) per page that gets added to the proxy cache table.

Update:

The checks in production show that this last change fixed the problem completely. (removing the assets iteration from the frontend hook)

[bmack]

I think we can drop this in v11.

[bmack]

Removed this in v3.4.0 - thanks for the pointer