feat: return 410 status code on missing JS assets

[barjin]

As per the findings of the SEO team:

The 404 JS files (e.g. docs.apify.com/assets/js/3b91d6c4.c7552f0b.js) are auto-generated during deployment. Every time the site is updated and published, these JS files are generated with new random names and the old ones are deleted. Googlebot had cached the old filenames and kept trying to fetch them after they were already gone, causing the 404s.
Why we should solve this: Googlebot is wasting its crawl budget on docs.apify.com requesting JS files that no longer exist, causing an 8% 404 instead of crawling actual content pages.

Returning 410 Gone instead of 404 Not Found on regenerated JS assets should send a stronger signal that the asset path is no longer valid to the consumer.

[apify-service-account]

🗑️ Preview for this PR was deleted.

[Copilot]

Pull request overview

Updates the Nginx reverse-proxy configuration for the Apify docs sites to return 410 Gone (instead of 404 Not Found) when hashed Docusaurus JS/CSS assets are missing, helping crawlers and caches stop requesting stale bundles after deployments.

Changes:

• Adds a shared named location that returns HTTP 410 and maps upstream 404s for hashed assets to it.
• Introduces dedicated regex location blocks for /assets/js/*.js and /assets/css/*.css in the main docs site and each proxied sub-site (/sdk/*, /api/client/*, /cli).

[barjin]

For the record, this doesn't seem to have helped (see, e.g., the response to https://docs.apify.com/assets/js/abc.js )

[TC-MO]

I think nginx takes a while to start working properly (or at least it did for regular 301's I was doing so maybe here is the same case? I'll observe this for the next day/two and see how it behaves)

[B4nan]

nginx is fast, but there is a CDN in front of it, which takes a bit. last time I was battling something here, I tried 7 different variants before surrending, only to see the other day that it actually worked.

[barjin]

Okay, confirmed now.