Aggressively cache entry points in process

[uranusjr]

importlib.metadata.distributions() reads information from the actual installations, which is a lot of IO that we can avoid by caching.

The benefit of this depends on how many packages you have in your installation. It’s nearly zero with a bare Airflow installation, and I observed a ~7% save (17s to 16s) for the webserver to finish init (launch until when_ready is emitted) in a setup with all official providers installed.

The downside is we are now persisting a lot of small objects in memory. I wonder whether there’s a good time we can purge those.

[ashb]

I was profling memory usage (not speed), and this seemed to be a cause of a lot of bloat -- and it's suprisingly expensive to get the dist name (involves parsing a lot of files for every dist in airflow).

I think there are three options here:

1. Only compute this key if the entrypoint group matches (this limits the expensive operation to just dists we actually care about, instead of all
2. Use the path component to be the cache key (see below)
3. Cache based on the entrypoint classpath
4. Don't cache at all. This only catches a case when you have multiple copies of the same dist in multiple cases (which is rare to hit outside of being an Airflow developer anyway).

On point 2, it doesn't seem possible to do this using public methods, but this:

In [14]: d._path
Out[14]: PosixPath('/home/ash/airflow/.venv/lib/python3.11/site-packages/greenlet-2.0.2.dist-info')

In [15]: d._path.stem
Out[15]: 'greenlet-2.0.2'

Doing this might break for less common ways of shipping dists though, so it's likely not a good option, not without a fallback anyway.

[uranusjr]

I checked the callers and currently this is used by loading airflow.plugins and apache_airflow_provider. Both of these implement their own deduplication logic, so I think it is safe to remove this entirely. Although this would not actually help the latter case, which still accesses metadata anyway…