Fix import latency [2/N]: Implement native _is_package_available

[albertvillanova]

Implement native _is_package_available.

This is the 2nd PR to fix the TRL import latency I am working on:

• Improve TRL import latency #5130

This PR updates the way the _is_package_available utility function is imported and implemented in the codebase. The main change is that the function is now defined locally in trl/import_utils.py instead of being imported from the transformers library, and all internal imports have been updated accordingly.

Context

Both trl/import_utils.py and trl/_compat.py import _is_package_available from transformers at the top level, which was introduced by:

• Use transformers utilities when possible #2064

Both files are eagerly loaded when import trl runs, triggering a full transformers load (~3s).

Solution

Replace transformers..utils.import_utils._is_package_available with local implementation of _is_package_available.

After the fix (#5128 and #5129), the import latency passed from ~3s to <0.5s:

import time, sys

t = time.time(); import trl
print(f"import trl: {time.time() - t:.2f}s")   # <0.5s
print("transformers" in sys.modules)           # False

Key changes:

Refactoring of package availability utility:

• Moved the _is_package_available function from being imported from transformers.utils.import_utils to a local implementation in trl/import_utils.py, ensuring the codebase no longer depends on the external implementation.
• Updated the import statement in trl/_compat.py to use the local _is_package_available from .import_utils instead of the one from transformers.utils.import_utils.

Enhancements to local implementation:

• The new _is_package_available function is copied from the transformers library and includes logic to check for a package's existence and optionally retrieve its version, handling edge cases such as distribution name differences and editable installs.

[HuggingFaceDocBuilderDev]

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

[qgallouedec]

Very cool. 2.5 seconds multiplied by the number of trl imports is probably a considerable amount of time you've saved humanity.