add openvino, trt

[michaelfeil]

openvino not working, need to use #454

openvino:

• tried openvino==2024.4 + nightly, but both keep breaking
• openvino via poetry install does not work. Needs to install via pip after installation (uninstall openvino and install the same version builds the wheel / extensions / etc correctly)

tensorrt:

• using cuda12
• cuda12 has tensorrt==10.2 as only compatible version.
• tensorrt==10.2 crashes on linux, as its linked to a windows dll
• tensorrt==10.3 works, but the optimum inplementation has issues
• onnxruntime==1.20 leaves tensorrt==10.5, but has has no compatible openvino distribution.

[greptile-apps]

PR Summary

This PR adds OpenVINO and TensorRT optimizations to improve model inference performance across different hardware platforms.

• Added OpenVINO support in /libs/infinity_emb/Docker.template.yaml with INFINITY_ENGINE="optimum" for CPU builds
• Updated TensorRT support with CUDA 12.3.2 and TensorRT 10.3.0 in /libs/infinity_emb/Docker.template.yaml
• Added provider-specific optimizations in /libs/infinity_emb/infinity_emb/transformer/utils_optimum.py for OpenVINO and TensorRT
• Added quantized model support for OpenVINO in /libs/infinity_emb/infinity_emb/transformer/embedder/optimum.py
• Updated dependencies in pyproject.toml to use OpenVINO 2024.4.0 and TensorRT 10.6.0

_{11 file(s) reviewed, 7 comment(s)
Edit PR Review Bot Settings | Greptile}

[codecov-commenter]

⚠️ Please install the to ensure uploads and comments are reliably processed by Codecov.

Codecov Report

Attention: Patch coverage is 33.33333% with 12 lines in your changes missing coverage. Please review.

Project coverage is 73.02%. Comparing base (cdbe888) to head (bb34e3e).

Files with missing lines	Patch %	Lines
...nity_emb/infinity_emb/transformer/utils_optimum.py	33.33%	12 Missing ⚠️

❗ Your organization needs to install the Codecov GitHub app to enable full functionality.

❗ There is a different number of reports uploaded between BASE (cdbe888) and HEAD (bb34e3e). Click for more details.

HEAD has 1 upload less than BASE

Flag	BASE (cdbe888)	HEAD (bb34e3e)
	2	1

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #460      +/-   ##
==========================================
- Coverage   79.23%   73.02%   -6.21%     
==========================================
  Files          42       42              
  Lines        3380     3392      +12     
==========================================
- Hits         2678     2477     -201     
- Misses        702      915     +213     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.