-
Notifications
You must be signed in to change notification settings - Fork 16.4k
Add Cohere Provider #34921
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Add Cohere Provider #34921
Conversation
phanikumv
left a comment
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Do we need the .svg files to be added to this PR?
f21e41f to
9a7f559
Compare
Yes, we do need to add them, as the provider dependency changes, the breeze output command interface also changes and that needs to be reflected. Also, they are auto-generated and enforced via |
tests/system/providers/cohere/example_cohere_embedding_operator.py
Outdated
Show resolved
Hide resolved
tests/system/providers/cohere/example_cohere_embedding_operator.py
Outdated
Show resolved
Hide resolved
tests/system/providers/cohere/example_cohere_embedding_operator.py
Outdated
Show resolved
Hide resolved
Co-authored-by: Josh Fell <48934154+josh-fell@users.noreply.github.com>
Co-authored-by: Josh Fell <48934154+josh-fell@users.noreply.github.com>
Co-authored-by: Josh Fell <48934154+josh-fell@users.noreply.github.com>
Co-authored-by: Hussein Awala <hussein@awala.fr>
Co-authored-by: Hussein Awala <hussein@awala.fr>
Co-authored-by: Pankaj Singh <98807258+pankajastro@users.noreply.github.com>
Co-authored-by: Josh Fell <48934154+josh-fell@users.noreply.github.com>
Co-authored-by: Josh Fell <48934154+josh-fell@users.noreply.github.com>
0bb4e54 to
7f1e0ae
Compare
* Add Cohere Provider * Add Cohere Provider * Move link to seealso sphinx directive * Updated check for parameters * Update dependency of the cohere * Move the dag out of rst and into system tests * Add dependency to of cohere python sdk * Add cache_property for the cohere client * Remove unwanted get_conn method * Add correct label to password field * Expose timeout, max_retries and api_url to user * Fix documentation * Update interface of CohereEmbeddingOperator operator * Updated testcases * Updated testcases * Fix static check and docs build * Update CONTRIBUTING.rst Co-authored-by: Josh Fell <48934154+josh-fell@users.noreply.github.com> * Update docs/apache-airflow-providers-cohere/operators/embedding.rst Co-authored-by: Josh Fell <48934154+josh-fell@users.noreply.github.com> * Update airflow/providers/cohere/operators/embedding.py Co-authored-by: Josh Fell <48934154+josh-fell@users.noreply.github.com> * Update airflow/providers/cohere/hooks/cohere.py Co-authored-by: Hussein Awala <hussein@awala.fr> * Update airflow/providers/cohere/CHANGELOG.rst Co-authored-by: Hussein Awala <hussein@awala.fr> * Address the PR comments * Resolve conflicts * Fix breaking tests * Fix static checks * Update airflow/providers/cohere/operators/embedding.py Co-authored-by: Pankaj Singh <98807258+pankajastro@users.noreply.github.com> * Fix docstring * Add note for initial release * Add security.rst file * Update airflow/providers/cohere/hooks/cohere.py * Update airflow/providers/cohere/operators/embedding.py Co-authored-by: Josh Fell <48934154+josh-fell@users.noreply.github.com> * Update docs/apache-airflow-providers-cohere/operators/embedding.rst Co-authored-by: Josh Fell <48934154+josh-fell@users.noreply.github.com> * Add ref to sequrity.rst * Update docs/apache-airflow-providers-cohere/security.rst * Add /changelog.rst * Resolve conflicts --------- Co-authored-by: Josh Fell <48934154+josh-fell@users.noreply.github.com> Co-authored-by: Hussein Awala <hussein@awala.fr> Co-authored-by: Pankaj Singh <98807258+pankajastro@users.noreply.github.com> Co-authored-by: Pankaj <pankaj.singh@astronomer.io>
This PR is part of our larger effort to add first-class integrations to support LLMOps that was presented at Airflow Summit. This PR specifically adds the Cohere Provider. Cohere is a renowned platform offering a range of AI Models tailored for various NLP tasks. In this iteration, we are integrating with their Embeddings Model.
The primary objective of this Provider is to present users with an alternative embedding model. This allows them to generate vectors for their proprietary data, a pivotal step towards establishing integrations with LLM models like ChatGPT.
Example DAG:
The
CohereEmbeddingOperatorcan accept either a list of strings or a callable returning a list of strings.Email Discussion related to the effort can be found here - https://lists.apache.org/thread/0d669fmy4hn29h5c0wj0ottdskd77ktp