M1/M2 MacOS Users can not process your models

[phdykd]

Hi,

I am on M2 MacOS that has 12 CPU + 38 GPU (as platform computer) + 96GB processor + 2TB storage.
This macOS systems are new and considered as "mps".
Apparently, I am not able to run a single library/model from your source.

Please modify your systems according to M2 MacOS users too. I would like to try your librarys. I hope someone can answer my query.
Thanks

[julian-risch]

@vblagoje You're using Haystack with Apple M2, right? Do you know any required steps that are not listed in the installation instructions in the documentation? https://docs.haystack.deepset.ai/docs/installation#apple-silicon-m1

[vblagoje]

@phdykd, which particular part didn't work for you? I made a quick test similar to this:

from haystack.nodes.prompt import PromptNode
pn = PromptNode("google/flan-t5-base", max_length=256, devices=["mps"], model_kwargs={"stream": True})
pn("Tell me more about Large Language Models in NLP, be elaborate")

Aside from a few unsupported torch ops, flan-t5-base works well using the latest transformers and torch installed as described on the installation page. I verified under debugger that the device used is indeed mps. Let me know more details and we'll dig deeper.

[phdykd]

Hi folks,
Thanks for your response.

I am on M2 Max Chip MacOS that has 12 CPU + 38 GPU.

1-I am following this link that you have: https://haystack.deepset.ai/tutorials/01_basic_qa_pipeline
1.1- However, it shows that it is not using my available 38 GPU.
"from haystack.document_stores import InMemoryDocumentStore
document_store = InMemoryDocumentStore(use_bm25=True)"

INFO - haystack.modeling.utils - Using devices: CPU - Number of GPUs: 0.
iIt should have picked up the 38 GPU.

1.2- Similar warning for : "from haystack.nodes import FARMReader

reader = FARMReader(model_name_or_path="deepset/roberta-base-squad2", use_gpu=True)
"

INFO - haystack.modeling.utils - Using devices: CPU - Number of GPUs: 0
INFO - haystack.modeling.utils - Using devices: CPU - Number of GPUs: 0
INFO - haystack.modeling.model.language_model - * LOADING MODEL: 'deepset/roberta-base-squad2' (Roberta)
/Users/phdykd/miniconda3/envs/Haystack/lib/python3.11/site-packages/torch/_utils.py:776: UserWarning: TypedStorage is deprecated. It will be removed in the future and UntypedStorage will be the only storage class. This should only matter to you if you are using storages directly. To access UntypedStorage directly, use tensor.untyped_storage() instead of tensor.storage()
return self.fget.get(instance, owner)()
INFO - haystack.modeling.model.language_model - Auto-detected model language: english
INFO - haystack.modeling.model.language_model - Loaded 'deepset/roberta-base-squad2' (Roberta model) from model hub.
INFO - haystack.modeling.utils - Using devices: CPU - Number of GPUs: 0

2- I am following this link that you have: https://haystack.deepset.ai/tutorials/03_scalable_qa_system
2.1- for this code: I got the following warning: %%bash

"pip install --upgrade pip
pip install farm-haystack[preprocessing,elasticsearch]"

WARNING: farm-haystack 0.1.0.post2 does not provide the extra 'elasticsearch'
WARNING: farm-haystack 0.1.0.post2 does not provide the extra 'preprocessing'

2.2- !pip install 'farm-haystack[all]' ## or 'all-gpu' for the GPU-enabled dependencies

WARNING: farm-haystack 0.1.0.post2 does not provide the extra 'all'

2.3- from haystack.telemetry import tutorial_running

tutorial_running(3)

06/05/2023 23:32:00 - INFO - root - Using localhost sqlite as the database backend. as Database not configured. Add a qa_config.py file in the Python path with DATABASE_URL set.Continuing with the default sqlite on localhost.

RuntimeError: Working outside of application context.

This typically means that you attempted to use functionality that needed
the current application. To solve this, set up an application context
with app.app_context(). See the documentation for more information.

2.4- %%bash

wget https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-7.9.2-linux-x86_64.tar.gz -q
tar -xzf elasticsearch-7.9.2-linux-x86_64.tar.gz
chown -R daemon:daemon elasticsearch-7.9.2

CalledProcessError: Command 'b'\nwget https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-7.9.2-linux-x86_64.tar.gz -q\ntar -xzf elasticsearch-7.9.2-linux-x86_64.tar.gz\nchown -R daemon:daemon elasticsearch-7.9.2\n'' returned non-zero exit status 1.

chown: elasticsearch-7.9.2/modules/x-pack-ccr/NOTICE 4.txt: Operation not permitted
chown: elasticsearch-7.9.2/modules/x-pack-ccr/.x-pack-ccr-7.9.2 2.jar.icloud: Operation not permitted
chown: elasticsearch-7.9.2/modules/x-pack-ccr/x-pack-ccr-7.9.2.jar: Operation not permitted
chown: elasticsearch-7.9.2/modules/x-pack-ccr/NOTICE 2.txt: Operation not permitted
chown: elasticsearch-7.9.2/modules/x-pack-ccr/NOTICE 3.txt: Operation not permitted
chown: elasticsearch-7.9.2/modules/x-pack-ccr/.plugin-descriptor 3.properties.icloud: Operation not permitted
chown: elasticsearch-7.9.2/modules/x-pack-ccr/.plugin-security 3.policy.icloud: Operation not permitted
chown: elasticsearch-7.9.2/modules/x-pack-ccr/LICENSE.txt: Operation not permitted
chown: elasticsearch-7.9.2/modules/x-pack-ccr/.plugin-descriptor 2.properties.icloud: Operation not permitted
chown: elasticsearch-7.9.2/modules/x-pack-ccr/.plugin-security 4.policy.icloud: Operation not permitted
chown: elasticsearch-7.9.2/modules/x-pack-ccr/plugin-descriptor.properties: Operation not permitted
chown: elasticsearch-7.9.2/modules/x-pack-ccr/plugin-security.policy: Operation not permitted
chown: elasticsearch-7.9.2/modules/x-pack-ccr: Operation not permitted
chown: elasticsearch-7.9.2/modules: Operation not permitted
chown: elasticsearch-7.9.2: Operation not permitted

2.5- from haystack.utils import launch_es
launch_es()

06/05/2023 23:34:25 - INFO - root - Using localhost sqlite as the database backend. as Database not configured. Add a qa_config.py file in the Python path with DATABASE_URL set.Continuing with the default sqlite on localhost.

RuntimeError: Working outside of application context.

This typically means that you attempted to use functionality that needed
the current application. To solve this, set up an application context
with app.app_context(). See the documentation for more information.

2.6- from haystack import Pipeline
from haystack.nodes import TextConverter, PreProcessor

indexing_pipeline = Pipeline()
text_converter = TextConverter()
preprocessor = PreProcessor(
clean_whitespace=True,
clean_header_footer=True,
clean_empty_lines=True,
split_by="word",
split_length=200,
split_overlap=20,
split_respect_sentence_boundary=True,
)

06/05/2023 23:36:01 - INFO - root - Using localhost sqlite as the database backend. as Database not configured. Add a qa_config.py file in the Python path with DATABASE_URL set.Continuing with the default sqlite on localhost.

3- bitsandbytes does not work with M2 Max chip MacOS.

Thanks for your help

[julian-risch]

Hello @phdykd could you please follow the installation instructions from the documentation on M1/M2 https://docs.haystack.deepset.ai/docs/installation#apple-silicon-m1 :

# some additional dependencies needed on m1 mac
brew install postgresql
brew install cmake
brew install rust

# haystack installation
GRPC_PYTHON_BUILD_SYSTEM_ZLIB=true pip install farm-haystack

Seeing version "farm-haystack 0.1.0.post2" in the logs you posted indicates that the installation didn't work because none of the versions released on pypi matched your platform. That should be resolved if you use the commands listed above. If should see version 1.17.1 in your logs during the installation then. 👍

[phdykd]

Thanks for your reply.
Please see below:

1-
%%bash
pip install farm-haystack[preprocessing,elasticsearch]
pip install --upgrade farm-haystack[preprocessing,elasticsearch] tells:
"requirement already satisfied".

But,
%%bash
pip show pip
pip show farm-haystack
pip show preprocessing
pip show elasticsearch

tells "WARNING: Package(s) not found: preprocessing"
See this:
"Name: pip
Version: 23.1.2
Summary: The PyPA recommended tool for installing Python packages.
Home-page: https://pip.pypa.io/
Author: The pip developers
Author-email: distutils-sig@python.org
License: MIT
Location: /Users/phdykd/miniconda3/envs/Haystack/lib/python3.11/site-packages
Requires:
Required-by:
Name: farm-haystack
Version: 1.17.1
Summary: Neural Question Answering & Semantic Search at Scale. Use modern transformer based models like BERT to find answers in large document collections
Home-page:
Author:
Author-email: "deepset.ai" malte.pietsch@deepset.ai
License:
Location: /Users/phdykd/miniconda3/envs/Haystack/lib/python3.11/site-packages
Requires: azure-ai-formrecognizer, boilerpy3, canals, dill, events, generalimport, huggingface-hub, jsonschema, more-itertools, networkx, pandas, posthog, protobuf, pydantic, quantulum3, rank-bm25, requests, requests-cache, scikit-learn, sentence-transformers, sseclient-py, tenacity, tiktoken, tqdm, transformers, typing-extensions
Required-by:
WARNING: Package(s) not found: preprocessing
Name: elasticsearch
Version: 7.17.9
Summary: Python client for Elasticsearch
Home-page: https://github.com/elastic/elasticsearch-py
Author: Honza Král, Nick Lang
Author-email: honza.kral@gmail.com, nick@nicklang.com
License: Apache-2.0
Location: /Users/phdykd/miniconda3/envs/Haystack/lib/python3.11/site-packages
Requires: certifi, urllib3
Required-by: "

2- The instructions given in the web is failing for M2 Max MacOS.
I have to do the following:
!brew install wget

%%bash
wget https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-7.17.9-darwin-aarch64.tar.gz -q
tar -xzf elasticsearch-7.17.9-darwin-aarch64.tar.gz

%%bash --bg
sudo -u daemon -- elasticsearch-7.17.9/bin/elasticsearch

3-
import time
time.sleep(30)

from haystack.utils import launch_es
launch_es()

/bin/sh: docker: command not found
WARNING - haystack.utils.doc_store - Tried to start Elasticsearch through Docker but this failed. It is likely that there is already an existing Elasticsearch instance running.

4- import os
from haystack.document_stores import ElasticsearchDocumentStore

Get the host where Elasticsearch is running, default to localhost
host = os.environ.get("ELASTICSEARCH_HOST", "localhost")

document_store = ElasticsearchDocumentStore(
host=host,
username="",
password="",
index="document"
)

gives this warning:
/Users/phdykd/miniconda3/envs/Haystack/lib/python3.11/site-packages/elasticsearch/connection/base.py:200: ElasticsearchWarning: Elasticsearch built-in security features are not enabled. Without authentication, your cluster could be accessible to anyone. See https://www.elastic.co/guide/en/elasticsearch/reference/7.17/security-minimal-setup.html to enable security.
warnings.warn(message, category=ElasticsearchWarning)

Should I use username and password for the above line? How can I set?

5- files_to_index = [doc_dir + "/" + f for f in os.listdir(doc_dir)]
indexing_pipeline.run_batch(file_paths=files_to_index)

Gives this:

“”INFO - haystack.pipelines.base - It seems that an indexing Pipeline is run, so using the nodes' run method instead of run_batch.”

“WARNING - haystack.nodes.preprocessor.preprocessor - We found one or more sentences whose word count is higher than the split length.
WARNING - haystack.nodes.preprocessor.preprocessor - Document 25d97fdfb5d033c825e91fcced3521ad is 10636 characters long after preprocessing, where the maximum length should be 10000. Something might be wrong with the splitting, check the document affected to prevent issues at query time.
WARNING - haystack.nodes.preprocessor.preprocessor - Document 8cde72af54363c7dbdbcbadef960ae0f is 12398 characters long after preprocessing, where the maximum length should be 10000. Something might be wrong with the splitting, check the document affected to prevent issues at query time.
WARNING - haystack.nodes.preprocessor.preprocessor - Document dd048b8e5bcb7de1be5bd3937f15442f is 14232 characters long after preprocessing, where the maximum length should be 10000. Something might be wrong with the splitting, check the document affected to prevent issues at query time.”

6-
from haystack.nodes import FARMReader

reader = FARMReader(model_name_or_path="deepset/roberta-base-squad2", use_gpu=True)

INFO - haystack.modeling.utils - Using devices: CPU - Number of GPUs: 0
INFO - haystack.modeling.utils - Using devices: CPU - Number of GPUs: 0
INFO - haystack.modeling.model.language_model - * LOADING MODEL: 'deepset/roberta-base-squad2' (Roberta)
/Users/phdykd/miniconda3/envs/Haystack/lib/python3.11/site-packages/torch/_utils.py:776: UserWarning: TypedStorage is deprecated. It will be removed in the future and UntypedStorage will be the only storage class. This should only matter to you if you are using storages directly. To access UntypedStorage directly, use tensor.untyped_storage() instead of tensor.storage()
return self.fget.get(instance, owner)()
INFO - haystack.modeling.model.language_model - Auto-detected model language: english
INFO - haystack.modeling.model.language_model - Loaded 'deepset/roberta-base-squad2' (Roberta model) from model hub.
INFO - haystack.modeling.utils - Using devices: CPU - Number of GPUs: 0

Why is this still not able to catch my GPUs from M2 Max chip MacOS?

7- I had to install Docker Desktop. Your instructions do not mention that I needed to install Docker.. Please update your instructions.

8- Most importantly, how can I import my own pdf, csv, text data and make query targeting my own documents?

Thanks for your time.
Regards

[julian-risch]

Nice, the latest release of Haystack is installed now on your M2 Mac. 👍 Whether elasticsearch is running properly you can check with curl localhost:9200/_cat/health in the command line. username and password aren't required for elasticsearch version 7.9 and I don't think that changed with version 7.17. If elasticsearch-7.9.2 doesn't work for you, I would recommend you start with the InMemoryDocumentStore, which is the best choice for the first experiments: https://docs.haystack.deepset.ai/docs/document_store#initialization

An alternative is to install and use docker. Once you have installed docker, you can also find the docker commands for elasticsearch, opensearch, etc. in the documentation: https://docs.haystack.deepset.ai/docs/document_store#initialization

I will close this issue because Haystack is correctly installed now. 👍 If the problem with connecting to elasticsearch remains and you assume it's a problem in Haystack, please open a separate issue. Thank you!

[phdykd]

I just updated my comments just after your comment.it will be great if you may address comment #1 and #8 that I posted above.
Thanks