Error when importing BERTopic

[Besteverandever]

Can someone please let me know how can i get rid of this error. I tried installing torch==1.9.0 and torch==1.8.0 but none of them work.

ImportError: cannot import name 'SAVE_STATE_WARNING' from 'torch.optim.lr_scheduler' (C:\Users\user\anaconda3\lib\site-
packages\torch\optim\lr_scheduler.py)

[MaartenGr]

This often happens when the transformers module does not properly match the torch version. I would advise starting from a completely fresh environment and installing BERTopic there.

Hopefully, this fixes your issue!

[Besteverandever]

Many thanks for your reply. I am able to get it working and try to perform topic modelling for the free open text from survey data. Can you please tell me which embedding model is the best to extract data about covid related experiences? I have noticed that the topic model does not remove the stop words automatically and I have done it manually. Is it so or am I missing something?

[MaartenGr]

There are a number of great models to be found here. If the text is in English and if you have a good enough GPU, I would advise going with all-mpnet-base-v2. However, the default already gives you good performance and is quite fast too!

With respect to the stop words, that depends on the number of topics that are generated. Typically, it will remove stopwords if many topics are generated but you can remove them using a custom CountVectorizer in BERTopic. There is a bit about that in the documentation here.

[Besteverandever]

Thank you for your suggestion, it is working for me now. Though I do not get very meaningful topics at the moment even after trying bigrams (e.g. like_school_kid_game, watch_name_kid_tv). I have also tried keybert for phrase extraction but all I could get ('home relationships time', 0.4998), ('home parents better', 0.5123). Is there any way of doing a seeded topic modeling or some kind of fine-tuning that could generate more meaningful insight into the free text data? Any ideas/suggestions are greatly appreciated.

[MaartenGr]

Could you tell me a bit more about the data you are using? How many documents do you have? Also, could you share the code for running BERTopic? Perhaps there are some parameters that could be optimized. Lastly, which version of BERTopic are you using?

[Besteverandever]

Thank you for your reply, if it is alright with you can I have your personal id to answer your questions?

[MaartenGr]

Sure, you can contact me through LinkedIn to continue this discussion.

[MaartenGr]

Seeing as the discussion continued through LinkedIn, I'll close this issue for now.

[emeanacj14]

@MaartenGr pls could you advise how to remove this error while trying to import BERTopic in GCP notebook AI instance. my pls see below my code snippet and error msg:

!pip install bertopic==0.9.4
(.................................................................
Successfully installed bertopic-0.9.4 cython-0.29.26 filelock-3.4.2 hdbscan-0.8.27 huggingface-hub-0.2.1 nltk-3.6.7 numpy-1.20.3 plotly-5.5.0 pynndescent-0.5.5 pyyaml-5.4.1 regex-2021.11.10 sacremoses-0.0.46 sentence-transformers-2.1.0 sentencepiece-0.1.96 tokenizers-0.10.3 transformers-4.15.0 umap-learn-0.5.2)

from bertopic import BERTopic

---

ValueError Traceback (most recent call last)
/tmp/ipykernel_3264/2806191270.py in
----> 1 from bertopic import BERTopic

/opt/conda/lib/python3.7/site-packages/bertopic/init.py in
----> 1 from bertopic._bertopic import BERTopic
2
3 version = "0.9.4"
4
5 all = [

/opt/conda/lib/python3.7/site-packages/bertopic/_bertopic.py in
19
20 # Models
---> 21 import hdbscan
22 from umap import UMAP
23 from sklearn.feature_extraction.text import CountVectorizer

/opt/conda/lib/python3.7/site-packages/hdbscan/init.py in
----> 1 from .hdbscan_ import HDBSCAN, hdbscan
2 from .robust_single_linkage_ import RobustSingleLinkage, robust_single_linkage
3 from .validity import validity_index
4 from .prediction import (approximate_predict,
5 membership_vector,

/opt/conda/lib/python3.7/site-packages/hdbscan/hdbscan_.py in
19 from scipy.sparse import csgraph
20
---> 21 from ._hdbscan_linkage import (single_linkage,
22 mst_linkage_core,
23 mst_linkage_core_vector,

hdbscan/_hdbscan_linkage.pyx in init hdbscan._hdbscan_linkage()

ValueError: numpy.ndarray size changed, may indicate binary incompatibility. Expected 88 from C header, got 80 from PyObject

[emeanacj14]

Successfully built hdbscan sentence-transformers umap-learn pynndescent
Installing collected packages: numpy, regex, pyyaml, filelock, tokenizers, sacremoses, huggingface-hub, transformers, sentencepiece, pynndescent, nltk, cython, umap-learn, sentence-transformers, plotly, hdbscan, bertopic
Attempting uninstall: numpy
Found existing installation: numpy 1.19.5
Uninstalling numpy-1.19.5:
Successfully uninstalled numpy-1.19.5
Attempting uninstall: pyyaml
Found existing installation: PyYAML 6.0
Uninstalling PyYAML-6.0:
Successfully uninstalled PyYAML-6.0
ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts.
thinc 8.0.13 requires typing-extensions<4.0.0.0,>=3.7.4.1; python_version < "3.8", but you have typing-extensions 4.0.1 which is incompatible.
spacy 3.2.1 requires typing-extensions<4.0.0.0,>=3.7.4; python_version < "3.8", but you have typing-extensions 4.0.1 which is incompatible.
Successfully installed bertopic-0.9.4 cython-0.29.26 filelock-3.4.2 hdbscan-0.8.27 huggingface-hub-0.2.1 nltk-3.6.7 numpy-1.20.3 plotly-5.5.0 pynndescent-0.5.5 pyyaml-5.4.1 regex-2021.11.10 sacremoses-0.0.46 sentence-transformers-2.1.0 sentencepiece-0.1.96 tokenizers-0.10.3 transformers-4.15.0 umap-learn-0.5.2