Cannot import BERTopic: SAVE_STATE_WARNING

[ziqizhang]

Hi

I am having the same issue described here but I cannot solve the problem: #341

I can confirm that I have started from a fresh environment. I created a conda environment from scratch, using Python 3.6. I ran all the installation commands:

pip install bertopic
pip install bertopic[flair]
pip install bertopic[gensim]
pip install bertopic[spacy]
pip install bertopic[use]

And no errors were reported. However, when trying to import bertopic, I got the error below. I know that this is often due to incompatible torch and transformers being installed, but I already started from a fresh environment. My torch=1.10.1, transformers=3.5.1. I don't know where to look for information if these two are compatible and if not, what are the compatible versions I have to install?

Thanks

Traceback (most recent call last):
  File "/home/zz/Work/bertopic/python/main.py", line 1, in <module>
    from bertopic import BERTopic
  File "/home/zz/Programs/anaconda3/envs/bertopic/lib/python3.6/site-packages/bertopic/__init__.py", line 1, in <module>
    from bertopic._bertopic import BERTopic
  File "/home/zz/Programs/anaconda3/envs/bertopic/lib/python3.6/site-packages/bertopic/_bertopic.py", line 16, in <module>
    from sentence_transformers import SentenceTransformer
  File "/home/zz/Programs/anaconda3/envs/bertopic/lib/python3.6/site-packages/sentence_transformers/__init__.py", line 3, in <module>
    from .datasets import SentencesDataset, ParallelSentencesDataset
  File "/home/zz/Programs/anaconda3/envs/bertopic/lib/python3.6/site-packages/sentence_transformers/datasets/__init__.py", line 3, in <module>
    from .ParallelSentencesDataset import ParallelSentencesDataset
  File "/home/zz/Programs/anaconda3/envs/bertopic/lib/python3.6/site-packages/sentence_transformers/datasets/ParallelSentencesDataset.py", line 4, in <module>
    from .. import SentenceTransformer
  File "/home/zz/Programs/anaconda3/envs/bertopic/lib/python3.6/site-packages/sentence_transformers/SentenceTransformer.py", line 11, in <module>
    import transformers
  File "/home/zz/Programs/anaconda3/envs/bertopic/lib/python3.6/site-packages/transformers/__init__.py", line 626, in <module>
    from .trainer import Trainer
  File "/home/zz/Programs/anaconda3/envs/bertopic/lib/python3.6/site-packages/transformers/trainer.py", line 69, in <module>
    from .trainer_pt_utils import (
  File "/home/zz/Programs/anaconda3/envs/bertopic/lib/python3.6/site-packages/transformers/trainer_pt_utils.py", line 40, in <module>
    from torch.optim.lr_scheduler import SAVE_STATE_WARNING
ImportError: cannot import name 'SAVE_STATE_WARNING'

Process finished with exit code 1

[ziqizhang]

I found this post (https://stackoverflow.com/questions/69473082/importerror-cannot-import-name-save-state-warning-from-torch-optim-lr-schedu) that says I need to upgrade transformer to a more recent version because SAVE_STATE_WARNING was removed from pytorch in late 2020 (huggingface/transformers#8979)

After doing so, that error disappeared. But pip reported a new error:
flair 0.7 requires transformers<=3.5.1,>=3.5.0, but you have transformers 4.16.2 which is incompatible.

I dont know if this will affect it?

[MaartenGr]

Typically, this should not affect BERTopic in general. However, it might be that Flair will stop working since you have incompatible dependencies. I would advise not installing all other back-ends (e..g, Gensim, Spacy) but only installing Flair as that will most likely prevent problems with conflicting dependencies.

[ziqizhang]

Thanks!