Unable to load model due to torchtext dependency

[RaahimSiddiqi]

Trying the latest version of SinaTools and am facing some issues when loading the NER model. I installed it using the pip command, and downloaded the NER model using the "download_files -f NER" command. I am attempting to run the sample script:

from sinatools.ner.entity_extractor import extract

extract('ذهب محمد إلى جامعة بيرزيت')

and am facing the following error:

ModuleNotFoundError: No module named 'torchtext'

Despite the latest release saying torchtext dependency was removed. Upon downloading torchtext and running this code again, I get the following error:

OSError: [WinError 127] The specified procedure could not be found

I have tried creating a fresh environment as well to no effect.

Environment:
Python 3.11.11
Windows 11
Dependencies: Installed using "pip install sinatools"

[RaahimSiddiqi]

After some code and release history inspection, I believe that the model available for download through the DataDownload folder is outdated. Is the new model trained on the newer version of torch available for download? The models on huggingface haven't been updated either.

[RaahimSiddiqi]

After reverting to sinatools 0.1.36, the model is now working. This confirms my belief the NER model available online is the old one, trained using torchtext.

[codex-corp]

The Core Issue (Likely Version Mismatch/Incompatibility):**

*   When a complex object (like a model or a data structure containing PyTorch/TorchText types) is pickled, the pickle file stores references to the *class definitions* of the objects it contains.
*   If the environment where you *load* the pickle doesn't have the exact same classes available (or if there have been significant internal changes in those libraries), the unpickling process fails because it can't reconstruct the original object. It's essentially looking for `torchtext.SomeInternalClass` and can't find it, even though the top-level `torchtext` module *is* installed.
*   This is a common problem when using pre-trained models or cached data structures that were created with different (often older) versions of libraries.

Connecting to SinaTools==0.1.41 and PyTorch Versions:

• As we established earlier, SinaTools==0.1.41 was likely developed and tested with older versions of PyTorch/TorchText (like the 1.13.x series).
• You are now trying to run it with much newer versions: torch==2.5.1 and torchtext==0.20.1.
• The pre-trained NER model (or its associated data files) shipped with SinaTools was probably pickled using torchtext 0.14.1 (or similar). When this model/data is loaded by the sinatools.ner module in an environment with torchtext 0.20.1, the internal structure or class names might have changed enough to break the unpickling process, leading to this confusing ModuleNotFoundError.

Conclusion:

The root cause is a version incompatibility between the SinaTools==0.1.41 package (and its pre-trained models/data) and the newer torchtext==0.20.1 you installed. The package expects an older internal structure from torchtext that no longer exists.

Solution:

You need to revert to the older, compatible versions of PyTorch and TorchText that SinaTools==0.1.41 was designed for. As we determined before, torch==1.13.1 and torchtext==0.14.1 are the correct pair.

Please update your Dockerfile's PyTorch installation section as follows:

Replace this section:

# ---- CPU-only PyTorch (PINNED) ----
# Use the official CPU wheel index so we don't pull CUDA.
# Torch 2.5.1 <-> TorchVision 0.20.1 are the matching pair.
RUN python -m pip install --no-cache-dir \
    --index-url https://download.pytorch.org/whl/cpu   \
    torch==2.5.1 torchvision==0.20.1

With this corrected section:

# ---- CPU-only PyTorch (PINNED for SinaTools 0.1.41 compatibility) ----
# Use the official CPU wheel index so we don't pull CUDA.
# Torch 1.13.1 <-> TorchText 0.14.1 are the compatible pair for SinaTools 0.1.41.
RUN python -m pip install --no-cache-dir \
    --index-url https://download.pytorch.org/whl/cpu \
    torch==1.13.1 torchtext==0.14.1

(Note: I removed torchvision as it's not strictly required by the parts of SinaTools you're using in main.py, but you can add torchvision==0.14.1 if needed for other parts or for completeness.)

After making this change, rebuild your image (docker build -t sinatools-test .) and run it again. This should resolve the ModuleNotFoundError by providing the exact torchtext version the pickled NER data expects.

[codex-corp]

Investigation Notes (2025-09-18)

• Verified the shipped NER model checkpoint at models/sinatools/Wj27012000.tar/. The tag_vocab.pkl inside the archive is a pickle of torchtext.vocab.vocab.Vocab objects.
• When running against PyTorch 2.x + modern torchtext, the import path torchtext.vocab.vocab.Vocab no longer exists, causing unpickling to fail before the state dict is loaded.
• The model weights themselves remain compatible with newer PyTorch releases; the blocker is the pickled vocab class.

Proposed Fix

1. Use an environment that still ships torchtext.vocab.vocab.Vocab (e.g. PyTorch 1.13.1 + torchtext 0.15.2) to load the vocabulary and immediately re-serialize it using SinaTools' native Vocab class.
2. Update SinaTools' loader (sinatools/ner/__init__.py) to prefer the SinaTools Vocab when available, ensuring future checkpoints are framework-independent.
3. After conversion, PyTorch 2.x + newer torchtext can unpickle the vocab, allowing the same model checkpoint to load without downgrading the runtime stack.

Immediate Plan

• Extend the loader to gracefully handle checkpoints without torchtext by preferring the internal Vocab class and falling back only when necessary.
• Provide a conversion utility that re-saves tag_vocab.pkl using the internal Vocab, keeping a backup of the original pickle.

[TymaaHammouda]

A new NER model has been released, trained on the Wojood and Konooz datasets. This should resolve the torch issue.