Tokens in BOW and Topic models

[bzz]

While trying to use a Topics model of repositories, I got confused with different number of tokens in Topics and BOW models that are published: topics has 2,015,336 tokens, while bow has 999,424.

import logging

from sourced.ml.models import BOW, Topics
from modelforge.backends import create_backend
from modelforge.index import GitIndex

logging.basicConfig(level=logging.INFO)

git_index = GitIndex(
    log_level=logging.INFO,
    index_repo="https://github.com/src-d/models",
    cache="~/.source{d}/models")
backend = create_backend(git_index=git_index)

topics = Topics(log_level=logging.INFO).load(backend=backend)
bow = BOW(log_level=logging.INFO).load(backend=backend)

assert topics.matrix.shape[1] == bow.matrix.shape[
    1], "Topics has %s tokens, BOW has %s" % (
        topics.matrix.shape[1], bow.matrix.shape[1])

Could anybody please help me understand, whether it should be the same number in order for these models to be used together? Or is BOW model is not per-repository?

Thanks in advance!

[bzz]

From https://github.com/src-d/models/blob/master/bow/1e3da42a-28b6-4b33-94a2-a5671f4102f4.md - model seems to be BOW per-repository for 112k repos.

[vmarkovtsev]

The topics model and the embeddings model are independent. Both define vectors for tokens but their meaning is different. Also, the current topics model is ancient and cries and begs to be updated. One day we will train it on PGA.

[bzz]

The topics model and the embeddings model are independent

I understand, but is BOW = embeddings in that context?

Was asking about BOW as, it seems that "weighted vector of tokens" representation of the repository is the input to the Topic model, so in order for those to be used together, they have to have same dimensionality.

From what I can see, it seems that right now our published BOW model can not be used together with published Topics mode (as it was once in tmsc before)

[vmarkovtsev]

@bzz BOW is https://en.wikipedia.org/wiki/Bag-of-words_model

It is not weighted vector of tokens, it is weighted bag of vectors. And nope, this is not the model of the topic modeling - it has nothing to do with embeddings.

Topic modeling is described in e.g. https://bigartm.readthedocs.io/en/stable/tutorials/python_tutorial.html
You multiply the matrix from the model (topics by words, sparse) by the vector from the repository (list of weights per word, very sparse).

[vmarkovtsev]

So while each word is represented by a vector (weights of topics), it is sparse and cannot work as a regular embedding. It can describe the result but cannot generate it. I mean of course you can do whatever but the quality of the solution based on such "embeddings" will be poor.

[bzz]

this is not the model of the topic modeling - it has nothing to do with embeddings

There is some kind of confusion: I never used word embedding in the question, so was very confused with your answer using it and just asked for clarifications.

Thank you for kind explanation: "weighted vector of tokens" should be "bag of weighted tokens" in my message above.

But somehow I feel that the terminology discussion still does not help to finalize the answer the original question on ability of using our published BoW model to convert a repository to a vector, and then using this vector to get "topics" from the published Topic model :/

[vmarkovtsev]

Token embeddings cannot turn a repo into a vector.

The topic model can turn a repo into a vector, but that vector is limited to the defined topics with all the implications of the topic metric space. That is, two similar repositories always have similar topic vectors, but two similar topic vectors do not always correspond to similar repositories.

[bzz]

Than you for kind explanations!