Custom Stopwords

[amoschoomy]

Is there any way to customise the stopwords rather than using NLTKs list of stopwords? I would like to extend the list of stopwords or replace it with another list of my own. Thanks

[TimSchopf]

No, currently there is no option to define a custom stopwords list. You can only define NLTKs top words list or set stop_words=None to remove no stopwords at all.

[amoschoomy]

Thanks for the feedback, I've decided not to remove any stopwords at all after I'm having the same issues as #8.
But I run into this error, when I put stop_words=None. It's a straightforward fix though by removing the validation for the typechecking.

    152                                                    spacy_pipeline=self.spacy_pipeline,
    153                                                    pos_pattern=self.pos_pattern,
--> 154                                                    lowercase=self.lowercase, workers=self.workers)
    155 
    156         # remove keyphrases that have more than 8 words, as they are probably no real keyphrases

/opt/conda/lib/python3.7/site-packages/keyphrase_vectorizers/keyphrase_vectorizer_mixin.py in _get_pos_keyphrases(self, document_list, stop_words, spacy_pipeline, pos_pattern, lowercase, workers)
    231         if not isinstance(stop_words, str):
    232             raise ValueError(
--> 233                 "'stop_words' parameter needs to be a string. E.g. 'english'"
    234             )
    235 

ValueError: 'stop_words' parameter needs to be a string. E.g. 'english'

[lill-Juvelen]

I get the same error, cannot use stop_words=None

[amoschoomy]

@lill-Juvelen you can fork the repository and remove the validation as I've mentioned above and pip install the forked repository while waiting for a fix from the author

[TimSchopf]

I get the same error, cannot use stop_words=None

The v0.0.8 release allows to set stop_words=None and should fix this issue. However, if you notice any other bugs that may be a result of this fix, please feel free to report them.

[TimSchopf]

With the v0.0.10 release I added the option to define custom stopwords. Now you can pass the stop_words parameter a list of custom stop words that you want to have removed.

[gboyega1]

I wonder if I'm using the stop_words parameter correctly as it seems not to exclude stop words from a custom list that I've provided. Below is an example where I require it to remove msc

no custom stop word list included

vectorizer = KeyphraseCountVectorizer()
kw_model.extract_keywords(strip_html(course[2]), vectorizer=vectorizer, top_n = 15, use_mmr = True, diversity = 0.45)

output

[('hrm msc students', 0.5963), ('human resources', 0.5466), ('organisational development', 0.505), ('student experience', 0.4352), ('business school', 0.4336), ('people profession', 0.4273), ('pwc staff', 0.416), ('london offices', 0.4049), ('research leaders', 0.3931), ('professional stream skills workshop satisfy requirements', 0.3907), ('quality education', 0.3792), ('cipd accreditation', 0.3522), ('dissertation', 0.3428), ('relevant programmes', 0.3143), ('edge practice', 0.2627)]

including a custom stop word list to discard 'msc'

vectorizer = KeyphraseCountVectorizer(stop_words = stpwrds)
kw_model.extract_keywords(strip_html(course[2]), vectorizer=vectorizer, top_n = 15, use_mmr = True, diversity = 0.45)

output produces same keyphrases all having identical importance. Also 'msc is not discarded'

[('hrm msc students', 0.5963), ('human resources', 0.5466), ('organisational development', 0.505), ('student experience', 0.4352), ('business school', 0.4336), ('people profession', 0.4273), ('pwc staff', 0.416), ('london offices', 0.4049), ('research leaders', 0.3931), ('professional stream skills workshop satisfy requirements', 0.3907), ('quality education', 0.3792), ('cipd accreditation', 0.3522), ('dissertation', 0.3428), ('relevant programmes', 0.3143), ('edge practice', 0.2627)]

below is the list of the custom stop words

print(vectzr.get_params())

{'binary': False, 'custom_pos_tagger': None, 'dtype': <class 'numpy.int64'>, 'lowercase': True, 'max_df': None, 'min_df': None, 'pos_pattern': '<J.*>*<N.*>+', 'spacy_exclude': None, 'spacy_pipeline': 'en_core_web_sm', 'stop_words': ['i', 'me', 'my', 'myself', 'we', 'our', 'ours', 'ourselves', 'you', "you're", "you've", "you'll", "you'd", 'your', 'yours', 'yourself', 'yourselves', 'he', 'him', 'his', 'himself', 'she', "she's", 'her', 'hers', 'herself', 'it', "it's", 'its', 'itself', 'they', 'them', 'their', 'theirs', 'themselves', 'what', 'which', 'who', 'whom', 'this', 'that', "that'll", 'these', 'those', 'am', 'is', 'are', 'was', 'were', 'be', 'been', 'being', 'have', 'has', 'had', 'having', 'do', 'does', 'did', 'doing', 'a', 'an', 'the', 'and', 'but', 'if', 'or', 'because', 'as', 'until', 'while', 'of', 'at', 'by', 'for', 'with', 'about', 'against', 'between', 'into', 'through', 'during', 'before', 'after', 'above', 'below', 'to', 'from', 'up', 'down', 'in', 'out', 'on', 'off', 'over', 'under', 'again', 'further', 'then', 'once', 'here', 'there', 'when', 'where', 'why', 'how', 'all', 'any', 'both', 'each', 'few', 'more', 'most', 'other', 'some', 'such', 'no', 'nor', 'not', 'only', 'own', 'same', 'so', 'than', 'too', 'very', 's', 't', 'can', 'will', 'just', 'don', "don't", 'should', "should've", 'now', 'd', 'll', 'm', 'o', 're', 've', 'y', 'ain', 'aren', "aren't", 'couldn', "couldn't", 'didn', "didn't", 'doesn', "doesn't", 'hadn', "hadn't", 'hasn', "hasn't", 'haven', "haven't", 'isn', "isn't", 'ma', 'mightn', "mightn't", 'mustn', "mustn't", 'needn', "needn't", 'shan', "shan't", 'shouldn', "shouldn't", 'wasn', "wasn't", 'weren', "weren't", 'won', "won't", 'wouldn', "wouldn't", 'msc', 'ma', 'pgdip', 'pgcert', 'msw', 'mphil', 'mph', 'llm', 'master', 'pgd', 'pgc', 'mres', 'eng', 'pgce', 'pgprofdev', 'med', 'mba', 'mscr', 'pgc', 'mbbch', 'mlitt', 'mmed'], 'workers': 1}

[phuclh]

@TimSchopf Seems the setting stop_words: None doesn't work. I tested with 'You have to cook anything but no pasta. Pasta is from Italy. Where does the pasta come from?' but it only printed anything, italy, pasta