nltk.stem.cistem.Cistem

class documentation

class Cistem(StemmerI): (source)

Constructor: Cistem(case_insensitive)

CISTEM Stemmer for German

This is the official Python implementation of the CISTEM stemmer. It is based on the paper Leonie Weissweiler, Alexander Fraser (2017). Developing a Stemmer for German Based on a Comparative Analysis of Publicly Available Stemmers. In Proceedings of the German Society for Computational Linguistics and Language Technology (GSCL) which can be read here: http://www.cis.lmu.de/~weissweiler/cistem/

In the paper, we conducted an analysis of publicly available stemmers, developed two gold standards for German stemming and evaluated the stemmers based on the two gold standards. We then proposed the stemmer implemented here and show that it achieves slightly better f-measure than the other stemmers and is thrice as fast as the Snowball stemmer for German while being about as fast as most other stemmers.

case_insensitive is a a boolean specifying if case-insensitive stemming should be used. Case insensitivity improves performance only if words in the text may be incorrectly upper case. For all-lowercase and correctly cased text, best performance is achieved by setting case_insensitive for false.

Parameters
case_insensitive	if True, the stemming is case insensitive. False by default.

Static Method	`replace_back`	Undocumented
Static Method	`replace_to`	Undocumented
Method	`__init__`	Undocumented
Method	`segment`	This method works very similarly to stem ('cistem.stem'). The difference is that in addition to returning the stem, it also returns the rest that was removed at the end. To be able to return the stem unchanged so the stem and the rest can be concatenated to form the original word, all subsitutions that altered the stem in any other way than by removing letters at the end were left out.
Method	`stem`	This method takes the word to be stemmed and returns the stemmed word.
Class Variable	`repl_xx`	Undocumented
Class Variable	`repl_xx_back`	Undocumented
Class Variable	`strip_emr`	Undocumented
Class Variable	`strip_esn`	Undocumented
Class Variable	`strip_ge`	Undocumented
Class Variable	`strip_nd`	Undocumented
Class Variable	`strip_t`	Undocumented
Instance Variable	`_case_insensitive`	Undocumented

@staticmethod
def replace_back(word): (source) ¶

Undocumented

@staticmethod
def replace_to(word): (source) ¶

Undocumented

def __init__(self, case_insensitive=False): (source) ¶

Undocumented

def segment(self, word): (source) ¶

This method works very similarly to stem ('cistem.stem'). The difference is that in addition to returning the stem, it also returns the rest that was removed at the end. To be able to return the stem unchanged so the stem and the rest can be concatenated to form the original word, all subsitutions that altered the stem in any other way than by removing letters at the end were left out.

>>> from nltk.stem.cistem import Cistem
>>> stemmer = Cistem()
>>> s1 = "Speicherbehältern"
>>> print("('" + stemmer.segment(s1)[0] + "', '" + stemmer.segment(s1)[1] + "')")
('speicherbehält', 'ern')
>>> s2 = "Grenzpostens"
>>> stemmer.segment(s2)
('grenzpost', 'ens')
>>> s3 = "Ausgefeiltere"
>>> stemmer.segment(s3)
('ausgefeilt', 'ere')
>>> stemmer = Cistem(True)
>>> print("('" + stemmer.segment(s1)[0] + "', '" + stemmer.segment(s1)[1] + "')")
('speicherbehäl', 'tern')
>>> stemmer.segment(s2)
('grenzpo', 'stens')
>>> stemmer.segment(s3)
('ausgefeil', 'tere')

Parameters
word:unicode	the word that is to be stemmed
Returns
unicode	the removed suffix

def stem(self, word): (source) ¶

overrides nltk.stem.api.StemmerI.stem

This method takes the word to be stemmed and returns the stemmed word.

>>> from nltk.stem.cistem import Cistem
>>> stemmer = Cistem()
>>> s1 = "Speicherbehältern"
>>> stemmer.stem(s1)
'speicherbehalt'
>>> s2 = "Grenzpostens"
>>> stemmer.stem(s2)
'grenzpost'
>>> s3 = "Ausgefeiltere"
>>> stemmer.stem(s3)
'ausgefeilt'
>>> stemmer = Cistem(True)
>>> stemmer.stem(s1)
'speicherbehal'
>>> stemmer.stem(s2)
'grenzpo'
>>> stemmer.stem(s3)
'ausgefeil'

Parameters
word:unicode	the word that is to be stemmed
Returns
unicode	the stemmed word

repl_xx = (source) ¶

Undocumented

repl_xx_back = (source) ¶

Undocumented

strip_emr = (source) ¶

Undocumented

strip_esn = (source) ¶

Undocumented

strip_ge = (source) ¶

Undocumented

strip_nd = (source) ¶

Undocumented

strip_t = (source) ¶

Undocumented

_case_insensitive = (source) ¶

Undocumented