«

nltk.tag.tnt.TnT

class documentation

class TnT(TaggerI): (source)

Constructor: TnT(unk, Trained, N, C)

View In Hierarchy

TnT - Statistical POS tagger

IMPORTANT NOTES:

  • DOES NOT AUTOMATICALLY DEAL WITH UNSEEN WORDS
    • It is possible to provide an untrained POS tagger to create tags for unknown words, see __init__ function
  • SHOULD BE USED WITH SENTENCE-DELIMITED INPUT
    • Due to the nature of this tagger, it works best when trained over sentence delimited input.
    • However it still produces good results if the training data and testing data are separated on all punctuation eg: [,.?!]
    • Input for training is expected to be a list of sentences where each sentence is a list of (word, tag) tuples
    • Input for tag function is a single sentence Input for tagdata function is a list of sentences Output is of a similar form
  • Function provided to process text that is unsegmented
    • Please see basic_sent_chop()

TnT uses a second order Markov model to produce tags for a sequence of input, specifically:

argmax [Proj(P(t_i|t_i-1,t_i-2)P(w_i|t_i))] P(t_T+1 | t_T)

IE: the maximum projection of a set of probabilities

The set of possible tags for a given word is derived from the training data. It is the set of all tags that exact word has been assigned.

To speed up and get more precision, we can use log addition to instead multiplication, specifically:

argmax [Sigma(log(P(t_i|t_i-1,t_i-2))+log(P(w_i|t_i)))] +
log(P(t_T+1|t_T))

The probability of a tag for a given word is the linear interpolation of 3 markov models; a zero-order, first-order, and a second order model.

P(t_i| t_i-1, t_i-2) = l1*P(t_i) + l2*P(t_i| t_i-1) +
l3*P(t_i| t_i-1, t_i-2)

A beam search is used to limit the memory usage of the algorithm. The degree of the beam can be changed using N in the initialization. N represents the maximum number of possible solutions to maintain while tagging.

It is possible to differentiate the tags which are assigned to capitalized words. However this does not result in a significant gain in the accuracy of the results.

Method __init__ Construct a TnT statistical tagger. Tagger must be trained before being used to tag input.
Method tag Tags a single sentence
Method tagdata Tags each sentence in a list of sentences
Method train Uses a set of tagged data to train the tagger. If an unknown word tagger is specified, it is trained on the same data.
Instance Variable known Undocumented
Instance Variable unknown Undocumented
Method _compute_lambda creates lambda values based upon training data
Method _safe_div Safe floating point division function, does not allow division by 0 returns -1 if the denominator is 0
Method _tagword :param sent : List of words remaining in the sentence :type sent : [word,] :param current_states : List of possible tag combinations for
Instance Variable _bi Undocumented
Instance Variable _C Undocumented
Instance Variable _eos Undocumented
Instance Variable _l1 Undocumented
Instance Variable _l2 Undocumented
Instance Variable _l3 Undocumented
Instance Variable _N Undocumented
Instance Variable _T Undocumented
Instance Variable _tri Undocumented
Instance Variable _uni Undocumented
Instance Variable _unk Undocumented
Instance Variable _wd Undocumented

Inherited from TaggerI:

Method evaluate Score the accuracy of the tagger against the gold standard. Strip the tags from the gold standard text, retag it using the tagger, then compute the accuracy score.
Method tag_sents Apply self.tag() to each element of sentences. I.e.:
Method _check_params Undocumented
def __init__(self, unk=None, Trained=False, N=1000, C=False): (source)

Construct a TnT statistical tagger. Tagger must be trained before being used to tag input.

:type unk:(TaggerI) :param Trained: Indication that the POS tagger is trained or not :type Trained: boolean :param N: Beam search degree (see above) :type N:(int) :param C: Capitalization flag :type C: boolean

Initializer, creates frequency distributions to be used for tagging

_lx values represent the portion of the tri/bi/uni taggers to be used to calculate the probability

N value is the number of possible solutions to maintain while tagging. A good value for this is 1000

C is a boolean value which specifies to use or not use the Capitalization of the word as additional information for tagging. NOTE: using capitalization may not increase the accuracy of the tagger

Parameters
unkinstance of a POS tagger, conforms to TaggerI
TrainedUndocumented
NUndocumented
CUndocumented
def tag(self, data): (source)
overrides nltk.tag.api.TaggerI.tag

Tags a single sentence

Calls recursive function '_tagword' to produce a list of tags

Associates the sequence of returned tags with the correct words in the input sequence

returns a list of (word, tag) tuples

Parameters
data:[string,]list of words
Returns
[(word, tag),]
def tagdata(self, data): (source)

Tags each sentence in a list of sentences

:param data:list of list of words :type data: [[string,],] :return: list of list of (word, tag) tuples

Invokes tag(sent) function for each sentence compiles the results into a list of tagged sentences each tagged sentence is a list of (word, tag) tuples

def train(self, data): (source)

Uses a set of tagged data to train the tagger. If an unknown word tagger is specified, it is trained on the same data.

Parameters
data:tuple(str)List of lists of (word, tag) tuples
known: int = (source)

Undocumented

unknown: int = (source)

Undocumented

def _compute_lambda(self): (source)

creates lambda values based upon training data

NOTE: no need to explicitly reference C, it is contained within the tag variable :: tag == (tag,C)

for each tag trigram (t1, t2, t3) depending on the maximum value of - f(t1,t2,t3)-1 / f(t1,t2)-1 - f(t2,t3)-1 / f(t2)-1 - f(t3)-1 / N-1

increment l3,l2, or l1 by f(t1,t2,t3)

ISSUES -- Resolutions: if 2 values are equal, increment both lambda values by (f(t1,t2,t3) / 2)

def _safe_div(self, v1, v2): (source)

Safe floating point division function, does not allow division by 0 returns -1 if the denominator is 0

def _tagword(self, sent, current_states): (source)

:param sent : List of words remaining in the sentence :type sent : [word,] :param current_states : List of possible tag combinations for

the sentence so far, and the log probability associated with each tag combination

:type current_states : [([tag, ], logprob), ]

Tags the first word in the sentence and recursively tags the reminder of sentence

Uses formula specified above to calculate the probability of a particular tag

_bi = (source)

Undocumented

_C = (source)

Undocumented

_eos = (source)

Undocumented

_l1 = (source)

Undocumented

_l2 = (source)

Undocumented

_l3 = (source)

Undocumented

_N = (source)

Undocumented

_T = (source)

Undocumented

_tri = (source)

Undocumented

_uni = (source)

Undocumented

_unk = (source)

Undocumented

_wd = (source)

Undocumented