nltk.translate.chrf

module documentation

(source)

ChrF score implementation

Function	`chrf_precision_recall_fscore_support`	This function computes the precision, recall and fscore from the ngram overlaps. It returns the `support` which is the true positive score.
Function	`corpus_chrf`	Calculates the corpus level CHRF (Character n-gram F-score), it is the macro-averaged value of the sentence/segment level CHRF score.
Function	`sentence_chrf`	Maja Popovic. 2015. CHRF: Character n-gram F-score for Automatic MT Evaluation. In Proceedings of the 10th Workshop on Machine Translation. http://www.statmt.org/wmt15/pdf/WMT49.pdf
Function	`_preprocess`	Undocumented

def chrf_precision_recall_fscore_support(reference, hypothesis, n, beta=3.0, epsilon=1e-16): (source) ¶

This function computes the precision, recall and fscore from the ngram overlaps. It returns the support which is the true positive score.

By underspecifying the input type, the function will be agnostic as to how it computes the ngrams and simply take the whichever element in the list; it could be either token or character.

Parameters
reference:list	The reference sentence.
hypothesis:list	The hypothesis sentence.
n:int	Extract up to the n-th order ngrams
beta:float	The parameter to assign more importance to recall over precision.
epsilon:float	The fallback value if the hypothesis or reference is empty.
Returns
tuple(float)	Returns the precision, recall and f-score and support (true positive).

def corpus_chrf(references, hypotheses, min_len=1, max_len=6, beta=3.0, ignore_whitespace=True): (source) ¶

Calculates the corpus level CHRF (Character n-gram F-score), it is the macro-averaged value of the sentence/segment level CHRF score.

This implementation of CHRF only supports a single reference at the moment.

>>> ref1 = str('It is a guide to action that ensures that the military '
...            'will forever heed Party commands').split()
>>> ref2 = str('It is the guiding principle which guarantees the military '
...            'forces always being under the command of the Party').split()
>>>
>>> hyp1 = str('It is a guide to action which ensures that the military '
...            'always obeys the commands of the party').split()
>>> hyp2 = str('It is to insure the troops forever hearing the activity '
...            'guidebook that party direct')
>>> corpus_chrf([ref1, ref2, ref1, ref2], [hyp1, hyp2, hyp2, hyp1]) # doctest: +ELLIPSIS
0.3910...

Parameters
references:list(list(str))	a corpus of list of reference sentences, w.r.t. hypotheses
hypotheses:list(list(str))	a list of hypothesis sentences
min_len:int	The minimum order of n-gram this function should extract.
max_len:int	The maximum order of n-gram this function should extract.
beta:float	the parameter to assign more importance to recall over precision
ignore_whitespace:bool	ignore whitespace characters in scoring
Returns
float	the sentence level CHRF score.

def sentence_chrf(reference, hypothesis, min_len=1, max_len=6, beta=3.0, ignore_whitespace=True): (source) ¶

Calculates the sentence level CHRF (Character n-gram F-score) described in

Maja Popovic. 2015. CHRF: Character n-gram F-score for Automatic MT Evaluation. In Proceedings of the 10th Workshop on Machine Translation. http://www.statmt.org/wmt15/pdf/WMT49.pdf
Maja Popovic. 2016. CHRF Deconstructed: β Parameters and n-gram Weights. In Proceedings of the 1st Conference on Machine Translation. http://www.statmt.org/wmt16/pdf/W16-2341.pdf

This implementation of CHRF only supports a single reference at the moment.

For details not reported in the paper, consult Maja Popovic's original implementation: https://github.com/m-popovic/chrF

The code should output results equivalent to running CHRF++ with the following options: -nw 0 -b 3

An example from the original BLEU paper http://www.aclweb.org/anthology/P02-1040.pdf

>>> ref1 = str('It is a guide to action that ensures that the military '
...            'will forever heed Party commands').split()
>>> hyp1 = str('It is a guide to action which ensures that the military '
...            'always obeys the commands of the party').split()
>>> hyp2 = str('It is to insure the troops forever hearing the activity '
...            'guidebook that party direct').split()
>>> sentence_chrf(ref1, hyp1) # doctest: +ELLIPSIS
0.6349...
>>> sentence_chrf(ref1, hyp2) # doctest: +ELLIPSIS
0.3330...

The infamous "the the the ... " example

>>> ref = 'the cat is on the mat'.split()
>>> hyp = 'the the the the the the the'.split()
>>> sentence_chrf(ref, hyp)  # doctest: +ELLIPSIS
0.1468...

An example to show that this function allows users to use strings instead of tokens, i.e. list(str) as inputs.

>>> ref1 = str('It is a guide to action that ensures that the military '
...            'will forever heed Party commands')
>>> hyp1 = str('It is a guide to action which ensures that the military '
...            'always obeys the commands of the party')
>>> sentence_chrf(ref1, hyp1) # doctest: +ELLIPSIS
0.6349...
>>> type(ref1) == type(hyp1) == str
True
>>> sentence_chrf(ref1.split(), hyp1.split()) # doctest: +ELLIPSIS
0.6349...

To skip the unigrams and only use 2- to 3-grams:

>>> sentence_chrf(ref1, hyp1, min_len=2, max_len=3) # doctest: +ELLIPSIS
0.6617...

Parameters
reference	Undocumented
hypothesis:list(str) / str	a hypothesis sentence
min_len:int	The minimum order of n-gram this function should extract.
max_len:int	The maximum order of n-gram this function should extract.
beta:float	the parameter to assign more importance to recall over precision
ignore_whitespace:bool	ignore whitespace characters in scoring
references:list(str) / str	reference sentence
Returns
float	the sentence level CHRF score.

def _preprocess(sent, ignore_whitespace): (source) ¶

Undocumented

nltk.translate.chrf_score

`nltk.translate.chrf_score`