nltk.translate.meteor_score

Aligns/matches words in the hypothesis to reference by sequentially applying exact match, stemmed match and wordnet based synonym match. In case there are multiple matches the match which has the least number of crossing is chosen.

matches exact words in hypothesis and reference and returns a word mapping based on the enumerated word id between hypothesis and reference

Calculates METEOR score for hypothesis with multiple references as described in "Meteor: An Automatic Metric for MT Evaluation with HighLevels of Correlation with Human Judgments" by Alon Lavie and Abhaya Agarwal, in Proceedings of ACL. http://www.cs.cmu.edu/~alavie/METEOR/pdf/Lavie-Agarwal-2007-METEOR.pdf

In case of multiple references the best score is chosen. This method iterates over single_meteor_score and picks the best pair among all the references for a given hypothesis

>>> hypothesis1 = 'It is a guide to action which ensures that the military always obeys the commands of the party'
>>> hypothesis2 = 'It is to insure the troops forever hearing the activity guidebook that party direct'

>>> reference1 = 'It is a guide to action that ensures that the military will forever heed Party commands'
>>> reference2 = 'It is the guiding principle which guarantees the military forces always being under the command of the Party'
>>> reference3 = 'It is the practical guide for the army always to heed the directions of the party'

>>> round(meteor_score([reference1, reference2, reference3], hypothesis1),4)
0.7398

If there is no words match during the alignment the method returns the score as 0. We can safely return a zero instead of raising a division by zero error as no match usually implies a bad translation.

>>> round(meteor_score(['this is a cat'], 'non matching hypothesis'),4)
0.0

Calculates METEOR score for single hypothesis and reference as per "Meteor: An Automatic Metric for MT Evaluation with HighLevels of Correlation with Human Judgments" by Alon Lavie and Abhaya Agarwal, in Proceedings of ACL. http://www.cs.cmu.edu/~alavie/METEOR/pdf/Lavie-Agarwal-2007-METEOR.pdf

>>> hypothesis1 = 'It is a guide to action which ensures that the military always obeys the commands of the party'

>>> reference1 = 'It is a guide to action that ensures that the military will forever heed Party commands'

>>> round(single_meteor_score(reference1, hypothesis1),4)
0.7398

If there is no words match during the alignment the method returns the score as 0. We can safely return a zero instead of raising a division by zero error as no match usually implies a bad translation.

>>> round(meteor_score('this is a cat', 'non matching hypothesis'),4)
0.0

Stems each word and matches them in hypothesis and reference and returns a word mapping between hypothesis and reference

Matches each word in reference to a word in hypothesis if any synonym of a hypothesis word is the exact match to the reference word.