neural_compressor.experimental.metric.bleu

Script for BLEU metric.

Module Contents

Classes

UnicodeRegex

Ad-hoc hack to recognize all punctuation and symbols.

BLEU

Computes the BLEU (Bilingual Evaluation Understudy) score.

Functions

bleu_tokenize(→ List[str])

Tokenize a string following the official BLEU implementation.

class neural_compressor.experimental.metric.bleu.UnicodeRegex

Bases: object

Ad-hoc hack to recognize all punctuation and symbols.

nondigit_punct_re

The compiled regular expressions to recognize punctuation preceded with a digit.

punct_nondigit_re

The compiled regular expressions to recognize punctuation followed by a digit.

symbol_re

The compiled regular expressions to recognize symbols.

property_chars(prefix: str) str

Collect all Unicode strings starting with a specific prefix.

Parameters:

prefix – The specific prefix.

Returns:

The join result of all Unicode strings starting

with a specific prefix.

Return type:

punctuation

neural_compressor.experimental.metric.bleu.bleu_tokenize(string: str) List[str]

Tokenize a string following the official BLEU implementation.

See https://github.com/moses-smt/mosesdecoder/

“blob/master/scripts/generic/mteval-v14.pl#L954-L983

Parameters:

string – The string to be tokenized.

Returns:

A list of tokens.

Return type:

tokens

class neural_compressor.experimental.metric.bleu.BLEU

Bases: object

Computes the BLEU (Bilingual Evaluation Understudy) score.

BLEU is an algorithm for evaluating the quality of text which has been machine-translated from one natural language to another. This implementent approximate the BLEU score since we do not glue word pieces or decode the ids and tokenize the output. By default, we use ngram order of 4 and use brevity penalty. Also, this does not have beam search.

predictions

List of translations to score.

labels

List of the reference corresponding to the prediction result.

reset() None

Clear the predictions and labels in the cache.

update(prediction: Sequence[str], label: Sequence[str]) None

Add the prediction and label.

Parameters:
  • prediction – The prediction result.

  • label – The reference corresponding to the prediction result.

Raises:
  • ValueError – An error occurred when the length of the prediction

  • and label are different.

result() float

Compute the BLEU score.

Returns:

The approximate BLEU score.

Return type:

bleu_score