# The BLEU translation quality metric

[BLEU](https://wikipedia.org/wiki/BLEU) (Bilingual Evaluation Understudy)
is a metric for evaluating machine-translated text. A BLEU score is a
number between zero and one that measures the similarity of the
machine-translated text to a set of high-quality reference translations:

- A score of 0 means that the machine-translated output has no overlap with the reference translation, which indicates a low quality of translation.
- A score of 1 means there is perfect overlap with the reference translations, which indicates a high quality of translation.

## BLEU limitations

- **BLEU is a Corpus-based Metric.** The BLEU metric performs poorly when
  used to evaluate individual sentences.
  Single sentences can get very low BLEU scores, even when they
  capture most of the meaning. Because n-gram statistics for individual
  sentences are less meaningful, BLEU is by design a corpus-based metric,
  where statistics are accumulated over an entire corpus when computing the
  score. The BLEU metric can't be factorized for individual sentences.

- **BLEU makes no distinction between content and function words.**
  The BLEU metric doesn't distinguish content from function words.
  A dropped function word like *a* gets the same penalty as if
  the name *NASA* were erroneously replaced with *ESA*.

- **BLEU is not good at capturing sentence meaning and grammaticality.**
  Dropping a single word like *not* can change sentence polarity.
  Also, taking only n-grams into account with n≤4 ignores long-range
  dependencies, so BLEU often imposes only a small penalty for
  ungrammatical sentences.

- **BLEU relies on normalization and tokenization.** Prior to computing the
  BLEU score, the reference and candidate translations are normalized and
  tokenized. The choice of steps in those processes significantly affect
  the final BLEU score.

## How to interpret BLEU scores

The following is a rough guideline suggesting how to interpret BLEU scores that
are expressed as percentages, not decimals:

| BLEU % Score | Interpretation |
|---|---|
| \< 10 | Almost useless |
| 10 - 19 | Hard to get the gist |
| 20 - 29 | The gist is clear, but has significant grammatical errors |
| 30 - 40 | Understandable to good translations |
| 40 - 50 | High quality translations |
| 50 - 60 | Very high quality, adequate, and fluent translations |
| \> 60 | Quality often better than human |

The following color gradient can be used as a general scale
[interpretation of the BLEU score](https://www.cs.cmu.edu/%7Ealavie/Presentations/MT-Evaluation-MT-Summit-Tutorial-19Sep11.pdf):

![General interpretability of scale](https://docs.cloud.google.com/static/translate/automl/docs/images/bleu_score_range.png "General interpretability of scale")

## Mathematical details

Mathematically, the BLEU score is defined as:
$$ \\text{BLEU} = \\underbrace{\\vphantom{\\prod_i\^4}\\min\\Big(1, \\exp\\big(1-\\frac{\\text{reference-length}} {\\text{output-length}}\\big)\\Big)}_{\\text{brevity penalty}} \\underbrace{\\Big(\\prod_{i=1}\^{4} precision_i\\Big)\^{1/4}}_{\\text{n-gram overlap}} $$

with

\\\[
precision_i = \\dfrac{\\sum_{\\text{snt}\\in\\text{Cand-Corpus}}\\sum_{i\\in\\text{snt}}\\min(m\^i_{cand}, m\^i_{ref})}
{w_t\^i = \\sum_{\\text{snt'}\\in\\text{Cand-Corpus}}\\sum_{i'\\in\\text{snt'}} m\^{i'}_{cand}}
\\\]

where

- \\(m_{cand}\^i\\hphantom{xi}\\) is the count of i-gram in candidate matching the reference translation.
- \\(m_{ref}\^i\\hphantom{xxx}\\) is the count of i-gram in the reference translation.
- \\(w_t\^i\\hphantom{m_{max}}\\) is the total number of i-grams in candidate translation.

<br />

The formula consists of two parts: the brevity penalty and the n-gram overlap.

- **Brevity penalty.** The brevity penalty penalizes generated translations
  that are too short compared to the closest reference length with an
  exponential decay. The brevity penalty compensates for the fact that the
  BLEU score has no
  [recall](https://developers.google.com/machine-learning/crash-course/glossary#recall)
  term.

- **N-Gram overlap.** The n-gram overlap counts how many unigrams, bigrams,
  trigrams, and four-grams (*i* =1,...,4) match their n-gram counterpart in
  the reference translations. This term acts as a
  [precision](https://developers.google.com/machine-learning/crash-course/glossary#precision)
  metric. Unigrams account for
  *adequacy* while longer n-grams account for *fluency* of the translation.
  To avoid overcounting, the n-gram counts are clipped to the maximal n-gram
  count occurring in the reference (\\(m_{ref}\^n\\)).

### Example: Calculating \\(precision_1\\)

Consider this reference sentence and candidate translation:

**Reference:** `the cat is on the mat`  

**Candidate:** `the the the cat mat`

The first step is to count the occurrences of each unigram in the reference and
the candidate. Note that the BLEU metric is case-sensitive.

| Unigram | \\(m_{cand}\^i\\hphantom{xi}\\) | \\(m_{ref}\^i\\hphantom{xxx}\\) | \\(\\min(m\^i_{cand}, m\^i_{ref})\\) |
|---|---|---|---|
| `the` | 3 | 2 | 2 |
| `cat` | 1 | 1 | 1 |
| `is` | 0 | 1 | 0 |
| `on` | 0 | 1 | 0 |
| `mat` | 1 | 1 | 1 |

The total number of unigrams in the candidate (\\(w_t\^1\\)) is 5, so \\(precision_1\\) = (2 + 1 + 1)/5 = 0.8.

### Example: Calculating the BLEU score

**Reference:**
`The NASA Opportunity rover is battling a massive dust storm on Mars.`  

**Candidate 1:**
`The Opportunity rover is combating a big sandstorm on Mars.`  

**Candidate 2:**
`A NASA rover is fighting a massive storm on Mars.`

The preceding example consists of a single reference and two candidate
translations. The sentences are tokenized prior to computing the BLEU score.
For example, the final period is counted as a separate token.

To compute the BLEU score for each translation, we compute the following statistics.

- **N-Gram Precisions.** The following table contains the n-gram precisions for both candidates.
- **Brevity-Penalty.** The brevity-penalty is the same for candidate 1 and candidate 2 since both sentences consist of 11 tokens.
- **BLEU-Score.** At least one matching 4-gram is required to get a BLEU score \> 0. Since candidate translation 1 has no matching 4-gram, it has a BLEU score of 0.

| Metric | Candidate 1 | Candidate 2 |
|---|---|---|
| \\(precision_1\\) (1gram) | 8/11 | 9/11 |
| \\(precision_2\\) (2gram) | 4/10 | 5/10 |
| \\(precision_3\\) (3gram) | 2/9 | 2/9 |
| \\(precision_4\\) (4gram) | 0/8 | 1/8 |
| Brevity-Penalty | 0.83 | 0.83 |
| **BLEU-Score** | **0.0** | **0.27** |