GLUE and SuperGLUE: measuring language understanding

By Hiva Mohammadzadeh · Stanford MSCS, UC Berkeley EECS

Why we need a benchmark

Before GLUE existed, comparing language models was inconsistent. Every paper picked its own tasks, its own datasets, and its own metrics. You could not read two papers and say which model understood language better, because they measured different things.

GLUE gave the community a single, standardized evaluation suite that covers a broad range of language understanding capabilities. It provides one number to compare models and one leaderboard to track progress. Without a shared benchmark, the field cannot measure whether it makes real progress or just overfits to individual datasets.

What is GLUE?

GLUE, short for General Language Understanding Evaluation, is a collection of resources for training, evaluating, and analyzing natural language understanding systems. It bundles together:

• Nine language understanding tasks built on established existing datasets, selected to cover a diverse range of dataset sizes, text genres, and degrees of difficulty.
• A diagnostic dataset designed to evaluate and analyze model performance with respect to a wide range of linguistic phenomena found in natural language.
• A public leaderboard for tracking performance on the benchmark and a dashboard for visualizing the performance of models on the diagnostic set.
• A handcrafted diagnostic test suite that enables detailed linguistic analysis of models.

The tasks span question answering, sentiment analysis, and textual entailment. All tasks are single-sentence or sentence-pair classification, except STS-B, which is a regression task.

How the GLUE score works

GLUE lets researchers compare models with a single number, produced in four steps:

1. Swap the head. Remove the pretraining classification layer from your model and replace it with one that accommodates the output of the GLUE task.
2. Fine-tune. Train the model on each of the nine tasks.
3. Score. Evaluate the model on all nine tasks.
4. Average. The resulting average score of those nine tasks is the model's final GLUE performance score.

The process is the same regardless of the underlying model. BERT, RoBERTa, and XLNet all go through this pipeline, and that uniformity is what makes the comparison meaningful.

The nine GLUE tasks

The nine tasks fall into three categories: single-sentence tasks, similarity and paraphrase tasks, and inference tasks. I will walk through each one with its key numbers and a concrete example so you can see exactly what the model is asked to do.

Single-sentence tasks

These tasks give the model a single sentence and ask it to classify it along some dimension.

CoLA (Corpus of Linguistic Acceptability)

What it tests: Grammatical correctness. Given a sentence, is it linguistically acceptable?

CoLA contains 10,657 sentences drawn from 23 linguistics publications, split into 8,551 train / 1,043 validation / 1,063 test. It is a binary classification task scored by Matthews Correlation Coefficient (MCC) rather than accuracy, because the classes can be imbalanced and MCC handles that better.

Example:

Sentence: "Our friends won't buy this analysis, let alone the next one we propose." Label: 1 (acceptable)

The CoLA sentences come from linguistics papers, so they include deliberately constructed examples that probe subtle grammatical boundaries. The task tests whether the model has internalized the formal rules of English syntax.

SST-2 (Stanford Sentiment Treebank)

What it tests: Sentiment classification. Is the expressed opinion positive or negative?

SST-2 has 70,042 sentences from movie reviews, split into 67,349 train / 872 validation / 1,822 test. Binary classification, scored by accuracy.

Example:

Sentence: "that loves its characters and communicates something rather beautiful about human nature" Label: 1 (positive)

SST-2 is one of the larger GLUE tasks, and movie reviews give it rich, expressive language. The sentences range from obvious ("this movie is terrible") to nuanced, requiring the model to track sentiment through complex syntactic structures.

Similarity and paraphrase tasks

These tasks give the model two sentences and ask whether they mean the same thing, or how similar they are.

MRPC (Microsoft Research Paraphrase Corpus)

What it tests: Semantic equivalence. Are two sentences paraphrases of each other?

MRPC contains 5,800 sentence pairs from online news sources, split into 3,700 train / 1,700 test. Binary classification, scored by accuracy and F1. The dataset is imbalanced, with 68% of pairs positive (paraphrases), which is why F1 matters alongside accuracy.

Example:

Sentence 1: "Automaker sales were up 2.5 percent in the first quarter." Sentence 2: "Sales at automakers rose 2.5 percent in the January-March period." Label: 1 (paraphrase)

This pair shows what makes paraphrase detection hard. The two sentences say the same thing with different words and different syntactic structures. The model has to understand that "first quarter" and "January-March period" are the same concept, and that "were up" and "rose" are semantically equivalent.

QQP (Quora Question Pairs)

What it tests: Question semantic equivalence. Are two questions asking the same thing?

QQP is a large dataset of 795,242 sentence pairs, split into 363,846 train / 40,430 validation / 390,965 test. Binary classification, scored by accuracy and F1. The dataset is imbalanced in the opposite direction from MRPC, with 63% of pairs negative (not duplicates).

With nearly 800K pairs, QQP is one of the largest GLUE tasks and stress-tests how models handle massive training sets. The questions come from Quora, so they cover a wide range of topics and use the informal, real-world language that carefully curated academic datasets sometimes miss.

STS-B (Semantic Textual Similarity Benchmark)

What it tests: Degree of semantic similarity on a continuous scale from 1 to 5.

STS-B has 8,628 sentence pairs from news headlines, image captions, and NLI data, split into 5,749 train / 1,500 validation / 1,379 test. It is the only regression task in GLUE. Instead of a class label, the model outputs a continuous similarity score, evaluated with Pearson and Spearman correlation coefficients.

Example:

Sentence 1: "A plane is taking off." Sentence 2: "An air plane is taking off." Score: 5.000

A score of 5.000 means the sentences are semantically identical. The regression format makes STS-B harder than the classification tasks, because the model cannot learn a single decision boundary. It has to learn a meaningful ordering across a continuous range.

Inference tasks

These tasks test whether a model can reason about the relationship between two sentences, typically whether one entails, contradicts, or is neutral with respect to the other.

MNLI (Multi-Genre Natural Language Inference)

What it tests: Textual entailment. Given a premise and a hypothesis, is the relationship entailment, contradiction, or neutral?

MNLI is the largest inference task with 392,702 training pairs drawn from 10 different genres of text. It has two validation sets: matched (9,815 examples from the same genres as training) and mismatched (9,832 examples from different genres). Three-class classification, scored by accuracy on both matched and mismatched sets.

The matched/mismatched split is MNLI's key design choice. It explicitly tests whether models generalize across genres. A model that does well on matched but poorly on mismatched has overfit to the training distribution, and MNLI exposes that.

QNLI (Question Natural Language Inference)

What it tests: Whether a sentence contains the answer to a question.

QNLI has approximately 110,000 sentence pairs converted from the Stanford Question Answering Dataset, with about 105,000 train / 5,400 test. The sentences are paired with paragraphs from Wikipedia. Binary classification (entailment or not), scored by accuracy.

QNLI is derived from SQuAD by converting each question-paragraph pair into a set of sentence-level pairs. If a sentence contains the answer to the question, it is labeled as entailment. This reformulation turns an extractive QA task into a sentence-pair classification task, which fits the GLUE format.

RTE (Recognizing Textual Entailment)

What it tests: Whether a hypothesis can be inferred from a premise.

RTE aggregates data from a series of annual textual entailment challenges, with approximately 5,500 sentence pairs (2,500 train / 3,000 test). The text comes from news and Wikipedia. Binary classification (entailment or not), scored by accuracy.

RTE is one of the smaller GLUE tasks, which makes it a test of how well models learn from limited data. Many strong models struggle here because there is not enough training signal to fully adapt the pretrained representations.

WNLI (Winograd Schema Challenge)

What it tests: Pronoun resolution. What does a pronoun refer to in context?

WNLI has about 1,000 sentence pairs (634 train / 146 test) drawn from fiction books. The test set is imbalanced, with 65% of examples labeled as not entailment. Scored by accuracy.

WNLI is difficult and small. Many GLUE submissions skip it or report near-chance performance. The pronoun resolution task requires genuine commonsense reasoning, which statistical pattern matching alone often cannot solve.

GLUE tasks at a glance

Enter SuperGLUE

GLUE served its purpose until models got too good at it. When BERT and its successors started approaching or exceeding human-level performance on GLUE, the benchmark lost its ability to differentiate between models. Once every model scores above 89, the leaderboard stops being informative.

SuperGLUE answered that with a harder benchmark. It upgrades GLUE with a new set of more difficult language understanding tasks, a software toolkit, and a public leaderboard. The design philosophy is the same, a standardized suite that produces a single comparable score, but the tasks are deliberately chosen to be beyond the reach of then-current models.

What SuperGLUE improved

SuperGLUE made several structural changes beyond harder tasks:

• More challenging tasks. SuperGLUE retains the two hardest tasks from GLUE (RTE and the Winograd-style challenge) and adds new tasks selected from an open call for contributions. The bar for inclusion was high, because tasks had to show substantial headroom between BERT-level baselines and human performance.
• More diverse task formats. GLUE is entirely sentence classification and regression. SuperGLUE adds coreference resolution and question answering formats, testing a broader range of model capabilities.
• Human baselines. Every SuperGLUE task comes with carefully measured human performance baselines, making it possible to quantify exactly how far models have to go.
• Improved code support. SuperGLUE ships with a new modular toolkit built on PyTorch and AllenNLP, making it easier to run evaluations consistently.
• Refined usage rules. The rules were updated to ensure fair competition and give full credit to the original data and task creators.

The eight SuperGLUE tasks

SuperGLUE includes eight tasks:

BoolQ is yes/no question answering. Given a passage and a question, the model answers yes or no. It contains 15,942 examples (9,427 train / 3,270 validation / 3,245 test) sourced from Google search queries paired with Wikipedia passages. Scored by accuracy.

CB (CommitmentBank) is natural language inference focused on the degree of commitment a speaker has to a clause. It is a small but challenging dataset with 557 examples (250 train / 57 validation / 250 test). Scored by accuracy and F1.

COPA (Choice of Plausible Alternatives) tests causal reasoning. Given a premise, the model chooses which of two alternatives is the more plausible cause or effect. It contains 1,000 examples (400 train / 100 validation / 500 test) from blogs and a photography encyclopedia. Scored by accuracy.

MultiRC is multi-sentence reading comprehension. Given a passage and a question, the model selects all correct answers from a set of candidates. This tests whether models can synthesize information across multiple sentences.

ReCoRD is reading comprehension with commonsense reasoning. The model must fill in a missing entity in a passage summary, which requires both passage understanding and world knowledge.

RTE is retained from GLUE, with the same textual entailment task and format. It appears in both benchmarks because it is still hard.

WiC (Word-in-Context) is word sense disambiguation. Given a word used in two different sentences, the model determines whether it has the same meaning in both. This isolates a core challenge in NLP, because the same word can mean different things depending on context.

WSC (Winograd Schema Challenge) is pronoun coreference resolution, an upgraded version of GLUE's WNLI reframed as a coreference task. The model must determine which noun a pronoun refers to, which requires commonsense reasoning about the described situation.

Key takeaways

• Benchmarks exist to make progress measurable. Before GLUE, comparing language models was subjective and inconsistent. GLUE gave the field a common yardstick, and SuperGLUE raised the bar when that yardstick was no longer discriminative.
• The GLUE score is a single number, but it summarizes nine very different capabilities. Grammar checking, sentiment analysis, paraphrase detection, textual entailment, and pronoun resolution each test something distinct. A high GLUE score means the model handles language understanding broadly rather than one narrow skill.
• Task diversity matters. GLUE deliberately spans different dataset sizes (from 634 to 392,702 training examples), different text genres (news, fiction, movie reviews, Quora questions), and different difficulty levels. A model that only works on large datasets or easy tasks will not score well.
• When models saturate a benchmark, you need a harder one. That is why SuperGLUE exists. Progress on GLUE plateaued because the tasks were no longer hard enough to differentiate top models, not because language understanding was solved.
• Small datasets are the real test. RTE, WNLI, MRPC, and CoLA are all relatively small. They test whether a pretrained model can transfer its knowledge to settings with limited fine-tuning data, which is often the realistic scenario in practice.
• SuperGLUE is better methodology as well as harder tasks. Measured human baselines, diverse task formats, and refined evaluation rules make it a more rigorous benchmark as well as a more difficult one.

The path from GLUE to SuperGLUE is a template for how the field evolves. You build a benchmark, push models until they saturate it, identify the gaps, and build a harder benchmark. The tasks change each round, but the method of measuring progress stays the same.

Based on presentations I created in 2023.