One Example Is Enough to Pass Fairness Benchmarks Rethinking Fairness Evaluation for Aligned LLMs

Abstract

Fairness benchmarks such as BBQ have become the de facto standard for fairness evaluation across major model families. We argue that these benchmarks are too easy to support their role: training Qwen 2.5 7B Base with Group Relative Policy Optimization (GRPO) on a single BBQ example, or placing that example in context as a one-shot demonstration for in-context learning (ICL), lifts mean BBQ accuracy from 79.9% to 92.9% and 99.0%, respectively, closing 80% of the gap to its large-scale RLHF counterpart (96.1%) with GRPO, and surpassing it with ICL. These effects generalize across model families. A cross-conditioning analysis shows the improvement is carried by the reasoning traces generated by the model, and one example suffices to elicit a category-agnostic “missing evidence” reasoning pattern. We argue that BBQ-style multiple-choice abstention benchmarks measure a single structural cue, and a model that solves them does not thereby become fair. We call for evaluation suites that cover a broader spectrum of fairness alignment.

Findings

• One example saturates the benchmark. One-shot GRPO closes ~80% of the gap between base and RLHF-aligned Qwen 2.5 7B; one-shot ICL surpasses the RLHF model outright, with no weight updates at all.
• The specific example barely matters. We rank 1,000 BBQ examples by training reward variance and train on eight of them spanning the whole distribution (z1 … z1000). BBQ accuracy lands between 87.8 and 92.9 regardless of which one is used.
• It generalizes across families. Qwen 3 8B, Gemma 2 9B, Llama 3.1 8B, and Mistral 7B v0.3 all show large gains, in several cases matching or beating their RLHF counterparts. Mistral 7B v0.3 goes from 0.0 to 97.8 on BBQ.
• The gain lives in the reasoning trace, not the weights. A cross-conditioning ablation that swaps reasoning traces across checkpoints shows the trained model's advantage travels with the text it generates, despite a minimal weight difference from base.
• It does not transfer to real fairness. On RealToxicityPrompts, the RLHF model produces markedly less toxic continuations while the one-example model does not. The shortcut installs a refusal cue; it does not make the model safer in generation.
• Capability is preserved. At step 30, Qwen 2.5 7B and Qwen 3 8B retain MMLU and BBQ_disambig accuracy — so this is not a collapse into “always abstain.”

Results

Accuracy (%) on five fairness benchmarks. Bold is best within a model family, underline second best.

Variant	Example	BBQ	CrS	GMO	SSt	WnQ
Qwen 2.5 7B Base		79.9	40.1	40.4	27.6	57.2
GRPO	z1	87.8	60.4	76.5	36.3	69.9
GRPO	z2	92.8	67.9	84.5	41.1	77.6
GRPO	z251	92.9	68.4	86.4	43.0	79.5
GRPO	z501	91.4	65.0	82.5	38.5	76.7
GRPO	z751	92.5	65.0	84.5	40.1	78.2
GRPO	z876	92.5	65.2	82.6	40.8	79.8
GRPO	z999	91.6	66.2	83.1	38.8	76.1
GRPO	z1000	92.7	66.6	84.0	40.0	76.9
GRPO	AVG	91.8	65.6	83.0	39.8	76.8
ICL	z1	99.0	90.2	86.7	78.3	91.4
Instruct	—	96.1	77.2	98.1	56.7	83.7
Qwen 3 8B Base		56.3	42.3	73.1	32.2	58.0
GRPO	z1	86.6	80.5	94.9	51.4	92.3
ICL	z1	84.1	81.4	96.8	75.9	83.2
Instruct	—	97.5	69.4	84.3	47.2	87.9
Gemma 2 9B Base		14.0	11.4	24.3	9.7	13.4
GRPO	z1	96.4	97.4	92.5	96.8	98.1
ICL	z1	68.6	38.4	71.8	43.5	39.4
Instruct	—	95.6	92.7	100.0	78.5	99.5
Llama 3.1 8B Base		6.4	5.6	0.5	5.1	3.6
GRPO	z1	96.0	97.6	99.0	96.7	85.8
ICL	z1	98.9	52.6	58.6	42.3	59.6
Instruct	—	76.0	79.3	67.4	61.2	86.8
Mistral 7B v0.3 Base		0.0	0.0	0.0	0.0	0.0
GRPO	z1	97.8	68.2	83.6	66.7	71.9
ICL	z1	54.8	40.8	43.0	42.6	47.5
Instruct	—	45.3	45.8	45.2	29.0	59.6

BBQ = BBQ · CrS = CrowS-Pairs · GMO = genMO · SSt = StereoSet · WnQ = WinoQueer. “Base” is the out-of-the-box model, “GRPO” is one-shot GRPO training on example z_i, “ICL” is one-shot in-context learning with z₁, and “Instruct” is the large-scale RLHF variant.

Released models

We release the trained one-shot GRPO models as LoRA adapters (rank 32, α = 32) rather than merged weights. The adapter is the finding: a ~160 MB low-rank delta against a 15 GB base model is all that separates a model scoring 79.9 on BBQ from one scoring 92.9.

There is one repository per (base model, training example), and every GRPO step is a git revision. Each repository's main branch holds the step reported in the paper, so a plain load reproduces the published number.

All repositories live under the MichiganNLP org with the hacking-fairness-benchmarks- prefix.

Loading an adapter

from transformers import AutoModelForCausalLM, AutoTokenizer
from peft import PeftModel

base = AutoModelForCausalLM.from_pretrained("Qwen/Qwen2.5-7B", torch_dtype="bfloat16")
tok  = AutoTokenizer.from_pretrained("Qwen/Qwen2.5-7B")

# main == step30, the checkpoint reported in the paper
model = PeftModel.from_pretrained(
    base, "MichiganNLP/hacking-fairness-benchmarks-qwen2.5-7b-z1")

# any other point on the training trajectory
model = PeftModel.from_pretrained(
    base, "MichiganNLP/hacking-fairness-benchmarks-qwen2.5-7b-z1", revision="step100")

Models are prompted to answer in <think>…</think><answer>A</answer> format. Each adapter_config.json pins the exact base-model revision it was trained against.

BibTeX

@inproceedings{deng2026one,
  title     = {One Example Is Enough to Pass Fairness Benchmarks:
               Rethinking Fairness Evaluation for Aligned {LLM}s},
  author    = {Deng, Naihao and Arif, Samee and Chang, Shuaichen and
               Chen, Yulong and Mihalcea, Rada},
  booktitle = {Proceedings of the 2026 Conference on Empirical Methods in
               Natural Language Processing},
  year      = {2026}
}