Leaderboards Grade the Best Draw. Diversity Never Shows Up.

The scoreboard we keep using

We still rank models on the best single answer. Then we act surprised when production agents spit out interchangeable UIs, jokes, and SVG compositions.

I used to wave this off as an academic nitpick. Then I actually sat with what the evaluation literature measures.

NoveltyBench audited 67 COLM 2024 and ICLR 2025 benchmark papers. Over 90% score only a single generation or the best one. Across 1,100 prompts, all 20 frontier models landed below human diversity. The researcher site is at novelty-bench.github.io.

Peak quality and plural outputs are not the same axis. Most leaderboards only instrument the first.

What “best draw” hides

Pass@1 answers a useful question: did at least one sample work? It does not answer whether the system still has optionality under the same brief.

What we measure	What we ignore
Best / single generation	Spread across the draw
Appropriateness / criteria pass	Originality and creative range
Peak score on a lucky sample	Whether peers and adversaries can predict the default

NoveltyBench’s punchline is not that models are “bad at creativity” in the abstract. It is that our field mostly never measures whether a model can produce distinct high-quality continuations when the task is open-ended. Larger models in a family often look less diverse, not more — which is the opposite of the story leaderboards tell when they only report peak quality.

"We grade the trophy. We never open the rest of the deck."

Product-side receipt: Design Creativity Bench

If NoveltyBench is the meta-eval critique, Design Creativity Bench is the product-side receipt. Kombai scores UI work on originality (cross-model distinctiveness on the same prompt), creative range (within-model change across domain-paired briefs), and appropriateness (criteria pass rate). Live numbers sit on the Kombai leaderboard.

Signal	Models	Humans
Overall creativity (best model vs reference)	GPT-6 Astra 65.5	92.3
Appropriateness	Every tested frontier model >90% (Claude Opus 5.5 99.2%)	98.0%
Originality (pooled)	0.592	Human–model 0.764
Creative range (pooled)	0.581	0.902

Appropriateness above 90% can coexist with collapsed originality. That is the uncomfortable combination: models clear the brief and still explore a narrower design space than humans. The paper run covered 15 leading models × 160 designs each (2,400 primary model designs) across 80 tasks; Dipanjan Dey’s announcement framed the same gap across 16 frontier models on 80 open-ended UI tasks (X post).

One nuance from the paper that matters for systems people: originality ≠ adaptation. GPT-6 Astra can rank high on originality versus other models while sitting near the bottom on creative range across domain pairs. A model that looks distinctive in a head-to-head grid can still freeze when the product domain shifts.

The pelican is optional — the collapse is not

Aman Rusia’s essay used Simon Willison’s bare prompt — Generate an SVG of a pelican riding a bicycle — to show thirteen recent LLMs converging on near-identical compositions (side view, similar sky/road/sun/scarf motifs). Willison’s tag archive is at simonwillison.net/tags/pelican-riding-a-bicycle. The HN thread (162 points, 138 comments) split into familiar camps: underspecified prompts should regress to the mean, versus cross-model clones that go beyond “generic.”

I treat the pelican as illustration, not thesis. The thesis is distributional: when post-trained models share attractor basins, underspecified creative work collapses — whether the artifact is an SVG joke, a UI layout, or an agent’s first plan.

HN commenter jvanderbot pushed the systems implication I care about for agent stacks: same-y defaults are predictable to peers and adversaries. If every agent picks the same path from the same prompt, you have not only aesthetic sameness — you have a cleanroom reverse-engineering problem.

What this means for agent evals

I keep coming back to this when I design multi-agent evals.

1. Pass@1 is a sample, not a distribution. A green check means one draw worked. It does not mean the harness still has room when you regenerate, fan out, or ask for alternatives.
2. Appropriateness can mask collapse. High criteria-pass rates coexist with low originality and creative range. Instrument both, or you will ship “correct” and “interchangeable” at the same time.
3. Default paths compound. Agent graphs that always pick the same first tool, the same outline, or the same UI scaffold inherit the model’s attractor — then amplify it across steps.
4. Adversarial predictability is a feature of sameness. Identical defaults are easier to anticipate, imitate, and attack than a system that still has optionality under the same brief.

A sketch of the difference in how you score an open-ended agent step:

# Peak-only: did the lucky sample pass?
def pass_at_1(samples: list[str], judge) -> bool:
    return any(judge(s).ok for s in samples[:1])

# Distributional: how many distinct-and-good draws before patience runs out?
def distinct_good_k(samples: list[str], judge, equiv, k: int = 5) -> int:
    good = [s for s in samples if judge(s).ok]
    clusters = equiv.cluster(good)  # NoveltyBench-style equivalence
    return min(len(clusters), k)

NoveltyBench’s utility framing (user patience over distinct successful replies) is closer to how product users actually behave than a single trophy number. You do not need their exact classifier to adopt the mindset: score the room left in the distribution, not only the best card on the table.

Competing views worth keeping

Not every sameness story is mode collapse. Some HN voices argue a bare creative prompt should yield a boring side-view bike scene (joshkel, randomblock1). Prompt quality and external randomization shift the distribution; temperature alone is not creativity. Willison has also warned about benchmark contamination if labs train on published pelican SVGs.

Those caveats do not rescue peak-only leaderboards. Even if you believe underspecification explains one SVG meme, NoveltyBench’s paper audit and Design Creativity Bench’s appropriateness-vs-creativity split still say the same thing: we instrument “did it work once,” not “does the distribution still have room.”

Mohammadi’s earlier work on alignment tax and attractor states and CreativityPrism’s finding that quality / novelty / diversity do not collapse into one score (arXiv 2510.20091) point the same direction from different angles.

---

In a nutshell

Leaderboards still grade the best single draw. NoveltyBench shows that more than 90% of recent benchmark papers never measure diversity at all, and that frontier models sit below humans on that axis across 1,100 prompts. Design Creativity Bench shows the product version of the trap: appropriateness above 90% alongside creativity scores stuck near 65 versus a human reference of 92.3. For agent evals, the future is not a taller podium of peak answers — it is distributional metrics that tell you whether optionality is still there.