Skip to content
Our LLM-judge paper was accepted at two ICML 2026 workshops. Read it here

Research

We build evaluation models, so we spend most of our time on a hard question: how do you know an evaluator is right? These are our own experiments and findings — judge reliability, reward model benchmarks, and the failure patterns we see inside production AI systems.

We publish the negative results too — the techniques that didn't work go out alongside the ones that did. Code and data are open where the underlying data allows it. One accepted workshop paper, an open benchmark, a failure-mode ontology, and more on the way.

Papers

  1. arXiv:2604.13717

    ICML 2026 workshop paper

    On Cost-Effective LLM-as-a-Judge Improvement Techniques

    Ryan Lail, Luke Markham ·

    Accepted at two ICML 2026 workshops — Statistical Frameworks for Uncertainty in Agentic Systems, and Combining Theory and Benchmarks — Seoul, July 2026.

    We tested four drop-in techniques for improving LLM-as-a-judge accuracy on RewardBench 2. Two of them — task-specific criteria injection and ensembling — reach up to 85.8% accuracy, 13.5 points above a 71.7% baseline, at 1.3x baseline cost, with the gains holding across both the OpenAI GPT and Anthropic Claude families. No fine-tuning required. The techniques that didn't win are published alongside the ones that did.

    Read the paper → Write-up Code

In the pipeline

What we're writing up now. Preprints are added above the day they go public.

  • Submitting NeurIPS 2026 workshop

    What LLM judges miss when something has been left out of a clinical note, and what recovers it

  • Submitting NeurIPS 2026 workshop

    Which of a judge's uncertain cases are actually worth a human's time

  • In progress

    What happens to a judge's validity when you optimise it for agreement

Write-ups

A Living Map of LLM Failure Modes: Dominant, Emerging, Squashed

Most failure-mode analysis treats failures as a static taxonomy. In production they're a living distribution - some dominant, some emerging, some being squashed by recent fixes. Here's a clustering pipeline that tracks all three.

Luke Markham

An Ontology of LLM Failure Modes

A structured taxonomy of 60+ failure modes across eight categories, synthesizing recent research into a practical framework for understanding how and why large language models fail.

Luke Markham

LLMs: Great Witnesses, Terrible Judges

LLM-as-a-judge consistency is largely illusory. The same hallucination produces wildly different scores depending on scale configuration, undermining evaluation trust.

Ryan Lail

Code & datasets

Open, so you can check the results rather than take our word for them.

llm-judge-criteria-ensembling

Code, prompts and raw results behind the RewardBench 2 paper — every condition we ran, including the techniques that didn't win.

MIT licensed

GitHub →

PrimeBench

Our open preference benchmark for real-world evaluation. Paired responses edited along conflicting criteria — comprehensive vs concise, hallucinated vs factual, technical vs simple — over FinQA, XSUM, PubMed and TechQA, to test whether an evaluator can tell nuanced differences apart.

Open dataset + evaluation scripts

Hugging Face → GitHub →

Looking for practical guides instead? Read the blog.