The measurement problem
Two approaches dominate the evaluation of agentic and GenAI systems today, and both fall short.
LLM-as-judge is cheap and scalable, but systematically biased. LLM judges favour verbose answers, agree with their own outputs and produce scores that are noisy proxies.
Human annotation is the gold standard, but it does not scale. A statistically meaningful sample requires thousands of labels per cycle, which makes continuous evaluation economically unviable.
As a result, many product teams ship systems whose true performance is unknown, iterate on metrics they cannot trust, and have no reliable way to detect when a system’s performance degrades.
A third way: prediction-powered inference
Rather than choosing between human and machine judgement, prediction-powered inference combines them. A small set of human labels is used to estimate and correct the LLM judge’s systematic bias across large volumes of proxy judgements. The result: valid, tight confidence intervals at a fraction of the annotation cost.
The underlying technique, published in Science, does not ask the LLM judge to be right. It asks it to be consistently wrong in a measurable way. Once the bias is characterised, it can be corrected.
Introducing GLIDE
GLIDE is Emerton Data’s open-source Python package that makes these state-of-the-art techniques accessible to every practitioner, including a suite of samplers that optimise where your annotation budget is spent. It structures evaluation into three stages:
- Sampling: allocating your annotation budget optimally.
- Annotation: collecting the minimum viable set of human labels.
- Estimation: producing an unbiased metric with formal confidence intervals.
In a few lines of code, teams can combine LLM proxy scores with a small human-annotated sample to produce performance estimates they can defend to their peers, their stakeholders and themselves.
GLIDE does not stop at one-shot evaluation. It extends the same estimates to continuous monitoring in production, tracking system performance over time and flagging statistically significant drift before it becomes a business problem.
GLIDE is available on GitHub and PyPI. The development team shares its progress and real-world use cases in a monthly newsletter on LinkedIn.
Organisations that measure accurately iterate faster, catch regressions earlier and make better deployment decisions.
