The measurement problem
Two approaches dominate Agentic and GenAI evaluation today, and both fall short.
LLM-as-Judge is cheap and scalable, but systematically biased. LLM judges favor verbose answers, agree with their own outputs, and produce scores that are noisy proxies.
Human annotation is the gold standard, but it doesn't scale. A statistically meaningful sample requires thousands of labels per cycle, making continuous evaluation economically unviable.
Many product teams ship systems whose true performance is unknown, iterate on metrics they cannot trust, and have no reliable way to detect when a system's performance degrades.
A third way: Prediction-Powered Inference
Rather than choosing between human and machine judgment, Prediction-Powered Inference combines them. A small set of human labels is used to estimate and correct the LLM judge's systematic bias across large volumes of proxy judgments. The result: valid, tight confidence intervals at a fraction of the annotation cost.
The underlying technique, published in Science, doesn't ask the LLM judge to be right. It asks it to be consistently wrong in a measurable way. Once the bias is characterized, it can be corrected.
Introducing GLIDE
GLIDE is Emerton Data's open-source Python package that makes these state-of-the-art techniques accessible to every practitioner, including a suite of samplers that optimize where your annotation budget is spent. It structures evaluation into three stages: sampling (allocating your annotation budget optimally), annotation (collecting the minimum viable set of human labels), and estimation (producing an unbiased metric with formal confidence intervals).
In a few lines of code, teams can combine LLM proxy scores with a small human-annotated sample to produce performance estimates they can defend to their peers, their stakeholders, and themselves.
GLIDE doesn't stop at one-shot evaluation. It extends the same PPI-based estimates to continuous monitoring in production, tracking system performance over time and flagging statistically significant drift before it becomes a business problem.
GLIDE is available on GitHub and PyPI, and the development team share their progress as well as real-world in a dedicated monthly newsletter available on Linkedin.
Organisations that measure accurately iterate faster, catch regressions earlier, and make better deployment decisions.