When AI grades AI

This article is authored by Spurthi Tallam, senior software engineer - AI and ML, USA.

India is buying Artificial Intelligence (AI) at scale. Government systems now put language models in front of citizens. Kisan e-Mitra, the chatbot on the PM-KISAN portal, answers farmers' grievance queries in eleven languages. Banks and insurers are wiring similar systems into customer service and claims processing, and vendors compete for public and enterprise contracts on performance claims.

Artificial intelligence (Representative)

A claim like "94%" on a tender document looks objective and safe to sign.

Behind a growing share of those numbers sits a practise most buyers have never heard of. When a vendor's system answers ten thousand test questions, somebody has to decide which answers were good. Increasingly, that somebody is not a person but another AI: a large language model, prompted to score the outputs of the system under test.

The industry calls this "Large Language Model (LLM)-as-a-judge." It is fast, cheap, and scales to volumes no human review team can touch. The appeal is real: human evaluation at a national scale is genuinely slow and expensive, which is why LLM-as-a-judge has become standard in corporate evaluation pipelines and on the public leaderboards that rank the world's leading models.

Model judges fail in ways the number does not reveal. The Berkeley-led team that formalised the method in 2023 documented its flaws at the same time: judges favour longer answers, and favour whichever response happens to appear first.

They also favour their own outputs. Peer-reviewed work in 2024 and 2025 has since traced the phenomenon precisely: an evaluator scores its own output higher than human annotators do, and the bias extends to any model related to the judge, whether that is the same model, one from the same family, or one trained on its outputs. Researchers call it preference leakage and find it pervasive in AlpacaEval and Arena-Hard, two rankings the industry watches closely. AlpacaEval's own maintainers warn of it in their documentation. This phenomenon matters commercially, because a handful of foundation models sit behind both the judges and the products they judge.

Worse, model judges are least dependable exactly where evaluation matters most. On routine comparisons, the original 2023 study found GPT-4-class judges agreeing with human preferences more than 80 per cent of the time, often enough to seem trustworthy. But on JudgeBench, a harder test from the same Berkeley group, built from pairs where one answer is objectively correct and the other wrong, leading judges scored barely better than a coin flip. This March, RAND researchers found that none of the four state-of-the-art judges they tested were uniformly reliable; their consistency broke when the graded text was merely reformatted.

94% of what, judged by whom, against what standard?

I have spent nine years building machine learning systems for production, including owning evaluation for an LLM-powered agent that gave parenting guidance to consumers. In that domain a fluent wrong answer is worse than no answer at all.

An evaluation has to establish several things before its score means anything. It starts with a gold-standard judgement set: hundreds or thousands of real queries sampled from actual traffic, each response judged by trained human reviewers against a written rubric, disagreements surfaced and adjudicated rather than averaged away, how often reviewers agree with each other measured and reported, and the whole set kept aside so no system is ever tuned on it.

Building one is slow and expensive, and that expense is what makes the word "accurate" mean something.

Automated judging still has a real role, as a filter rather than a verdict. A model judge is good at catching that today's system is worse than yesterday's on the same thousand cases and at screening a million outputs to surface the few thousand that a human should read. Validate it against the human gold set and report the agreement rate alongside every score. Send ambiguous and low-confidence cases to people. Refresh the gold set as the product and its users drift.

Under those conditions, a score becomes an instrument reading with known error bars. Without them, the score is just a press release.

Which brings us to procurement. A ministry, a bank, or a state e-governance agency evaluating a vendor's claimed accuracy needs no doctorate, only five questions.

Who judged the outputs behind this number: humans or a model? If a model was used, which one was it, and does it belong to the same family as the system being scored? What human-judged standard validated the judge, and what was the agreement rate across how many cases? Is the test data held out and disclosed or supplied by the vendor? And can we re-run the evaluation ourselves, on our own cases, in our own languages?

Every one of these can be answered on a single page. Almost nobody in the Indian procurement chain is asking them. A vendor can clear a tender's accuracy threshold with a number no one outside the company has ever examined, because the tender asks for the threshold, not the method. New Delhi knows this: the government's own AI Governance Guidelines, released by MeitY last November, concede that officials often lack the technical grounding to evaluate AI procurements.

India's national AI conversation focuses on compute, foundation models, and skilling targets. All are necessary. None of these is the binding constraint on deployment quality. That constraint is evaluation capacity: shared test sets in Indian languages and contexts, disclosure norms for vendor scores, benchmarks that reflect how these systems are actually used, and institutions that can check a claim rather than admire it.

Those same guidelines already task the IndiaAI Safety Institute with testing and evaluating AI systems and developing standards, evaluation metrics and benchmarks. Procurement is the fastest way to make that commitment real: a single page of required evaluation disclosure in public tenders would change vendor behaviour faster than any regulation of models could.

A score is a claim. Infrastructure is the capacity to check it. Building that capacity is the next thing worth doing.

(The views expressed are personal)

This article is authored by Spurthi Tallam, senior software engineer - AI and ML, USA.