Benchmarking: The Race to Evaluate Artificial Intelligence

Give me a fixed point and I will move the world. - Archimedes

There's an arms race at the heart of artificial intelligence development in general, but I'm going to focus on AI applied to language in particular. Developing an AI system means trying to achieve a set of capabilities that are, in principle, cognitive, human, or useful to humans. Whether that's writing text, comprehension, translation, summarizing, being polite, being fair, producing intelligence reports, declaring that we've always been at war with Eurasia, deceiving, keeping files on political opponents, winning thermonuclear wars, etc. For any of those tasks, there has to be a way to evaluate whether the system does it well or badly, and, in particular, how it compares to other systems designed with the same goal.

This is a problem that affects both the people who design (or grow) the models and the people who'd like to hire or use them. Those who build artificial intelligence systems need measurable targets to aim at; those who want to hire models or are thinking of using them want to know which one is best. But how do you do that? Well, one way is to go by vibes, which is the sophisticated version of what everybody does when they use a language model. You use it, you give it a problem you think it won't be able to solve, it solves it, your jaw hits the floor and you go "damn, you have no idea." But as many of us have noticed, that could just be luck, or precisely the kind of problem the model was trained for, and the fact that it did that well doesn't guarantee it can do other things well.

Those who build artificial intelligence systems need measurable targets to aim at; those who want to hire models or are thinking of using them want to know which one is best. But how do you do that?

A more sophisticated way to evaluate language model performance is to have experts assess them directly on their skills/knowledge. This works a bit better, but it's much more expensive, slow, and still fallible. Another way is to make the models compete against each other on vibes (like they do at chatbot arena, with ELO rankings): they give you two models without telling you which is which and you say which one you like better. However, this has basically the same problems as eyeballing it: it rests on mere opinions. It gives you a lot, but not enough. A fourth way is to take a model and crack it open, like dissecting a brain, and see what happens. These are "white box" analyses (the opposite of a black box: a box you can look inside of) which are either barely informative for certain models, or too slow and impractical. Not to mention that neither OpenAI nor Anthropic is going to give you access to their goose that lays the golden tokens. That leaves giving them standardized tests.

Benchmarks, or standardized tests for language models

The word "benchmark" has many meanings, and it sounds like we're talking about some kind of "test bench" or something like that, but that's not where it comes from. Etymology isn't always relevant, but in this case it can help us understand the issue. The English word "benchmark" comes from geodesy, the discipline concerned with measuring the earth, which feeds the work of topographers and land surveyors. Cities have different elevations, which means you need to account for the relative height of different structures, for example when laying out pipes and things like that, to account for how water drains through a city.

To that end, authorities in modern cities (and by modern I mean roughly from the 18th century on) took care to mark geodetic "fixed points" ("puntos fijos," which is what we call geodetic benchmarks in Spanish) in different parts of the city. Cities have a fixed reference point, or point zero, which in the Autonomous City of Buenos Aires is located at the entrance of the Cathedral. It's a compass rose. All of the capital's geodetic benchmarks are marked at a height relative to that point zero.

The city's fundamental benchmark, the one at the Cathedral, looks like this:

But there are benchmarks all over the country, placed by the Instituto Geográfico Nacional (successor to the Instituto Geográfico Militar):

This gives land surveyors, topographers, cartographers, architects, and civil engineers relative reference points for their work, mapped to an elevation and a geographic position. Nowadays a lot of this is done with GPS, but no country is dumb enough (yet, at least) to rely entirely on a satellite system for this crucial information. Nothing beats a chunk of iron embedded in a wall and a whole lot of spreadsheets (low tech, high life, as Ruocco would say).

Benchmarks in computing

As an extrapolation from the original use of "benchmarking" (making horizontal marks in stone or granite on which to rest a level and form a bench or seat for a leveling rod), various fields use the term to refer to a (relative) performance test against which to compare several things of the same kind. In computing, we gamers use them to run performance tests of our systems in certain games or reference tests that give us a rough idea of how good our machine is compared to others.

Benchmarks in computational linguistics and linguistic AI (LLMs)

A common practice in machine learning is to take a dataset (a set of data) to train the model on and split it into 80% for training and 20% for evaluation. If the model trained on the 80%, without ever seeing the evaluation data, performs well at predicting the data in the evaluation set, you can say the model generalized from the data it saw in the training set and in some sense "learned." Otherwise, it didn't. A model that can only solve problems it has already seen is useless.

In practice, we can think of it like this: in the training dataset we have a bunch of geography, algebra, or calculus problems, a set of possible answers, and the correct answer (they could be grammar problems, or questions about Magic cards). In the evaluation dataset we have problems that didn't show up in training, and we don't show the model the answer; we keep it to ourselves to grade it. We give it a set of options (like a multiple choice test) and see whether it gets it right. The hit rate gives us its precision, or accuracy, on the dataset, which we can treat as a benchmark.

In this sense, an evaluation dataset (whether or not it's part of a training dataset) that can be administered to many language models plays the role of a benchmark.

Construct validity and psychometrics

Now, your machine getting a certain score on a graphics benchmark isn't important in itself, but as a sign of what your machine can do when you use it for gaming, or rendering videos, or whatever it is you want to do (which isn't the benchmark). Something similar happens with language models. You don't care whether a language model is good at the benchmark because that means something in itself; you care that it does well if the fact that it does well tells you something about what it can do that isn't the benchmark.

You don't care whether a language model is good at the benchmark because that means something in itself; you care that it does well if the fact that it does well tells you something about what it can do that isn't the benchmark.

This is very similar to what we teachers do when we give an exam. Assessment instruments don't matter in themselves. A lot of high school students (and plenty of college students) sometimes operate under the (mistaken) idea that the assignments their teachers give them, or the essays they write, are some kind of commission the teacher is going to get something out of reading, as if the teacher were going to learn something. As if the assignment were some kind of report a company commissions from a consulting firm. I'm not saying we teachers can't learn from our students (we do it all the time), but that's not why we put them through assessments.

One of the roles of assessment is diagnosis (there are others, like a formative role: I want you to rack your brain so you gradually build up the target skill involved): I want to know how you're doing so I can eventually get some idea of how you'd fare solving similar problems in your professional life (at the university level, for example). In other words: exams for humans are often instruments that give us reasons to attribute to a person a competence, understood as a disposition to solve similar problems. A sort of sophisticated cousin of meeting someone, catching them lying to us several times, and taking that as reason enough to attribute to them the property of being a liar. Getting labeled a liar is dangerous (like in the story of the boy who cried wolf) because people treat you as if you're going to keep lying, and if they treat you that way, they stop taking you seriously (it's a particular kind of social death). On the other hand, getting labeled a lawyer means you beat every professor in the program in a round (like Ash against the Pokémon masters), and you're supposed to be able to keep doing it in the future.

Well, just as a driving test isn't merely a permit, but a permit based on an attribution of competence and the responsibility that comes with it (if you're capable of driving well, you're responsible for driving badly), the same goes for more abstract competences. Cognitive abilities, for example. Psychometricians have been trying for a long time to design ways to assess more abstract competences, including the competence of competences, the Holy Grail of human skill permits: intelligence.

In the words of artificial intelligence specialist François Chollet (creator of the ARC-AGI benchmark), your ability to solve a task speaks to how skilled you are, while intelligence measures how efficient (fast, say) you are at acquiring new abilities to solve new tasks. That is, when we say someone is intelligent, we're not saying they're capable of doing anything in particular, just that, if they had the desire, the opportunity, and the time, it would take them less time or effort than other people to become proficient at the skill in question (which is how we can square the fact that there are so many intelligent people who are good for nothing, in their own eyes or in others').

So intelligence can't be measured directly, because a lot of your competence may be what psychometricians call "crystallized intelligence" (or crystallized g factor, Gc) as opposed to "fluid intelligence" (or Gf). Psychologists who specialize in measuring cognitive abilities don't care much about crystallized intelligence, because it's possible to know a lot about something and still be an idiot (their words, don't blame me). The "idiot savant" is the figure who remembers facts but can't reason beyond the facts they've memorized. Maybe Borges's "Funes the Memorious" illustrates this better than anything.

Sidebar: IQ tests

These tests aren't directly measuring your intelligence (that doesn't seem to be possible), they're measuring which average score for a given age your score resembles. If you get a score similar to that of people older than you, it would mean you're more efficient at acquiring cognitive skills than people your age. The older the age whose average score yours matches, the higher the quotient: (MentalAge/Age)*100 = IQ.

A word of warning to the twenty-somethings who brag about having a high IQ: it means that, with a lot of luck, you score like the average 30-year-old (an 18-year-old with a mental age of 36 would have an IQ of 200, the measurable limit; 130 points means being 18 and taking IQ tests like a 24-year-old...), the very people you love to trash. Having a high IQ at 30 as a guy just means you get to have a crisis about not knowing what you're doing with your life, get dumped by your partner, and buy a Harley Davidson (or a PlayStation to play GTA VI if you're not a trust-fund kid) before you hit 40. Congratulations!

Only the dead have seen the end of stupid intergenerational wars. The (discourse around) IQ is just another one of their battlefields. In any case, let's remember that the only reason it sounds cool to have the same IQ as Newton (good luck with that) is that Newton did something interesting with his life. I'd focus on that.

The same question can be asked about LLMs: are they intelligent, or did they memorize data? Ted Chiang (the hugely famous science fiction writer), for instance, says they're merely a kind of data compression (like a .zip or .rar). A lossy kind of compression (lossy compression): he used the image of LLMs as a blurry .jpeg of the internet. So how do we evaluate language models that slurped down a smoothie of all the written data humanity has ever digitized, in order to argue that they're not just regurgitating and that they can reach true understanding? That's the quadrillion-dollar question, and the benchmark is us.

The finish line of the future: AGI

Artificial General Intelligence is the counterpart to our natural general intelligence. So we have to think about how we can claim that we're intelligent, what we're capable of doing, what actually makes us attribute intelligence to one another, and then try to extrapolate those ideas to language models. If we can design evaluations that predict good general performance on any cognitive task, we'll have reasons to believe they're intelligent. That means designing a theoretical construct called "general intelligence," a model of what we think intelligence is, and putting together tasks (tests) to use as benchmarks for language models, such that if LLMs solved them, we'd have real reasons to say they have the thing we assume we're evaluating (the theoretical construct). The kind of thing where we'd say "if it can solve that, I'll hand over my finances" (or my nukes, if you're the US or China).

How do we evaluate language models that slurped down a smoothie of all the written data humanity has ever digitized, in order to argue that they're not just regurgitating? That's the quadrillion-dollar question, and the benchmark is us.

But it's not that easy. These days, language model design needs benchmarks to know which research directions to pursue. Inventing new language model architectures doesn't have much to hold on to if we can't tell whether those new architectures are better than the old ones. It's not enough for them to seem better. Worse still, we can take for granted that every benchmark made public will show up in the training data of future models, so the information we get from a model solving it is limited: once we publish a benchmark (the idea is for it to be repeatable and fair, and one way to do that is to make it public), Anthropic or the folks at Alibaba can train the model on the benchmark, making sure their model aces the benchmark without necessarily being good at the skill the benchmark was designed to measure.

The industry and the public need better evaluations, but it's a thankless, expensive and difficult job, trust me. Nobody likes teachers when we give exams, and it's not that different with language model evaluations. There's a lot more money in designing these machines than in evaluating them. And that's just the problem tied to something we have a reference for: our own abilities, even if we don't know how to measure them well in people or in machines. The big question is: what would the benchmark (or benchmarks) for Silicon Valley's God-in-the-machine look like? How would we know we'd achieved artificial superintelligence? Maybe what Gibson hints at in some of his novels, or what happens in Spike Jonze's Her, would happen: we wouldn't notice. They'd pass us right by, and we'd have lost cognitive dominion over the world without even knowing it. Or maybe it's not such a good question after all.