xAI steals the show in Grok 3 benchmarks

This week, an OpenAI employee blamed Elon Musk's artificial intelligence company, xAI, for publishing misleading benchmark results for its latest AI model, Grok 3.

In a blog post on xAI , the company published a graph showing Grok 3’s performance on the AIME 2025, a collection of challenging math questions from a recent test of the model. Some experts question the validity of the AIME as an AI benchmark. However, the AIME 2025 and older versions of the test are commonly used to investigate a model’s mathematical prowess.

See more articles from iGuRu.gr when you search for news on Google.

iGuRu.gr preferred source

xAI’s graph shows two variants of Grok 3, Grok 3 Reasoning Beta and Grok 3 mini Reasoning, outperforming OpenAI ’s o3-mini-high model on AIME 2025. But OpenAI employees at X were quick to point out that xAI’s graph did not include a 2-AI score with 20-AI “cons@64.”

What is cons@64?

Well, it’s short for “consensus@64” and it gives a model 64 attempts to answer each problem in a benchmark and takes the answers produced as the final answers. As you can imagine, cons@64 tends to boost the benchmark scores of models quite a bit, and omitting it from a graph can make it look like one model is outperforming another, when in fact that’s not the case.

The scores of Grok 3 Reasoning Beta and Grok 3 mini Reasoning for AIME 2025 at “@1” – which is the score for the first attempt that the xAI models got in the benchmark – fall below the score of o3-mini-high. Grok 3 Reasoning Beta also trails only slightly behind OpenAI’s o1 model, which is defined as an “intermediate” computational model.

However, xAI advertises Grok 3 as the “ world’s smartest AI .”

Of course, as artificial intelligence researcher Nathan Lambert pointed out in a publication , for the most important metric:

The computational (and monetary) cost it took for each model to achieve its best score.

giorgos

George is still wondering what he is doing here….