Gary Marcus

Rank 14 of 47

Score 83

@HumanoidRobotsX @emollick @_alexreisner ha, that’s a good one!

but i think it is really really hard to make good benchmarks. at some point Ernest Davis and I will explain many of the problems people have encountered. (Most people in the field realize benchmarks have been problematic.)

3/6/2025, 7:04:24 PM

In reply to:

HumanoidRobots.com

@HumanoidRobotsX

302d

@GaryMarcus @emollick @_alexreisner Sounds like someone needs to create http://benchmarcus.com/

Gary Marcus

@GaryMarcus

302d

@emollick You are looking at a single measure with (just) 198 questions and a serious possibility of data contamination, not to mention unknown data augmentation techniques. I think your conclusion is premature.

Have you read e.g., the article today in the Atlantic by @_alexreisner?

Ethan Mollick

@emollick

302d

So it looks like the First Scaling Law (the bigger the model the “smarter”) still holds- order of magnitude increases in compute lead to linear improvements in ability

GPT-3.5 Turbo scored 30% on GPQA, GPT-4 Turbo got 47%, now GPT-4.5 got 70%

And Reasoners add a new Scaling Law

The statement engages in a discussion about the challenges of creating effective benchmarks in AI, acknowledging existing problems and suggesting future clarification. It contributes to public discourse by addressing issues in AI evaluation.

Principle 1:
I will strive to do no harm with my words and actions.
The statement does no harm and acknowledges existing issues in AI benchmarks. [+1]
Principle 3:
I will use my words and actions to promote understanding, empathy, and compassion.
It promotes understanding by indicating a willingness to explain the problems with benchmarks. [+1]
Principle 4:
I will engage in constructive criticism and dialogue with those in disagreement and will not engage in personal attacks or ad hominem arguments.
The statement engages constructively by planning to address the issues without personal attacks. [+1]