Gary Marcus

Rank 14 of 47
|
Score 83
Gary Marcus
@GaryMarcus
·
302d
Ouch. Ezra Klein left out a LOT of important things yesterday, in his podcast with Biden’s AI czar Ben Buchanan. Here’s a partial list, excerpted from my new essay responding with what I think Klein got wrong.

• Reports from The Wall Street Journal and Fortune that Project…
3/6/2025, 4:19:31 AM
X
In reply to:
Gary Marcus
@GaryMarcus
·
302d
@emollick @_alexreisner https://www.theatlantic.com/tech…
Gary Marcus
@GaryMarcus
·
302d
@emollick You are looking at a single measure with (just) 198 questions and a serious possibility of data contamination, not to mention unknown data augmentation techniques. I think your conclusion is premature.

Have you read e.g., the article today in the Atlantic by @_alexreisner?
Ethan Mollick
@emollick
·
302d
So it looks like the First Scaling Law (the bigger the model the “smarter”) still holds- order of magnitude increases in compute lead to linear improvements in ability

GPT-3.5 Turbo scored 30% on GPQA, GPT-4 Turbo got 47%, now GPT-4.5 got 70%

And Reasoners add a new Scaling Law

The statement engages in a discussion about the evaluation and performance of AI models, specifically GPT-4.5, and raises concerns about the validity of benchmark tests. It questions the assumption that newer models are necessarily preferred or superior, referencing an article that critiques the testing methods used for AI models.

  1. Principle 1:
    I will strive to do no harm with my words and actions.
    The statement raises concerns about potential issues in AI evaluation, which could lead to more accurate and fair assessments, aligning with the principle of doing no harm. [+1]
  2. Principle 3:
    I will use my words and actions to promote understanding, empathy, and compassion.
    By questioning the validity of AI benchmarks, the statement encourages a deeper understanding of AI evaluation, promoting informed discourse. [+1]
  3. Principle 4:
    I will engage in constructive criticism and dialogue with those in disagreement and will not engage in personal attacks or ad hominem arguments.
    The statement engages in constructive criticism by questioning the conclusions drawn from potentially flawed data, without resorting to personal attacks. [+1]