Gary Marcus

Rank 14 of 47
|
Score 83
3/6/2025, 3:26:55 AM
X
In reply to:
Gary Marcus
@GaryMarcus
·
302d
@emollick You are looking at a single measure with (just) 198 questions and a serious possibility of data contamination, not to mention unknown data augmentation techniques. I think your conclusion is premature.

Have you read e.g., the article today in the Atlantic by @_alexreisner?
Ethan Mollick
@emollick
·
302d
So it looks like the First Scaling Law (the bigger the model the “smarter”) still holds- order of magnitude increases in compute lead to linear improvements in ability

GPT-3.5 Turbo scored 30% on GPQA, GPT-4 Turbo got 47%, now GPT-4.5 got 70%

And Reasoners add a new Scaling Law

The statement and linked article engage in public discourse by addressing the integrity of AI benchmark tests, a significant issue in technology and AI development. The conversation critiques the validity of AI performance metrics, suggesting potential data contamination and questioning the conclusions drawn from these tests.

  1. Principle 1:
    I will strive to do no harm with my words and actions.
    The statement raises concerns about the potential harm of misleading AI performance metrics, aligning with the principle of striving to do no harm. [+1]
  2. Principle 3:
    I will use my words and actions to promote understanding, empathy, and compassion.
    By questioning the validity of AI benchmarks, the statement promotes understanding and critical evaluation of AI technologies. [+1]
  3. Principle 4:
    I will engage in constructive criticism and dialogue with those in disagreement and will not engage in personal attacks or ad hominem arguments.
    The statement engages in constructive criticism by challenging the conclusions drawn from AI performance data without resorting to personal attacks. [+1]