Gary Marcus

Rank 13 of 47
|
Score 83
5/18/2025, 12:29:23 AM
X
In reply to:
JB
@JasonBotterill3
·
241d
@GaryMarcus Even small pretraining gains seem to get hugely amplified by RL. The evals for each reasoning model (o1-preview, o1, o3) keep getting better.
Gary Marcus
@GaryMarcus
·
241d
@JasonBotterill3 We were when I wrote deep learning is hitting a wall.

My point was that we couldn’t maintain that for long.
JB
@JasonBotterill3
·
241d
@GaryMarcus Gary we are not relying on pure pretraining
Gary Marcus
@GaryMarcus
·
241d
!!!

If this true, it’s compelling evidence that pure pretraining scaling of LLMs has failed, and that that particular approach (which was I specifically said would hit a wall) has failed.
Flowers
@flowersslop
·
241d
I just got feedback from someone who def knows firsthand: GPT-5 will be built from entirely new models that haven’t been released yet. So we’re leaving GPT-4 and all its variants behind and getting a system made from components that aren’t part of anything currently available.

The statement is part of a technical discussion about the effectiveness and limitations of pretraining and reinforcement learning in AI models. It critiques the reliability of certain AI models across different domains, using chess as an example.

  1. Principle 1:
    I will strive to do no harm with my words and actions.
    The statement does not use harmful language and focuses on a technical critique, adhering to the principle of doing no harm. [+1]
  2. Principle 3:
    I will use my words and actions to promote understanding, empathy, and compassion.
    The statement aims to promote understanding by providing a counterpoint to the previous claim, though it could be more explanatory to enhance empathy and compassion. [+1]
  3. Principle 4:
    I will engage in constructive criticism and dialogue with those in disagreement and will not engage in personal attacks or ad hominem arguments.
    The statement engages in constructive criticism by addressing the technical aspects of the discussion without resorting to personal attacks. [+2]