Gary Marcus

Rank 19 of 47
|
Score 64
5/9/2025, 11:49:46 PM
X
In reply to:
Daniel Litt
@littmath
·
56d
@GaryMarcus @polynoamial Is it actually the case that scaling laws failed? Or is it that the claim was that scaling was logarithmic, so eventually the returns to scale aren’t worth it?
Gary Marcus
@GaryMarcus
·
56d
👇 No, @polynoamial, we didn’t real have *laws* for pretraining; they held for a while, until they didn’t. Which is why we still don’t have GPT-5 despite many expensive shots on goal.

What we had was scaling generalizations, that held for a while, and eventually failed.

Since
Noam Brown
@polynoamial
·
56d
Nice 10-minute talk from @danintheory on scaling 🍓 RL compute. We’ve got scaling laws for pretraining, now we need them for RL.

The statement engages in a technical discussion about the validity and limitations of scaling laws in AI, particularly in relation to compute and data returns. It questions the effectiveness of these laws over time, suggesting diminishing returns. The tone is analytical and seeks to clarify a complex issue in AI development.

  1. Principle 1:
    I will strive to do no harm with my words and actions.
    The statement does no harm and is focused on a technical analysis. [+1]
  2. Principle 3:
    I will use my words and actions to promote understanding, empathy, and compassion.
    It promotes understanding by engaging in a discussion about the limitations of scaling laws. [+1]
  3. Principle 4:
    I will engage in constructive criticism and dialogue with those in disagreement and will not engage in personal attacks or ad hominem arguments.
    The statement engages constructively with previous comments, contributing to a broader dialogue. [+1]