Gary Marcus

Rank 19 of 47
|
Score 64
5/10/2025, 12:41:29 AM
X
In reply to:
Cameron Williams
@Wasgo
·
56d
@littmath @GaryMarcus @polynoamial There’s two entirely different meanings of AI scaling laws. OpenAI covers theirs in their papers and it’s not about model performance, it’s about fitting vs overfitting. https://arxiv.org/pdf/2001.08361

The general public talk about scaling laws as performance and there’s no definition.
Daniel Litt
@littmath
·
56d
@GaryMarcus @polynoamial I’m not sure I understand. Is the claim that scaling laws were predicting faster than logarithmic returns to scale, or that actual returns to scale are slower than logarithmic?
Gary Marcus
@GaryMarcus
·
56d
@littmath @polynoamial as i understand it the “chinchilla laws” no longer hold, and people no longer got the return for increases in compute and data that they were expecting per those alleged laws (but instead got diminishing though non zero returns).
Daniel Litt
@littmath
·
56d
@GaryMarcus @polynoamial Is it actually the case that scaling laws failed? Or is it that the claim was that scaling was logarithmic, so eventually the returns to scale aren’t worth it?
Gary Marcus
@GaryMarcus
·
56d
👇 No, @polynoamial, we didn’t real have *laws* for pretraining; they held for a while, until they didn’t. Which is why we still don’t have GPT-5 despite many expensive shots on goal.

What we had was scaling generalizations, that held for a while, and eventually failed.

Since
Noam Brown
@polynoamial
·
56d
Nice 10-minute talk from @danintheory on scaling 🍓 RL compute. We’ve got scaling laws for pretraining, now we need them for RL.

The statement is part of a technical discussion on AI scaling laws, specifically related to large language models (LLMs) and their performance. It contributes to public discourse by engaging in a substantive debate about AI research, a significant public issue. The tone is informative and aims to clarify misunderstandings about AI scaling laws.

  1. Principle 1:
    I will strive to do no harm with my words and actions.
    The statement does no harm and is focused on sharing information. [+1]
  2. Principle 3:
    I will use my words and actions to promote understanding, empathy, and compassion.
    The statement promotes understanding by providing a resource for further reading on the topic. [+2]
  3. Principle 4:
    I will engage in constructive criticism and dialogue with those in disagreement and will not engage in personal attacks or ad hominem arguments.
    The statement engages in constructive dialogue by referencing a review to clarify claims, without personal attacks. [+2]
  4. Principle 7:
    I will uphold the principles of free speech and use my platform responsibly and with integrity.
    The statement uses the platform responsibly to discuss a public issue in AI research. [+1]