Gary Marcus

Rank 19 of 47
|
Score 64
5/10/2025, 12:30:10 AM
X
In reply to:
Cameron Williams
@Wasgo
·
56d
@littmath @GaryMarcus @polynoamial OpenAI’s definition has a very specific meaning, and it’s not about actual performance so I don’t see how it could fail.

It only considers the accuracy of answers that are already known within the dataset and is more about accuracy vs overfitting.
Cameron Williams
@Wasgo
·
56d
@littmath @GaryMarcus @polynoamial There’s two entirely different meanings of AI scaling laws. OpenAI covers theirs in their papers and it’s not about model performance, it’s about fitting vs overfitting. https://arxiv.org/pdf/2001.08361

The general public talk about scaling laws as performance and there’s no definition.
Daniel Litt
@littmath
·
56d
@GaryMarcus @polynoamial I’m not sure I understand. Is the claim that scaling laws were predicting faster than logarithmic returns to scale, or that actual returns to scale are slower than logarithmic?
Gary Marcus
@GaryMarcus
·
56d
@littmath @polynoamial as i understand it the “chinchilla laws” no longer hold, and people no longer got the return for increases in compute and data that they were expecting per those alleged laws (but instead got diminishing though non zero returns).
Daniel Litt
@littmath
·
56d
@GaryMarcus @polynoamial Is it actually the case that scaling laws failed? Or is it that the claim was that scaling was logarithmic, so eventually the returns to scale aren’t worth it?
Gary Marcus
@GaryMarcus
·
56d
👇 No, @polynoamial, we didn’t real have *laws* for pretraining; they held for a while, until they didn’t. Which is why we still don’t have GPT-5 despite many expensive shots on goal.

What we had was scaling generalizations, that held for a while, and eventually failed.

Since

The statement engages in a technical discussion about AI scaling laws, specifically addressing the definitions and interpretations of these laws within the AI research community. It critiques the perceived redefinition of these laws and emphasizes the original intent as outlined by Kaplan et al. This constitutes public discourse as it engages with a substantive issue in AI research, a field with significant societal implications.

  1. Principle 1:
    I will strive to do no harm with my words and actions.
    The statement uses strong language ('bullshit') which could be seen as harmful or disrespectful, violating the principle of doing no harm with words. [-1]
  2. Principle 2:
    I will respect the privacy and dignity of others and will not engage in cyberbullying, harassment, or hate speech.
    The statement does not engage in cyberbullying, harassment, or hate speech, but the use of strong language could be seen as disrespectful to the dignity of others. [-1]
  3. Principle 3:
    I will use my words and actions to promote understanding, empathy, and compassion.
    The statement does not promote understanding, empathy, or compassion, as it focuses on critiquing definitions rather than fostering dialogue. [-1]
  4. Principle 4:
    I will engage in constructive criticism and dialogue with those in disagreement and will not engage in personal attacks or ad hominem arguments.
    The statement does not engage in personal attacks but uses dismissive language ('bullshit') which does not contribute to constructive criticism or dialogue. [-1]