Gary Marcus

Rank 19 of 47
|
Score 64
5/10/2025, 1:04:53 AM
X
In reply to:
Cameron Williams
@Wasgo
·
55d
@GaryMarcus @littmath @polynoamial And I completely agree that generative AI performance has hit a wall.

But people are conflating a performance wall with scaling law failing. Scaling law can hold and we can still hit a performance wall, like we have.
Gary Marcus
@GaryMarcus
·
55d
@Wasgo @littmath @polynoamial I understand what you’re saying esp around the Kaplan paper in particular.

But people didn’t spend half a trillion dollars on the pure mathematical endeavor; they thought the math and empirical observations around *performance* justified exponential increases in data and
Cameron Williams
@Wasgo
·
55d
@GaryMarcus @littmath @polynoamial Scaling law is a pure mathematical endeavour and performance isn’t pure math. They’re separate targets.
Cameron Williams
@Wasgo
·
55d
@GaryMarcus @littmath @polynoamial Even this post notes it’s not performance scaling:

“…test loss is not an easily interpretable metric and try to predict a variety of other performance metrics. For example, a scaling law is fit to predict the LLM’s pass rate on the HumanEval coding benchmark.”
Gary Marcus
@GaryMarcus
·
55d
@Wasgo @littmath @polynoamial This is perhaps a good review of the various claims that have been made

https://cameronrwolfe.substack.c…
Cameron Williams
@Wasgo
·
55d
@littmath @GaryMarcus @polynoamial There’s two entirely different meanings of AI scaling laws. OpenAI covers theirs in their papers and it’s not about model performance, it’s about fitting vs overfitting. https://arxiv.org/pdf/2001.08361

The general public talk about scaling laws as performance and there’s no definition.

The statement is part of a technical discussion on AI scaling laws and their implications for AI performance, engaging with public issues related to AI development and its societal impact. The tone is analytical and seeks to clarify misconceptions about AI scaling laws and performance.

  1. Principle 1:
    I will strive to do no harm with my words and actions.
    The statement does not use harmful language and maintains a neutral tone, adhering to the principle of doing no harm. [+1]
  2. Principle 3:
    I will use my words and actions to promote understanding, empathy, and compassion.
    The statement aims to promote understanding by clarifying the distinction between scaling laws and performance, aligning with the principle of promoting understanding and empathy. [+1]
  3. Principle 4:
    I will engage in constructive criticism and dialogue with those in disagreement and will not engage in personal attacks or ad hominem arguments.
    The statement engages in constructive dialogue by addressing misconceptions and providing clarification without resorting to personal attacks, adhering to the principle of constructive criticism. [+1]
  4. Principle 7:
    I will uphold the principles of free speech and use my platform responsibly and with integrity.
    The statement contributes to the public discourse on AI responsibly, upholding the principles of free speech and integrity. [+1]