Gary Marcus

Rank 19 of 47
|
Score 64
5/10/2025, 12:44:24 AM
X
In reply to:
Cameron Williams
@Wasgo
·
55d
@GaryMarcus @littmath @polynoamial Key quote from their abstract:

“These relationships allow us to determine the optimal allocation of a fixed compute budget.”

It’s about optimal allocation. Not performance between models.
Cameron Williams
@Wasgo
·
55d
@GaryMarcus @littmath @polynoamial You can look at their paper. They’re not talking about overall performance when they talk scaling. The whole paper is about optimal scaling.

In the video, it’s the same thing, which is why they’re looking at scaling within a specific model, not between models.
Gary Marcus
@GaryMarcus
·
55d
@Wasgo @littmath @polynoamial What is their definition? The original scaling laws were from them, in the paper by Kaplan et al.

The whole field presumed that performance could be scaled; everything else is bullshit and redefinition.
Cameron Williams
@Wasgo
·
55d
@littmath @GaryMarcus @polynoamial OpenAI’s definition has a very specific meaning, and it’s not about actual performance so I don’t see how it could fail.

It only considers the accuracy of answers that are already known within the dataset and is more about accuracy vs overfitting.
Cameron Williams
@Wasgo
·
55d
@littmath @GaryMarcus @polynoamial There’s two entirely different meanings of AI scaling laws. OpenAI covers theirs in their papers and it’s not about model performance, it’s about fitting vs overfitting. https://arxiv.org/pdf/2001.08361

The general public talk about scaling laws as performance and there’s no definition.
Daniel Litt
@littmath
·
55d
@GaryMarcus @polynoamial I’m not sure I understand. Is the claim that scaling laws were predicting faster than logarithmic returns to scale, or that actual returns to scale are slower than logarithmic?

The statement is part of a technical discussion on AI scaling laws, specifically addressing the interpretation of a 2020 paper's findings on model size, dataset size, and compute usage. It engages in a detailed debate about the optimal allocation of resources in AI models, which is a substantive public issue in the field of AI research.

  1. Principle 1:
    I will strive to do no harm with my words and actions.
    The statement does not cause harm and is focused on clarifying technical details, adhering to the principle of doing no harm. [+1]
  2. Principle 3:
    I will use my words and actions to promote understanding, empathy, and compassion.
    The statement aims to promote understanding by clarifying the technical aspects of the paper, aligning with the principle of promoting understanding and empathy. [+1]
  3. Principle 4:
    I will engage in constructive criticism and dialogue with those in disagreement and will not engage in personal attacks or ad hominem arguments.
    The statement engages in constructive dialogue by addressing specific points in the conversation without resorting to personal attacks, adhering to the principle of constructive criticism. [+1]