Gary Marcus

Rank 18 of 47
|
Score 64
5/10/2025, 1:28:56 AM
X
In reply to:
Cameron Williams
@Wasgo
·
54d
@littmath @GaryMarcus @polynoamial Right. But the problem isn’t the exact benchmark, it’s a question of whether or not we’re evaluating the model on novel problems, or ones within the dataset.

Never models are still improving on problems within the dataset (no wall) but not consistently on novel ones (wall).
Daniel Litt
@littmath
·
54d
@Wasgo @GaryMarcus @polynoamial I don’t think it needs to be formal to be a claim we can try to evaluate heuristically by benchmarking. Lots of benchmarks seem to scale logarithmically with compute in some (pre-saturation) regime.
Cameron Williams
@Wasgo
·
54d
@littmath @GaryMarcus @polynoamial Is there any formal definition for this version of the law though? Altman refers to intelligence which isn’t actually a defined standard. He seems to cherry pick what intelligence means every time he refers to it.
Daniel Litt
@littmath
·
54d
@GaryMarcus @Wasgo @polynoamial This is what I understood by scaling laws; it’s obviously compatible with diminishing returns to scale.
Gary Marcus
@GaryMarcus
·
54d
@Wasgo @littmath @polynoamial How about this one? Source: https://blog.samaltman.com/three…
Cameron Williams
@Wasgo
·
54d
@GaryMarcus @littmath @polynoamial Even this post notes it’s not performance scaling:

“…test loss is not an easily interpretable metric and try to predict a variety of other performance metrics. For example, a scaling law is fit to predict the LLM’s pass rate on the HumanEval coding benchmark.”

The statement is part of a technical discussion about AI model evaluation and data augmentation. It engages with the topic of AI performance and benchmarking, contributing to the broader discourse on AI development and evaluation methods.

  1. Principle 1:
    I will strive to do no harm with my words and actions.
    The statement is neutral and does not cause harm. It contributes to a technical discussion.
  2. Principle 3:
    I will use my words and actions to promote understanding, empathy, and compassion.
    The statement promotes understanding by engaging in a discussion about data augmentation and its impact on AI model performance. [+1]
  3. Principle 4:
    I will engage in constructive criticism and dialogue with those in disagreement and will not engage in personal attacks or ad hominem arguments.
    The statement engages constructively with others in the conversation, focusing on the topic rather than personal attacks. [+1]