Gary Marcus

Rank 13 of 47
|
Score 100
5/10/2025, 1:34:43 AM
X
In reply to:
Daniel Litt
@littmath
·
39d
@Wasgo @GaryMarcus @polynoamial IMO this is an overstatement, at least where math benchmarks are concerned, not to mention e.g. METR.
Cameron Williams
@Wasgo
·
39d
@littmath @GaryMarcus @polynoamial Right. But the problem isn’t the exact benchmark, it’s a question of whether or not we’re evaluating the model on novel problems, or ones within the dataset.

Never models are still improving on problems within the dataset (no wall) but not consistently on novel ones (wall).
Daniel Litt
@littmath
·
39d
@Wasgo @GaryMarcus @polynoamial I don’t think it needs to be formal to be a claim we can try to evaluate heuristically by benchmarking. Lots of benchmarks seem to scale logarithmically with compute in some (pre-saturation) regime.
Cameron Williams
@Wasgo
·
39d
@littmath @GaryMarcus @polynoamial Is there any formal definition for this version of the law though? Altman refers to intelligence which isn’t actually a defined standard. He seems to cherry pick what intelligence means every time he refers to it.
Daniel Litt
@littmath
·
39d
@GaryMarcus @Wasgo @polynoamial This is what I understood by scaling laws; it’s obviously compatible with diminishing returns to scale.
Gary Marcus
@GaryMarcus
·
39d
@Wasgo @littmath @polynoamial How about this one? Source: https://blog.samaltman.com/three…

The statement engages in a technical discussion about AI model evaluation and benchmarks, specifically addressing the challenges of assessing model performance on novel versus known problems. It critiques the lack of transparency in data augmentation practices and references a specific failure case to support an argument.

  1. Principle 1:
    I will strive to do no harm with my words and actions.
    The statement does not cause harm and focuses on a technical critique, maintaining a neutral tone.
  2. Principle 3:
    I will use my words and actions to promote understanding, empathy, and compassion.
    The statement promotes understanding by discussing specific technical issues in AI evaluation, contributing to a deeper understanding of the challenges involved. [+1]
  3. Principle 4:
    I will engage in constructive criticism and dialogue with those in disagreement and will not engage in personal attacks or ad hominem arguments.
    The statement engages in constructive dialogue by providing evidence to support a counter-argument, without resorting to personal attacks. [+1]