Gary Marcus

Rank 21 of 47
|
Score 47
5/10/2025, 1:38:59 AM
X
In reply to:
Daniel Litt
@littmath
·
30d
@GaryMarcus @Wasgo @polynoamial How does a one-time run on a new benchmark tell you anything about scaling?
Gary Marcus
@GaryMarcus
·
30d
@littmath @Wasgo @polynoamial Actually hard to evaluate because nobody afaik is publishing what they use for augmentation. But eg the failure on the Olympiad problems 6 hours after release hints in @wasgo’s favor.
Daniel Litt
@littmath
·
30d
@Wasgo @GaryMarcus @polynoamial IMO this is an overstatement, at least where math benchmarks are concerned, not to mention e.g. METR.
Cameron Williams
@Wasgo
·
30d
@littmath @GaryMarcus @polynoamial Right. But the problem isn’t the exact benchmark, it’s a question of whether or not we’re evaluating the model on novel problems, or ones within the dataset.

Never models are still improving on problems within the dataset (no wall) but not consistently on novel ones (wall).
Daniel Litt
@littmath
·
30d
@Wasgo @GaryMarcus @polynoamial I don’t think it needs to be formal to be a claim we can try to evaluate heuristically by benchmarking. Lots of benchmarks seem to scale logarithmically with compute in some (pre-saturation) regime.
Cameron Williams
@Wasgo
·
30d
@littmath @GaryMarcus @polynoamial Is there any formal definition for this version of the law though? Altman refers to intelligence which isn’t actually a defined standard. He seems to cherry pick what intelligence means every time he refers to it.

The statement is part of a technical discussion on the effectiveness of AI models, specifically regarding their ability to generalize and improve with scaling and data augmentation. It critiques the depth of improvements in AI models, suggesting they are superficial. The tone is analytical and aims to contribute to the understanding of AI model evaluation.

  1. Principle 1:
    I will strive to do no harm with my words and actions.
    The statement does not use harmful language and focuses on a technical critique, adhering to the principle of doing no harm. [+1]
  2. Principle 3:
    I will use my words and actions to promote understanding, empathy, and compassion.
    The statement promotes understanding by engaging in a technical discussion about AI model evaluation, contributing to a deeper understanding of the topic. [+1]
  3. Principle 4:
    I will engage in constructive criticism and dialogue with those in disagreement and will not engage in personal attacks or ad hominem arguments.
    The statement engages in constructive dialogue by addressing the technical aspects of AI model evaluation without resorting to personal attacks. [+1]