Gary Marcus

Rank 20 of 47
|
Score 60
5/10/2025, 1:43:05 AM
X
In reply to:
Daniel Litt
@littmath
·
29d
@GaryMarcus @Wasgo @polynoamial Why? IMO problems are very hard. To actually understand what they say about scaling you have to understand how they compare to other benchmarks.
Gary Marcus
@GaryMarcus
·
29d
@littmath @Wasgo @polynoamial It tells you that the improvements ar superficial rather than deep, despite massive scaling and massive data augmentation. And speaks to whether was is learned is general.
Daniel Litt
@littmath
·
29d
@GaryMarcus @Wasgo @polynoamial How does a one-time run on a new benchmark tell you anything about scaling?
Gary Marcus
@GaryMarcus
·
29d
@littmath @Wasgo @polynoamial Actually hard to evaluate because nobody afaik is publishing what they use for augmentation. But eg the failure on the Olympiad problems 6 hours after release hints in @wasgo’s favor.
Daniel Litt
@littmath
·
29d
@Wasgo @GaryMarcus @polynoamial IMO this is an overstatement, at least where math benchmarks are concerned, not to mention e.g. METR.
Cameron Williams
@Wasgo
·
29d
@littmath @GaryMarcus @polynoamial Right. But the problem isn’t the exact benchmark, it’s a question of whether or not we’re evaluating the model on novel problems, or ones within the dataset.

Never models are still improving on problems within the dataset (no wall) but not consistently on novel ones (wall).

The statement is part of a technical discussion on the performance of AI models on benchmarks, specifically addressing the issue of whether improvements are genuine or superficial. The conversation involves multiple participants discussing the validity and implications of benchmark results in AI research.

  1. Principle 1:
    I will strive to do no harm with my words and actions.
    The statement does not cause harm and is focused on a technical critique, adhering to the principle of doing no harm. [+1]
  2. Principle 3:
    I will use my words and actions to promote understanding, empathy, and compassion.
    The statement aims to clarify and promote understanding of the issue, aligning with the principle of promoting understanding and empathy. [+1]
  3. Principle 4:
    I will engage in constructive criticism and dialogue with those in disagreement and will not engage in personal attacks or ad hominem arguments.
    The statement engages in constructive dialogue without personal attacks, adhering to the principle of engaging in constructive criticism. [+1]