Gary Marcus

Rank 13 of 47
|
Score 83
5/17/2025, 11:36:08 PM
X
In reply to:
Daniel Litt
@littmath
·
239d
@GaryMarcus @colin_fraser @MelMitchell1 @rao2z I also ask lots of other problems, including subtle alterations of the FrontierMath problems and totally unrelated ones. IMO this is wrong. (I also think the idea that they’re training on FrontierMath is very unlikely.)
Gary Marcus
@GaryMarcus
·
239d
@littmath @colin_fraser @MelMitchell1 @rao2z But OpenAI has very likely been augmenting around those problems - which they have privileged access to - and no system did well on the USAMO when data leakage was controlled.

(Also I am betting one could subtly alter those problems and get wild results.)
Daniel Litt
@littmath
·
239d
@GaryMarcus @colin_fraser @MelMitchell1 @rao2z I mean, I regularly ask them hard math problems that they reliably get correct; e.g. I wrote a thread about one of the FrontierMath problems of this form.
Gary Marcus
@GaryMarcus
·
239d
@littmath @colin_fraser @MelMitchell1 @rao2z What is it that you think they can solve very reliably?

I didn’t say all and would appreciate your not putting words in my mouth.
Daniel Litt
@littmath
·
239d
@GaryMarcus @colin_fraser @MelMitchell1 @rao2z Let me put it this way—they can very reliably solve some out-of-distribution hard problems that humans can’t reliably solve. Being able to solve *all* problems is not a pre-condition for intelligence; otherwise people aren’t intelligent.
Gary Marcus
@GaryMarcus
·
239d
@littmath @colin_fraser @MelMitchell1 @rao2z Some, but not reliably. They can’t even solve large multiplication problems reliably.

LLMs exceed their predecessors but demonstrably still have not solved this problem, which is most clear when the training set is known, and when direct data augmentation is not possible.

Some

The statement is part of a public discourse on the capabilities and limitations of AI models, specifically in solving complex mathematical problems. The conversation involves multiple participants discussing the performance of AI on standardized tests like the USAMO, and whether data augmentation affects results.

  1. Principle 1:
    I will strive to do no harm with my words and actions.
    The statement does not cause harm and is focused on discussing AI performance, which is a neutral topic.
  2. Principle 3:
    I will use my words and actions to promote understanding, empathy, and compassion.
    The statement seeks to promote understanding by questioning the results of AI performance on a specific test, encouraging further discussion and analysis. [+1]
  3. Principle 4:
    I will engage in constructive criticism and dialogue with those in disagreement and will not engage in personal attacks or ad hominem arguments.
    The statement engages in constructive dialogue by asking for an explanation, rather than making personal attacks or ad hominem arguments. [+1]