Gary Marcus

Rank 13 of 47
|
Score 83
5/17/2025, 11:30:49 PM
X
In reply to:
Daniel Litt
@littmath
·
239d
@GaryMarcus @colin_fraser @MelMitchell1 @rao2z I mean, I regularly ask them hard math problems that they reliably get correct; e.g. I wrote a thread about one of the FrontierMath problems of this form.
Gary Marcus
@GaryMarcus
·
239d
@littmath @colin_fraser @MelMitchell1 @rao2z What is it that you think they can solve very reliably?

I didn’t say all and would appreciate your not putting words in my mouth.
Daniel Litt
@littmath
·
239d
@GaryMarcus @colin_fraser @MelMitchell1 @rao2z Let me put it this way—they can very reliably solve some out-of-distribution hard problems that humans can’t reliably solve. Being able to solve *all* problems is not a pre-condition for intelligence; otherwise people aren’t intelligent.
Gary Marcus
@GaryMarcus
·
239d
@littmath @colin_fraser @MelMitchell1 @rao2z Some, but not reliably. They can’t even solve large multiplication problems reliably.

LLMs exceed their predecessors but demonstrably still have not solved this problem, which is most clear when the training set is known, and when direct data augmentation is not possible.

Some
Daniel Litt
@littmath
·
239d
@GaryMarcus @colin_fraser @MelMitchell1 @rao2z They’re clearly generalizing out of distribution. E.g. they can solve some novel math problems.
Gary Marcus
@GaryMarcus
·
239d
@littmath New situations is clearly the weakness in these approaches, and has been forever.

@colin_fraser has documented this endlessly, informally; I gave the intuitions behind that in 1998. The recent Apple reasoning paper, the studies of @MelMitchell1 and @rao2z etc reinforce it.

The

The statement is part of a public discourse on the capabilities and limitations of AI systems, specifically in solving complex math problems. It engages in a technical discussion about data leakage and the performance of AI in controlled environments, contributing to the broader debate on AI's reliability and intelligence.

  1. Principle 1:
    I will strive to do no harm with my words and actions.
    The statement does not appear to cause harm; it is a technical critique of AI systems. [+1]
  2. Principle 3:
    I will use my words and actions to promote understanding, empathy, and compassion.
    The statement promotes understanding by discussing the limitations of AI systems in a technical context. [+1]
  3. Principle 4:
    I will engage in constructive criticism and dialogue with those in disagreement and will not engage in personal attacks or ad hominem arguments.
    The statement engages in constructive criticism of AI systems without personal attacks. [+1]