Gary Marcus

Rank 19 of 47
|
Score 64
Lee Gaul
@Leegaul
·
189d
o1 preview is %30 less accurate on Putnam-AXIOM math questions when it is challenged with truly novel problems, essentially proving over-fitting and data contamination are key "innovations" for reasoning models.

https://openreview.net/forum?id=…
1/7/2025, 3:54:34 AM
X
In reply to:
Gary Marcus
@GaryMarcus
·
184d
@bob_mankoff What you don’t know and are ignoring is how much training they specifically did around the task. We have reason to believe they did a lot of data augmentation geared at least roughly towards these specific tasks. The Putnam result I reported is consistent with that hypothesis.…
Robert Mankoff
@bob_mankoff
·
184d
@GaryMarcus Some context on this test showing o1 preview's achievement: The average human score on the William Lowell Putnam Mathematical Competition is typically often around 0 or 1 point out of a possible 120, meaning most participants do not solve even one problem completely correct on…
Gary Marcus
@GaryMarcus
·
185d
2. A recent paper on Putnam math problems showed a similar failure on distribution shift, showing 30% decement by o1 on math problems with minor variations eg on variable names.

Same old story, different set of problems.

https://openreview.net/forum?id=…
Gary Marcus
@GaryMarcus
·
185d
1. The problem of distribution shift, well-tested in Apple’s 2024 reasoning paper, which goes back to my own 1998 work, continues.

LLMs, like their predecessors generalize well to similar items but struggle with reliability in unfamiliar regimes, even minor variations sometimes…
Gary Marcus
@GaryMarcus
·
185d
An MIT professor just asked me, how can you be so confident that OpenAI isn’t very close to AGI?

Here is a thread of links to a several recent observations that I think show how far we are from robust intelligence.

Importantly, all of them are *known* problems, many that I…

The statement is part of a public discourse on the performance of large language models (LLMs) in mathematical reasoning tasks, specifically addressing the issue of distribution shift and data augmentation in AI training. The conversation involves multiple participants discussing the limitations and challenges faced by AI models in adapting to novel problems, which is a substantive engagement with public issues related to AI development and its implications.

  1. Principle 1:
    I will strive to do no harm with my words and actions.
    The statement does not appear to cause harm with words or actions, as it is focused on discussing technical aspects of AI performance. [+1]
  2. Principle 3:
    I will use my words and actions to promote understanding, empathy, and compassion.
    The statement contributes to understanding by providing context and insights into the challenges faced by AI models, promoting a deeper comprehension of the issue. [+2]
  3. Principle 4:
    I will engage in constructive criticism and dialogue with those in disagreement and will not engage in personal attacks or ad hominem arguments.
    The statement engages in constructive dialogue by addressing specific points about AI training and performance without resorting to personal attacks. [+2]
  4. Principle 7:
    I will uphold the principles of free speech and use my platform responsibly and with integrity.
    The statement upholds free speech principles by responsibly discussing AI-related issues, contributing to the broader discourse on technology and society. [+1]