Gary Marcus

Rank 9 of 47
|
Score 121
1/6/2025, 4:44:46 AM
X
In reply to:
Gary Marcus
@GaryMarcus
·
205d
4. o1 probably isn’t as robust as people might have hoped. Even on certain benchmarks, results may be fragile or difficult to replicate:
Alejandro Cuadron
@Alex_Cuadron
·
206d
Surprising find: OpenAI's O1 - reasoning-high only hit 30% on SWE-Bench Verified - far below their 48.9% claim. Even more interesting: Claude achieves 53% in the same framework. Something's off with O1's "enhanced reasoning"... 🧵1/8
Gary Marcus
@GaryMarcus
·
205d
3. Commonsense reasoning is still dodgy, as it has long been, review here, earlier today: https://garymarcus.substack.com/…
Gary Marcus
@GaryMarcus
·
205d
2. A recent paper on Putnam math problems showed a similar failure on distribution shift, showing 30% decement by o1 on math problems with minor variations eg on variable names.

Same old story, different set of problems.

https://openreview.net/forum?id=…
Gary Marcus
@GaryMarcus
·
205d
1. The problem of distribution shift, well-tested in Apple’s 2024 reasoning paper, which goes back to my own 1998 work, continues.

LLMs, like their predecessors generalize well to similar items but struggle with reliability in unfamiliar regimes, even minor variations sometimes…
Gary Marcus
@GaryMarcus
·
205d
An MIT professor just asked me, how can you be so confident that OpenAI isn’t very close to AGI?

Here is a thread of links to a several recent observations that I think show how far we are from robust intelligence.

Importantly, all of them are *known* problems, many that I…

The statement engages in a technical discussion about the performance and limitations of OpenAI's model, o1, in comparison to GPT-4 and other models. It critiques the robustness and generalization capabilities of o1, particularly in the context of data augmentation and specific language tasks. The statement is part of a broader conversation about the challenges faced by AI models in achieving robust intelligence.

  1. Principle 1:
    I will strive to do no harm with my words and actions.
    The statement does not use harmful language and focuses on technical critique, adhering to the principle of doing no harm. [+1]
  2. Principle 3:
    I will use my words and actions to promote understanding, empathy, and compassion.
    The statement promotes understanding by providing a detailed analysis of the model's performance, contributing to a nuanced discussion about AI capabilities. [+2]
  3. Principle 4:
    I will engage in constructive criticism and dialogue with those in disagreement and will not engage in personal attacks or ad hominem arguments.
    The statement engages in constructive criticism by addressing specific issues with the model's performance without resorting to personal attacks. [+2]