Gary Marcus

Rank 19 of 47
|
Score 64
Gary Marcus
@GaryMarcus
·
202d
Three o3 predictions:
– People will initially be amazed
- Once they dive in, they will see that it is not reliable
- It will work best in closed-domains (like math problems) but less reliably in open-ended domains (like everyday reasoning about the real world).
1/6/2025, 11:52:46 PM
X
In reply to:
Xingyao Wang
@xingyaow_
·
186d
I have a theory: the amount of information provided in the context differs significantly for these two types of agent scaffolds.
And this causes the reasoning model like o1 to perform differently.

Reasoning models are trained to THINK hard, e.g., by solving extremely…
Xingyao Wang
@xingyaow_
·
186d
In contrast, CodeAct in OpenHands doesn't have these human-crafted workflows—instead, it just tells the model, "Here's a file editor, here's a bash terminal. You can write & execute arbitrary code; feel free to do anything you want; GO!"

and reasoning model like o1 is pretty…
Xingyao Wang
@xingyaow_
·
186d
Reasoning models like o1 did pretty well on SWE-Bench IF they use a workflow-based scaffold, like Agentless -- the scaffold itself already decomposes the problem for the model.

In this case, the challenge of solving Github issue is decomposed into multiple different prompts…
Xingyao Wang
@xingyaow_
·
186d
I often get asked this question: Why is o1 not so good on OpenHands, but their official report shows a decent SWE-bench number? 🤔
🧵
Alejandro Cuadron
@Alex_Cuadron
·
186d
Surprising find: OpenAI's O1 - reasoning-high only hit 30% on SWE-Bench Verified - far below their 48.9% claim. Even more interesting: Claude achieves 53% in the same framework. Something's off with O1's "enhanced reasoning"... 🧵1/8

The statement is part of a technical discussion about the performance of reasoning models in different contexts. It involves predictions and analysis of model behavior, contributing to a broader conversation about AI capabilities and limitations.

  1. Principle 1:
    I will strive to do no harm with my words and actions.
    The statement does not cause harm and is focused on technical analysis. [+1]
  2. Principle 3:
    I will use my words and actions to promote understanding, empathy, and compassion.
    The statement promotes understanding by discussing model performance and predictions. [+1]
  3. Principle 4:
    I will engage in constructive criticism and dialogue with those in disagreement and will not engage in personal attacks or ad hominem arguments.
    The statement engages in constructive dialogue by building on previous points and providing predictions. [+1]