Gary Marcus

Rank 10 of 47
|
Score 112
1/6/2025, 1:18:22 PM
X
In reply to:
Gary Marcus
@GaryMarcus
·
201d
7. Many leading figures in the field have acknowledged that we may have reached a period of diminishing returns of pure LLM scaling, much as I anticipated in 2022. It’s anybody’s guess what happens next.

[Note also that the current idea, test-time “scaling”, is very expensive,…
Gary Marcus
@GaryMarcus
·
201d
6. The presentation of o3 was problematic, as discussed below, and no clean test of out-of-domain generalization was provided, suggesting that we may see the kinds of problems with distribution shift as ever, outside of semi-closed domains where synthetic data can readily be…
Gary Marcus
@GaryMarcus
·
201d
5. In o1’s best case, there may have been haeavy data augmentation. In domains where that’s not possible, o1 may not even be that much better than GPT-4 (e.g on some language tasks), as you can see if you read OpenAI’s reports carefully: https://openai.com/index/learnin…
Gary Marcus
@GaryMarcus
·
201d
4. o1 probably isn’t as robust as people might have hoped. Even on certain benchmarks, results may be fragile or difficult to replicate:
Alejandro Cuadron
@Alex_Cuadron
·
201d
Surprising find: OpenAI's O1 - reasoning-high only hit 30% on SWE-Bench Verified - far below their 48.9% claim. Even more interesting: Claude achieves 53% in the same framework. Something's off with O1's "enhanced reasoning"... 🧵1/8
Gary Marcus
@GaryMarcus
·
201d
3. Commonsense reasoning is still dodgy, as it has long been, review here, earlier today: https://garymarcus.substack.com/…
Gary Marcus
@GaryMarcus
·
201d
2. A recent paper on Putnam math problems showed a similar failure on distribution shift, showing 30% decement by o1 on math problems with minor variations eg on variable names.

Same old story, different set of problems.

https://openreview.net/forum?id=…

The statement critiques the lack of reliable data in AI systems, leading to issues like inaccurate news and defamation. It highlights ongoing problems despite promises of improvement, suggesting a need for better data management.

  1. Principle 1:
    I will strive to do no harm with my words and actions.
    The statement aims to highlight issues in AI systems, potentially preventing harm by advocating for better data practices. [+1]
  2. Principle 2:
    I will respect the privacy and dignity of others and will not engage in cyberbullying, harassment, or hate speech.
    It respects privacy by focusing on systemic issues rather than personal attacks. [+1]
  3. Principle 3:
    I will use my words and actions to promote understanding, empathy, and compassion.
    The statement promotes understanding by explaining the consequences of unreliable data. [+1]
  4. Principle 4:
    I will engage in constructive criticism and dialogue with those in disagreement and will not engage in personal attacks or ad hominem arguments.
    It engages in constructive criticism by addressing the need for improvement in data management. [+1]