Gary Marcus

Rank 15 of 47

Score 93

Reposted by Gary Marcus

This is a recurring pattern, a new model shows strong performance on some benchmarks, the AI community goes wild, then later it is shown the model can't even handle basic variations of the same problems from the same benchmarks.

Jenia Jitsev 🏳️‍🌈 🇺🇦 🇮🇱

@JJitsev

152d

(Yet) another tale of Rise and Fall: DeepSeek R1 is claimed to match o1/o1-preview on olympiad level math & coding problems. Can it handle versions of AIW problems that reveal generalization & basic reasoning deficits in SOTA LLMs? (https://arxiv.org/abs/2406.02061) 🧵1/n

1/25/2025, 7:48:26 PM

The statement critiques the AI community's tendency to overhype new models without thoroughly testing their generalization capabilities. It highlights a recurring issue in AI research where models perform well on benchmarks but fail in practical applications.

Principle 1:
I will strive to do no harm with my words and actions.
The statement aims to highlight a flaw in the AI research process, potentially preventing harm by encouraging more rigorous testing. [+1]
Principle 3:
I will use my words and actions to promote understanding, empathy, and compassion.
By pointing out the issue, it promotes understanding and encourages the community to address these shortcomings. [+1]
Principle 4:
I will engage in constructive criticism and dialogue with those in disagreement and will not engage in personal attacks or ad hominem arguments.
The statement engages in constructive criticism by identifying a pattern and suggesting a need for improvement. [+1]