Gary Marcus

Rank 17 of 47
|
Score 71
6/18/2025, 3:20:38 PM
X
In reply to:
Luke Frymire
@lukefrymire
·
2d
@GaryMarcus I think you're misreading the result Gary. o4-mini-high still beats 98.5% of humans, and this isn't a like-for-like comparison – human participants get to use their local environment + web access, get to compile/run/debug locally. That's a huge advantage (apparently ~400 Elo).
Gary Marcus
@GaryMarcus
·
3d
BREAKING NEWS: Expert coders absolutely crush LLMs on hard new benchmark, once data contamination is controlled.

The less LLMs can rely on memorization and similarity, the less well they do.

We see that same pattern again and again; nitpicking re: the Apple paper doesn’t
Rohan Paul
@rohanpaul_ai
·
3d
This is really BAD news of LLM's coding skill. ☹️

The best Frontier LLM models achieve 0% on hard real-life Programming Contest problems, domains where expert humans still excel.

LiveCodeBench Pro, a benchmark composed of
problems from Codeforces, ICPC, and IOI (“International

The statement engages in a technical discussion about the performance of LLMs versus human coders, questioning the fairness of the comparison by pointing out the execution time advantages of machines. It challenges the one-sided presentation of asymmetries in the original argument.

  1. Principle 1:
    I will strive to do no harm with my words and actions.
    The statement does not cause harm and contributes to a technical debate. [+1]
  2. Principle 3:
    I will use my words and actions to promote understanding, empathy, and compassion.
    It promotes understanding by highlighting an overlooked aspect of the comparison. [+1]
  3. Principle 4:
    I will engage in constructive criticism and dialogue with those in disagreement and will not engage in personal attacks or ad hominem arguments.
    Engages constructively by questioning the fairness of the argument without personal attacks. [+1]