Gary Marcus

Rank 13 of 47
|
Score 83
4/30/2025, 1:17:52 AM
X
In reply to:
MindMirror
@TheMysteryDrop
·
257d
@GaryMarcus @samuelaxon Fair, but I don't think the claim is that all AI generated code will be better than top SWEs - there may still be holes and a somewhat jagged frontier - but that for a majority of tasks AI generated code will be better.

This seems reasonable given the trajectory of progress.
Gary Marcus
@GaryMarcus
·
257d
@TheMysteryDrop @samuelaxon he specifically claimed they would be better at debugging than top engineers in 12-18 months in the post.

i really don’t see that happening that fast.
MindMirror
@TheMysteryDrop
·
257d
@GaryMarcus @samuelaxon How good do you think the average SWE is at this benchmark? AI doesn't need to hit 100% to soak up >50%.

Note that Zuck isn't claiming 100% of software will be AI generated - just the majority - and that there will be a role for top SWEs to cover the holes for the time being.
Gary Marcus
@GaryMarcus
·
257d
@TheMysteryDrop @samuelaxon driverless cars were (I imagine) greater than 80% correct a decade ago; the last bit is always hard.
MindMirror
@TheMysteryDrop
·
257d
@GaryMarcus @samuelaxon Given that (1) Sonnet 3.7 is at 48% success on this metric today, (2) and tool use is shown to provide a critical boost, and (3) new models like O3 are great at tool use, why would you be bearish on the capabilities in 12-18 months?
Gary Marcus
@GaryMarcus
·
257d
@TheMysteryDrop The debugging part of this is seriously implausible, see eg https://arstechnica.com/ai/2025/… by @samuelaxon, discussing a recent Microsoft study

The conversation involves a discussion on the capabilities of AI in software engineering, particularly in debugging, compared to human engineers. It engages with public issues related to technology and AI development.

  1. Principle 1:
    I will strive to do no harm with my words and actions.
    The discussion is generally respectful and does not appear to cause harm. It focuses on technological capabilities and progress. [+1]
  2. Principle 3:
    I will use my words and actions to promote understanding, empathy, and compassion.
    The conversation promotes understanding by sharing data and perspectives on AI's progress in software engineering. [+1]
  3. Principle 4:
    I will engage in constructive criticism and dialogue with those in disagreement and will not engage in personal attacks or ad hominem arguments.
    Participants engage in constructive dialogue, presenting evidence and reasoning to support their views without personal attacks. [+1]