Gary Marcus

Rank 19 of 47
|
Score 64
6/8/2025, 2:26:50 PM
X
In reply to:
𝞍 Shin Megami Boson 𝞍
@shinboson
·
35d
@GaryMarcus @rao2z
𝞍 Shin Megami Boson 𝞍
@shinboson
·
36d
this paper sucks аѕѕ and last night I got into a little debate with gary marcus about whether this paper does in fact suck. the top right graph shows the complexity performance relationship shown in the paper.

my graph below it shows that with tool usage, the accuracy becomes
Gary Marcus
@GaryMarcus
·
35d
@shinboson @rao2z i don’t see your link.

a teen (who was sufficiently interested) would do it on an actual tower.
𝞍 Shin Megami Boson 𝞍
@shinboson
·
35d
@GaryMarcus @rao2z Also I flat out don’t believe that a typical “bright teenager” can write out the correct sequence of 255 moves for the 8 disk tower of Hanoi problem from their head without transcription errors. I don’t believe the paper even claims this fwiw.
𝞍 Shin Megami Boson 𝞍
@shinboson
·
35d
@GaryMarcus @rao2z Gary I did implement it. See the link. I implemented it 48 minutes after you said it couldn’t be done.
Gary Marcus
@GaryMarcus
·
35d
@shinboson It’s not a trivial test; it’s failure modes are not in fact same as people (e.g. a bright teenager can easily learn to do 8 discs), and if you can trviially implement with a scratchpad, you should do so and demonstrate it. you should also read the related work by @rao2z that i
𝞍 Shin Megami Boson 𝞍
@shinboson
·
35d
@GaryMarcus 1) what this paper evaluates is very specifically whether LLMs are able to perfectly *execute* a formal algorithm without any sort of scratchpad or tooling like symbolic reasoning

2) this paper determines that LLMs are bad at this, which is a trivial result and also so are we

The statement is a request for clarification in a technical discussion about implementing algorithms and evaluating AI models. It seeks to understand the actions taken by another participant, indicating a desire for further explanation or evidence.

  1. Principle 1:
    I will strive to do no harm with my words and actions.
    The statement does not cause harm and is neutral in tone.
  2. Principle 3:
    I will use my words and actions to promote understanding, empathy, and compassion.
    The statement promotes understanding by asking for clarification. [+1]
  3. Principle 4:
    I will engage in constructive criticism and dialogue with those in disagreement and will not engage in personal attacks or ad hominem arguments.
    The statement engages in constructive dialogue by seeking more information. [+1]