Gary Marcus

Rank 19 of 47
|
Score 56
6/17/2024, 11:03:55 PM
X
In reply to:
Buck Shlegeris
@bshlgrs
·
458d
Ryan's approach involves a long, carefully-crafted few-shot prompt that he uses to generate many possible Python programs to implement the transformations. He generates ~5k guesses, selects the best ones using the examples, then has a debugging step.
Buck Shlegeris
@bshlgrs
·
458d
For context, ARC-AGI is a visual reasoning benchmark that requires guessing a rule from few examples. Its creator, @fchollet, claims that LLMs are unable to learn, which is why they can't perform well on this benchmark.
Buck Shlegeris
@bshlgrs
·
458d
ARC-AGI’s been hyped over the last week as a benchmark that LLMs can’t solve. This claim triggered my dear coworker Ryan Greenblatt so he spent the last week trying to solve it with LLMs. Ryan gets 71% accuracy on a set of examples where humans get 85%; this is SOTA.

The statement engages in a technical discussion about neurosymbolic AI and large language models (LLMs) in the context of solving the ARC-AGI benchmark. The tone is somewhat critical and questioning, aiming to understand the underlying mechanisms and implications of the approach being discussed. The statement also touches on the broader debate about the capabilities and limitations of LLMs versus neurosymbolic AI.

  1. Principle 1:
    I will strive to do no harm with my words and actions.
    The statement does not appear to cause harm; it is a technical inquiry and critique.
  2. Principle 2:
    I will respect the privacy and dignity of others and will not engage in cyberbullying, harassment, or hate speech.
    The statement respects the privacy and dignity of others, focusing on the technical aspects rather than personal attacks.
  3. Principle 3:
    I will use my words and actions to promote understanding, empathy, and compassion.
    The statement promotes understanding by questioning and seeking clarification on a complex topic. [+1]
  4. Principle 4:
    I will engage in constructive criticism and dialogue with those in disagreement and will not engage in personal attacks or ad hominem arguments.
    The statement engages in constructive criticism and dialogue, asking for clarification and expressing skepticism without resorting to personal attacks. [+1]