Gary Marcus

Rank 14 of 47
|
Score 83
12/21/2024, 10:48:16 PM
X
In reply to:
Merovingian
@f4talStrategies
·
373d
@GaryMarcus @brhydon Your mischaracterizations are wild.

1. Studying != Memorizing Idiosyncrasies:

ARC’s Challenge Set removes questions solvable by surface-level patterns or retrieval. Fine-tuning prepares models to reason under constraints, just like studying builds reasoning for humans.

2. ‘3…
Gary Marcus
@GaryMarcus
·
373d
@brhydon ARC folks, who you now have a partnership may have ok’d it but

1. it’s more like studying for the SAT then reading a book. It’s learning about the idiosyncrasies of the questions. we have seen this many times in benchmarks.

2. am pretty sure humans could do well on ARC by…
Brydon Eastman
@brhydon
·
373d
@GaryMarcus It's truly no different than reading a text book and then going to do a test.

There is no scandal and nothing up our sleeves - the Arc agi folks explicitly OK this.
Brydon Eastman
@brhydon
·
373d
@GaryMarcus it's the TRAIN set that was used not the TEST set. And it wasn't even the FULL train set.

The format for Arc is very bespoke and requires some explanation, using 3/4 of the TRAIN set we demonstrated the weird JSON format and let the thing fly on the PRIVATE TEST set.
Gary Marcus
@GaryMarcus
·
373d
If Taylor Swift lipsynced her way through her Eras Tours, it would be a huge scandal. (And she would never do such a thing.)

If OpenAI gave a big spiel about solving ARC-AGI and didn’t say out loud that they trained on 75% of the public examples, and didn’t dwell on the fact…

The statement is part of a public discourse on the topic of AI training and testing methodologies, specifically addressing the use of training examples in AI models. The tone is analytical and critical, aiming to clarify misunderstandings or mischaracterizations in the ongoing discussion.

  1. Principle 1:
    I will strive to do no harm with my words and actions.
    The statement does not appear to cause harm with words or actions. It focuses on clarifying a point about AI training without using harmful language. [+1]
  2. Principle 4:
    I will engage in constructive criticism and dialogue with those in disagreement and will not engage in personal attacks or ad hominem arguments.
    The statement engages in constructive criticism by addressing specific points in the conversation. It does not engage in personal attacks or ad hominem arguments, maintaining a focus on the topic at hand. [+1]