Gary Marcus

Rank 14 of 47
|
Score 83
12/21/2024, 10:48:40 PM
X
In reply to:
Merovingian
@f4talStrategies
·
373d
@GaryMarcus @brhydon Your mischaracterizations are wild.

1. Studying != Memorizing Idiosyncrasies:

ARC’s Challenge Set removes questions solvable by surface-level patterns or retrieval. Fine-tuning prepares models to reason under constraints, just like studying builds reasoning for humans.

2. ‘3…
Gary Marcus
@GaryMarcus
·
373d
@brhydon ARC folks, who you now have a partnership may have ok’d it but

1. it’s more like studying for the SAT then reading a book. It’s learning about the idiosyncrasies of the questions. we have seen this many times in benchmarks.

2. am pretty sure humans could do well on ARC by…
Brydon Eastman
@brhydon
·
373d
@GaryMarcus It's truly no different than reading a text book and then going to do a test.

There is no scandal and nothing up our sleeves - the Arc agi folks explicitly OK this.
Brydon Eastman
@brhydon
·
373d
@GaryMarcus it's the TRAIN set that was used not the TEST set. And it wasn't even the FULL train set.

The format for Arc is very bespoke and requires some explanation, using 3/4 of the TRAIN set we demonstrated the weird JSON format and let the thing fly on the PRIVATE TEST set.
Gary Marcus
@GaryMarcus
·
373d
If Taylor Swift lipsynced her way through her Eras Tours, it would be a huge scandal. (And she would never do such a thing.)

If OpenAI gave a big spiel about solving ARC-AGI and didn’t say out loud that they trained on 75% of the public examples, and didn’t dwell on the fact…

The statement is part of a public discourse involving a discussion on the ethical and methodological aspects of AI training and testing, specifically related to the ARC challenge. The conversation involves multiple parties discussing transparency and the nature of AI model training, which are significant public issues in the field of AI ethics and technology.

  1. Principle 1:
    I will strive to do no harm with my words and actions.
    The statement does not appear to cause harm directly, as it focuses on clarifying a point about the video presentation rather than attacking individuals.
  2. Principle 2:
    I will respect the privacy and dignity of others and will not engage in cyberbullying, harassment, or hate speech.
    The statement respects the privacy and dignity of others, as it does not engage in cyberbullying, harassment, or hate speech.
  3. Principle 3:
    I will use my words and actions to promote understanding, empathy, and compassion.
    The statement aims to promote understanding by clarifying a point about the video, which can contribute to a more informed discussion. [+1]
  4. Principle 4:
    I will engage in constructive criticism and dialogue with those in disagreement and will not engage in personal attacks or ad hominem arguments.
    The statement engages in constructive dialogue by addressing a specific aspect of the conversation without resorting to personal attacks. [+1]