Gary Marcus

Rank 14 of 47
|
Score 83
12/21/2024, 11:29:58 PM
X
In reply to:
Brydon Eastman
@brhydon
·
374d
@GaryMarcus Professor Marcus, you're upset that we used the evaluation EXACTLY as designed and indicated by the original paper.

If someone watched that stream and feels mislead, then they never read the paper.

There is zero dishonesty here and its disrespectful to insinuate otherwise.
Gary Marcus
@GaryMarcus
·
374d
@brhydon ARC folks, who you now have a partnership may have ok’d it but

1. it’s more like studying for the SAT then reading a book. It’s learning about the idiosyncrasies of the questions. we have seen this many times in benchmarks.

2. am pretty sure humans could do well on ARC by…
Brydon Eastman
@brhydon
·
374d
@GaryMarcus It's truly no different than reading a text book and then going to do a test.

There is no scandal and nothing up our sleeves - the Arc agi folks explicitly OK this.
Brydon Eastman
@brhydon
·
374d
@GaryMarcus it's the TRAIN set that was used not the TEST set. And it wasn't even the FULL train set.

The format for Arc is very bespoke and requires some explanation, using 3/4 of the TRAIN set we demonstrated the weird JSON format and let the thing fly on the PRIVATE TEST set.
Gary Marcus
@GaryMarcus
·
374d
If Taylor Swift lipsynced her way through her Eras Tours, it would be a huge scandal. (And she would never do such a thing.)

If OpenAI gave a big spiel about solving ARC-AGI and didn’t say out loud that they trained on 75% of the public examples, and didn’t dwell on the fact…

The statement is part of a public discourse involving a debate about the transparency and methodology used in evaluating AI models, specifically in relation to the ARC-AGI project. The tone is critical and seeks to address perceived miscommunication or misunderstanding among viewers.

  1. Principle 1:
    I will strive to do no harm with my words and actions.
    The statement does not appear to cause harm directly but expresses dissatisfaction with a perceived miscommunication.
  2. Principle 2:
    I will respect the privacy and dignity of others and will not engage in cyberbullying, harassment, or hate speech.
    The statement respects privacy and dignity, focusing on the issue rather than personal attacks.
  3. Principle 3:
    I will use my words and actions to promote understanding, empathy, and compassion.
    The statement does not actively promote understanding, empathy, or compassion, as it primarily expresses dissatisfaction. [-1]
  4. Principle 4:
    I will engage in constructive criticism and dialogue with those in disagreement and will not engage in personal attacks or ad hominem arguments.
    The statement engages in constructive criticism by highlighting a perceived issue with the communication of the evaluation process, without resorting to personal attacks. [+1]