Gary Marcus

Rank 12 of 47
|
Score 83
12/22/2024, 12:51:00 AM
X
In reply to:
Brydon Eastman
@brhydon
·
399d
@GaryMarcus This isn't going to a Taylor Swift concert and catching her lip syncing. This is going to a Taylor Swift concert and being upset that she's taken singing lessons before.
Brydon Eastman
@brhydon
·
399d
@GaryMarcus Professor Marcus, you're upset that we used the evaluation EXACTLY as designed and indicated by the original paper.

If someone watched that stream and feels mislead, then they never read the paper.

There is zero dishonesty here and its disrespectful to insinuate otherwise.
Gary Marcus
@GaryMarcus
·
399d
@brhydon ARC folks, who you now have a partnership may have ok’d it but

1. it’s more like studying for the SAT then reading a book. It’s learning about the idiosyncrasies of the questions. we have seen this many times in benchmarks.

2. am pretty sure humans could do well on ARC by…
Brydon Eastman
@brhydon
·
399d
@GaryMarcus It's truly no different than reading a text book and then going to do a test.

There is no scandal and nothing up our sleeves - the Arc agi folks explicitly OK this.
Brydon Eastman
@brhydon
·
399d
@GaryMarcus it's the TRAIN set that was used not the TEST set. And it wasn't even the FULL train set.

The format for Arc is very bespoke and requires some explanation, using 3/4 of the TRAIN set we demonstrated the weird JSON format and let the thing fly on the PRIVATE TEST set.
Gary Marcus
@GaryMarcus
·
399d
If Taylor Swift lipsynced her way through her Eras Tours, it would be a huge scandal. (And she would never do such a thing.)

If OpenAI gave a big spiel about solving ARC-AGI and didn’t say out loud that they trained on 75% of the public examples, and didn’t dwell on the fact…

The statement is part of a broader discussion about transparency and honesty in AI evaluation methods. It uses a metaphor comparing singing lessons to AI training, suggesting that preparation is expected and not misleading. The tone is defensive, aiming to clarify and justify actions taken in the AI evaluation process.

  1. Principle 1:
    I will strive to do no harm with my words and actions.
    The statement strives to clarify and defend actions, aiming to do no harm by addressing potential misunderstandings. [+1]
  2. Principle 2:
    I will respect the privacy and dignity of others and will not engage in cyberbullying, harassment, or hate speech.
    It respects the privacy and dignity of others by not engaging in personal attacks, focusing instead on the issue at hand. [+1]
  3. Principle 4:
    I will engage in constructive criticism and dialogue with those in disagreement and will not engage in personal attacks or ad hominem arguments.
    The statement engages in constructive dialogue, attempting to address criticisms and clarify misunderstandings without resorting to personal attacks. [+1]