Gary Marcus

Rank 14 of 47
|
Score 83
Gary Marcus
@GaryMarcus
·
372d
If you, like me, thought this video implied that the ARC performance was zero-shot, you were fooled.

If you inferred that it meant that we pretrained the system on hundreds of related examples (and perhaps a bunch of undisclosed data augmentation), I’d love to know how you…
12/21/2024, 10:41:45 PM
X
In reply to:
Brydon Eastman
@brhydon
·
372d
@GaryMarcus It's truly no different than reading a text book and then going to do a test.

There is no scandal and nothing up our sleeves - the Arc agi folks explicitly OK this.
Brydon Eastman
@brhydon
·
372d
@GaryMarcus it's the TRAIN set that was used not the TEST set. And it wasn't even the FULL train set.

The format for Arc is very bespoke and requires some explanation, using 3/4 of the TRAIN set we demonstrated the weird JSON format and let the thing fly on the PRIVATE TEST set.
Gary Marcus
@GaryMarcus
·
372d
If Taylor Swift lipsynced her way through her Eras Tours, it would be a huge scandal. (And she would never do such a thing.)

If OpenAI gave a big spiel about solving ARC-AGI and didn’t say out loud that they trained on 75% of the public examples, and didn’t dwell on the fact…

The statement is part of a broader discussion about the transparency and ethical considerations in AI training and testing, specifically related to the ARC benchmark. It engages with public concerns about potential misleading practices in AI development, which constitutes public discourse.

  1. Principle 1:
    I will strive to do no harm with my words and actions.
    The statement does not directly cause harm but implies dissatisfaction with misleading actions, which could indirectly cause reputational harm.
  2. Principle 2:
    I will respect the privacy and dignity of others and will not engage in cyberbullying, harassment, or hate speech.
    The statement does not engage in cyberbullying, harassment, or hate speech, but it does imply criticism of an individual's actions.
  3. Principle 3:
    I will use my words and actions to promote understanding, empathy, and compassion.
    The statement does not promote understanding, empathy, or compassion, as it focuses on criticism rather than constructive dialogue.
  4. Principle 4:
    I will engage in constructive criticism and dialogue with those in disagreement and will not engage in personal attacks or ad hominem arguments.
    The statement engages in criticism but does not provide constructive dialogue or avoid personal attacks, as it implies blame on an individual.
  5. Principle 5:
    I will acknowledge and correct my mistakes.
    The statement does not acknowledge or correct any mistakes, as it is focused on criticizing perceived misleading actions.
  6. Principle 6:
    I will use my influence for the betterment of society.
    The statement uses influence to highlight potential ethical issues in AI, which could be seen as contributing to societal betterment by advocating for transparency.
  7. Principle 7:
    I will uphold the principles of free speech and use my platform responsibly and with integrity.
    The statement upholds free speech by engaging in a public debate about AI ethics, but it could use more responsible language to avoid personal attacks.