Gary Marcus

Rank 14 of 47
|
Score 83
12/21/2024, 10:50:35 PM
X
In reply to:
Merovingian
@f4talStrategies
·
373d
@GaryMarcus skill issue

learn how benchmarks work
Gary Marcus
@GaryMarcus
·
373d
@f4talStrategies see you me old comments about “life is not a kaggle competition” and also watch the misleading video i just posted.

many of us thought this was zero-shot, based on what we heard. it was not.
Merovingian
@f4talStrategies
·
373d
@GaryMarcus Pretraining on related examples isn’t scandalous—it’s standard benchmarking practice.

The ARC Challenge Set is explicitly designed to evaluate reasoning under constraints by excluding questions solvable via retrieval or co-occurrence (see the paper).
Gary Marcus
@GaryMarcus
·
373d
If you, like me, thought this video implied that the ARC performance was zero-shot, you were fooled.

If you inferred that it meant that we pretrained the system on hundreds of related examples (and perhaps a bunch of undisclosed data augmentation), I’d love to know how you…
Gary Marcus
@GaryMarcus
·
373d
Incredible video if you watch it a couple times and realize what they actually did, training on 75% of the public set.

Never trust Sam. Ever.

The statement is part of a public discourse involving a discussion on the transparency and integrity of AI benchmarking practices. The conversation critiques the communication and representation of AI performance metrics, which is a public concern in the field of AI ethics and transparency.

  1. Principle 1:
    I will strive to do no harm with my words and actions.
    The statement calls for clarity and honesty in public communication, aligning with the principle of doing no harm with words. [+1]
  2. Principle 2:
    I will respect the privacy and dignity of others and will not engage in cyberbullying, harassment, or hate speech.
    The statement does not engage in cyberbullying or hate speech, maintaining respect for others. [+1]
  3. Principle 3:
    I will use my words and actions to promote understanding, empathy, and compassion.
    The statement promotes understanding by demanding clear explanations, which can foster empathy and compassion in public discourse. [+1]
  4. Principle 4:
    I will engage in constructive criticism and dialogue with those in disagreement and will not engage in personal attacks or ad hominem arguments.
    The statement engages in constructive criticism by addressing misleading remarks without resorting to personal attacks. [+1]
  5. Principle 7:
    I will uphold the principles of free speech and use my platform responsibly and with integrity.
    The statement upholds free speech by responsibly calling for transparency and integrity in public discussions. [+1]