Gary Marcus

Rank 14 of 47
|
Score 83
12/22/2024, 12:47:36 AM
X
In reply to:
David Balfour
@hal_incandenza4
·
374d
@GaryMarcus It would be worth calling out the zero-shot performance of the model if that’s what they did. There’s no obligation to detail all the rules of what the challenge allows if they’re just talking about it as is. Anyone else can then learn more about what arc-agi is
David Balfour
@hal_incandenza4
·
374d
@GaryMarcus I did. The performance improvement is relative to other models - which are all trained on the public dataset. Training on the training set available for precisely that purpose is part of the challenge and if you’re not aware of the challenge parameters that’s on you
Gary Marcus
@GaryMarcus
·
374d
@hal_incandenza4 Watch the video
David Balfour
@hal_incandenza4
·
374d
@GaryMarcus They did exactly what every other model taking the arc benchmark does - the training set is meant to be trained on - there was nothing underhanded here - stop fishing
Gary Marcus
@GaryMarcus
·
374d
Poll only for those who watched the o3 model announcement:

Did you walk away realizing they had trained on hundreds of ARC public examples, prior to testing? (If you did, how did you know that?)

The statement engages in a technical discussion about model training and performance, addressing concerns about transparency and methodology in AI research. The tone is defensive and corrective, aiming to clarify misunderstandings about the training process.

  1. Principle 1:
    I will strive to do no harm with my words and actions.
    The statement strives to clarify misconceptions, which aligns with the principle of doing no harm by preventing misinformation. [+1]
  2. Principle 3:
    I will use my words and actions to promote understanding, empathy, and compassion.
    The statement promotes understanding by explaining the training process, though the tone could be more empathetic. [+1]
  3. Principle 4:
    I will engage in constructive criticism and dialogue with those in disagreement and will not engage in personal attacks or ad hominem arguments.
    The statement engages in constructive dialogue by addressing specific points of disagreement without personal attacks. [+1]