Gary Marcus

Rank 14 of 47
|
Score 83
12/21/2024, 10:49:51 PM
X
In reply to:
Matt Schwartz
@mattcschwartz
·
373d
@GaryMarcus You mean at the end when he says they didn’t do anything special to target this challenge? That’s totally fair, they train on basically everything on the internet, which includes the ARC public set. I guarantee you all the models are trained on it and they couldn’t solve it
Gary Marcus
@GaryMarcus
·
373d
@mattcschwartz did you watch the clip? watch it a couple times, focusing on what Sam says.
Matt Schwartz
@mattcschwartz
·
373d
@GaryMarcus How?
Gary Marcus
@GaryMarcus
·
373d
@mattcschwartz it was portrayed as zero-shot.
Matt Schwartz
@mattcschwartz
·
373d
@GaryMarcus Probably because every system in the ARC challenge is trained on the public set. That’s the whole point of the public set
Gary Marcus
@GaryMarcus
·
373d
If you, like me, thought this video implied that the ARC performance was zero-shot, you were fooled.

If you inferred that it meant that we pretrained the system on hundreds of related examples (and perhaps a bunch of undisclosed data augmentation), I’d love to know how you…

The statement is part of a public discourse about the training of AI models and their performance on specific challenges. It engages in a technical discussion about AI training practices and transparency, which are relevant public issues.

  1. Principle 4:
    I will engage in constructive criticism and dialogue with those in disagreement and will not engage in personal attacks or ad hominem arguments.
    The statement engages in constructive criticism by acknowledging the performance improvement but raises a concern about the lack of transparency regarding the training data. It does not engage in personal attacks or ad hominem arguments. [+1]