Gary Marcus

Rank 14 of 47
|
Score 83
12/21/2024, 11:03:56 PM
X
In reply to:
Matt Schwartz
@mattcschwartz
·
374d
@GaryMarcus Cool. Glad I was able to help with your original question
Gary Marcus
@GaryMarcus
·
374d
@mattcschwartz I do see why you heard it that way - but also why many others didn’t.
Matt Schwartz
@mattcschwartz
·
374d
@GaryMarcus I think he just meant that they didn’t do anything special to try to solve ARC, like finetune a new version of the model. It’s very natural to use any training data you can get, including ARC problems
Gary Marcus
@GaryMarcus
·
374d
@mattcschwartz Saying they didn’t target mislead me, and i know how benchmarks work.

That seemed to be an affirmative statement that there was not extensive training towards the test.
Matt Schwartz
@mattcschwartz
·
374d
@GaryMarcus Okay, that’s fine, but I think people with a good understanding of the challenge probably did because they know that all models in the competition are trained on the public set. So I don’t think it’s fair to say they mislead anyone
Gary Marcus
@GaryMarcus
·
374d
@mattcschwartz i don’t dispute that their performance was better; i did not walk away from the video knowing that they had trained on hundreds of related examples.

The statement 'fair!' is a brief acknowledgment in a conversation about the interpretation of training data usage in a model competition. It does not engage substantively with public issues or contribute to civic dialogue, as it is part of a technical discussion between individuals.