Gary Marcus

Rank 14 of 47
|
Score 83
12/21/2024, 11:03:11 PM
X
In reply to:
Matt Schwartz
@mattcschwartz
·
374d
@GaryMarcus Also, the whole point of ARC is that every problem is different. That’s why they let you train on a public set in the first place, so your model understands the format of the challenge but doesn’t know how to solve other examples without very high capabilities
Matt Schwartz
@mattcschwartz
·
374d
@GaryMarcus I think he just meant that they didn’t do anything special to try to solve ARC, like finetune a new version of the model. It’s very natural to use any training data you can get, including ARC problems
Gary Marcus
@GaryMarcus
·
374d
@mattcschwartz Saying they didn’t target mislead me, and i know how benchmarks work.

That seemed to be an affirmative statement that there was not extensive training towards the test.
Matt Schwartz
@mattcschwartz
·
374d
@GaryMarcus Okay, that’s fine, but I think people with a good understanding of the challenge probably did because they know that all models in the competition are trained on the public set. So I don’t think it’s fair to say they mislead anyone
Gary Marcus
@GaryMarcus
·
374d
@mattcschwartz i don’t dispute that their performance was better; i did not walk away from the video knowing that they had trained on hundreds of related examples.
Matt Schwartz
@mattcschwartz
·
374d
@GaryMarcus You mean at the end when he says they didn’t do anything special to target this challenge? That’s totally fair, they train on basically everything on the internet, which includes the ARC public set. I guarantee you all the models are trained on it and they couldn’t solve it

The statement engages in a technical discussion about machine learning benchmarks and model training, referencing specific research to support the argument. It critiques assumptions about model capabilities and highlights weaknesses in generalization beyond training data.

  1. Principle 1:
    I will strive to do no harm with my words and actions.
    The statement strives to do no harm by focusing on a technical critique rather than personal attacks. [+1]
  2. Principle 3:
    I will use my words and actions to promote understanding, empathy, and compassion.
    It promotes understanding by referencing research and providing evidence to support its claims. [+2]
  3. Principle 4:
    I will engage in constructive criticism and dialogue with those in disagreement and will not engage in personal attacks or ad hominem arguments.
    The statement engages in constructive dialogue by addressing specific points in the conversation and providing evidence-based counterarguments. [+2]