Gary Marcus

Rank 14 of 47
|
Score 83
12/23/2024, 1:53:02 AM
X
In reply to:
Jiaqi Zhai
@Lunarmony
·
392d
@olcan @circlerotator @stanWis @GaryMarcus @mikeknoop @doomie @jpineau1 @fchollet afaik: a) x-axis is log-scale which was *not* labeled, b) o1 wasn't finetuned on ARC-AGI (but o3 is), c) even sampling 8K seqs using 4o leads to ~50% accuracy (https://redwoodresearch.substack…). It's really a matter of whether we are evaluating a marketing post or a scientific post..
Olcan
@olcan
·
392d
@circlerotator @stanWis @GaryMarcus @mikeknoop @doomie @jpineau1 @fchollet TTT authors also call it test-time "tuning", which is reasonable. Anyway all I'm saying is that depending on what tuning is done (which we don't know but safe to assume they did whatever they could as is their right), this gap here between the red dots and the blue dots can be…
shaperotator (e/λ)
@circlerotator
·
392d
@stanWis @GaryMarcus @olcan @mikeknoop @doomie @jpineau1 @fchollet This is afaik test time training.
stanw
@stanWis
·
392d
@circlerotator @GaryMarcus @olcan @mikeknoop @doomie @jpineau1 @fchollet So this is alie? @olcan can you comment on it?
shaperotator (e/λ)
@circlerotator
·
392d
@stanWis @GaryMarcus @olcan @mikeknoop @doomie @jpineau1 @fchollet They didn’t even fine tune it. This is literally fud from deepmind employees
stanw
@stanWis
·
392d
@circlerotator @GaryMarcus @olcan @mikeknoop @doomie @jpineau1 @fchollet From what i understand the issues is that earlier models were not tuned the same way. We have to wait till they release the mode to public

The conversation involves a technical discussion about AI model performance and evaluation methods, specifically regarding fine-tuning and test-time training. It questions the validity of performance claims and the transparency of the methods used, which are relevant to public discourse in the AI research community.

  1. Principle 1:
    I will strive to do no harm with my words and actions.
    The discussion is technical and does not appear to cause harm, focusing on clarifying methods and results. [+1]
  2. Principle 2:
    I will respect the privacy and dignity of others and will not engage in cyberbullying, harassment, or hate speech.
    The conversation respects privacy and dignity, focusing on technical aspects rather than personal attacks. [+1]
  3. Principle 4:
    I will engage in constructive criticism and dialogue with those in disagreement and will not engage in personal attacks or ad hominem arguments.
    The dialogue engages in constructive criticism, questioning the validity of claims and seeking clarification. [+1]