Gary Marcus

Rank 13 of 47
|
Score 83
12/22/2024, 3:38:02 AM
X
In reply to:
Olcan
@olcan
·
385d
@GaryMarcus @mikeknoop @doomie @circlerotator @jpineau1 my main issue is that the chart published by @fchollet on the arc foundation blog greatly exaggerates the jump ("breakthrough") to o3 series. we already know tuning takes llama 3 8b model from 9% to 45%, see figures 1 and 7 in https://arxiv.org/pdf/2411.07279
Gary Marcus
@GaryMarcus
·
385d
@olcan @mikeknoop @doomie @circlerotator My own thoughts below, referring to this thread. I appreciate people like @doomie speaking up from within the ML community to insist on transparency and proper science.

Think for a minute about @jpineau1’s Replicability Checklist. There is a reason why thrown together…
Gary Marcus
@GaryMarcus
·
386d
The replies in this thread shows that there are lots of people *within the ML community* raising the same concerns re how OpenAI discussed ARC that I was making. (And more concerns, e.g, in important comments about how the graph was labeled).

Importantly, this an issue about…
Olcan
@olcan
·
386d
@mikeknoop @doomie @circlerotator it just seems odd to put tuned methods and untuned models (or even differently tuned models) on the same chart since tuning makes such a huge difference; i mean there is a very similar looking chart in the TTT paper based on tuning ablations alone
Mike Knoop
@mikeknoop
·
386d
@doomie @circlerotator ARC Prize foundation is a nonprofit
Dumitru Erhan
@doomie
·
386d
@circlerotator @mikeknoop didn't realize the ARC folks are a business
shaperotator (e/λ)
@circlerotator
·
386d
@doomie @mikeknoop they just wanted to release in time for christmas, it's a business not a college social club

The statement 'jeezus!' is a brief, informal reaction to a complex discussion about machine learning models and their representation in charts. It does not engage substantively with the public issues being discussed, such as transparency in AI research or the accuracy of data representation.