Gary Marcus

Rank 13 of 47

Score 83

@JadeCole2112 I don’t know. After all this I lack clarity on what was done and what is new and what accounts for the gain and how big the gain is relative to prior work which maybe wasn’t all shown.

There may be something real here but transparency is definitely lacking.

12/22/2024, 4:40:46 AM

In reply to:

Jade Cole

@JadeCole2112

387d

@GaryMarcus Just so I’m understanding- this gain in performance was likely a fine tuning artifact?

Gary Marcus

@GaryMarcus

387d

Jaw drop.

Olcan

@olcan

387d

@GaryMarcus @mikeknoop @doomie @circlerotator @jpineau1 my main issue is that the chart published by @fchollet on the arc foundation blog greatly exaggerates the jump ("breakthrough") to o3 series. we already know tuning takes llama 3 8b model from 9% to 45%, see figures 1 and 7 in https://arxiv.org/pdf/2411.07279

The statement is part of a broader discussion on transparency and accuracy in reporting performance gains in machine learning models. It questions the clarity and transparency of the reported gains, suggesting that while there may be genuine improvements, the lack of transparency makes it difficult to assess the true extent of these gains.

Principle 1:
I will strive to do no harm with my words and actions.
The statement does not use harmful language and maintains a neutral tone, focusing on the need for clarity and transparency. [+1]
Principle 3:
I will use my words and actions to promote understanding, empathy, and compassion.
The statement promotes understanding by highlighting the need for transparency, which is crucial for informed discussions and assessments. [+1]
Principle 4:
I will engage in constructive criticism and dialogue with those in disagreement and will not engage in personal attacks or ad hominem arguments.
The statement engages in constructive criticism by questioning the transparency of the reported gains without resorting to personal attacks. [+1]
Principle 7:
I will uphold the principles of free speech and use my platform responsibly and with integrity.
The statement upholds free speech by responsibly questioning the transparency of the information presented, contributing to a more informed public discourse. [+1]