Gary Marcus

Rank 14 of 47

Score 83

Item (b) below is quite incredible. OpenAI showed a misleading graph comparing o1 with o3, ignoring literally everyone else on the leaderboard, and o3, which did better than o1, had more relevant exposure to the sample items.

You can’t conclude diddly-squat from such a…

Jiaqi Zhai

@Lunarmony

392d

@olcan @circlerotator @stanWis @GaryMarcus @mikeknoop @doomie @jpineau1 @fchollet afaik: a) x-axis is log-scale which was *not* labeled, b) o1 wasn't finetuned on ARC-AGI (but o3 is), c) even sampling 8K seqs using 4o leads to ~50% accuracy (https://redwoodresearch.substack…). It's really a matter of whether we are evaluating a marketing post or a scientific post..

12/23/2024, 1:56:28 AM

The statement critiques the presentation of data by OpenAI, suggesting that the graph is misleading and lacks comprehensive comparison. The tone is critical and aims to question the validity of the conclusions drawn from the data.

Principle 1:
I will strive to do no harm with my words and actions.
The statement raises concerns about potential misinformation, which aligns with striving to do no harm by promoting accurate information. [+1]
Principle 4:
I will engage in constructive criticism and dialogue with those in disagreement and will not engage in personal attacks or ad hominem arguments.
The statement engages in constructive criticism by questioning the methodology and presentation of data, rather than attacking individuals. [+1]