Gary Marcus

Rank 13 of 47
|
Score 83
4/23/2025, 5:01:47 PM
X
In reply to:
Daniel Litt
@littmath
·
272d
@GaryMarcus It’s been 4 months since the original announcement, and less than 24 hours since the replication you’re saying is a scandal.
Gary Marcus
@GaryMarcus
·
272d
@littmath it’s been four months, a bunch of models have been released with “o3” brandname but you can’t buy anything that does what they livestreamed at any price.

it’s also not the only problem with that announcement; there were many. Several I reported here: https://garymarcus.substack.com/…
Daniel Litt
@littmath
·
272d
@GaryMarcus Seems very premature to conclude this is a “major scandal” — why not wait a few days to see what went wrong with the eval?
Gary Marcus
@GaryMarcus
·
272d
MAJOR SCANDAL: OpenAI’s wildly hyped December o3 ARC demo can’t readily be replicated with the released model.

If that doesn’t give you Theranos vibes, I am not sure what will. Especially in conjunction with the fact that they had privileged access to FrontierMath that they
Toby Ord
@tobyordoxford
·
272d
@arcprize @OpenAI 2. The @arcprize team tried to test a high-compute version of o3, but it kept failing to answer. They spent >$50,000 trying to get it to work, but couldn't, so those December results can't realistically be replicated with the released model.

The statement critiques the replication attempts of a model, suggesting bias due to connections with OpenAI. It questions the validity of the replication process and implies a lack of transparency.

  1. Principle 1:
    I will strive to do no harm with my words and actions.
    The statement raises concerns about the integrity of the replication process, which could be seen as promoting accountability and transparency, aligning with the principle of doing no harm. [+1]
  2. Principle 2:
    I will respect the privacy and dignity of others and will not engage in cyberbullying, harassment, or hate speech.
    By questioning the connections of those involved in the replication, it respects the need for unbiased and fair evaluation, aligning with the principle of respecting privacy and dignity. [+1]
  3. Principle 4:
    I will engage in constructive criticism and dialogue with those in disagreement and will not engage in personal attacks or ad hominem arguments.
    The statement engages in constructive criticism by questioning the process rather than attacking individuals, aligning with the principle of engaging in constructive dialogue. [+1]