Gary Marcus

Rank 13 of 47
|
Score 83
Gary Marcus
@GaryMarcus
·
272d
@littmath it’s been four months, a bunch of models have been released with “o3” brandname but you can’t buy anything that does what they livestreamed at any price.

it’s also not the only problem with that announcement; there were many. Several I reported here: https://garymarcus.substack.com/…
4/23/2025, 5:00:12 PM
X
In reply to:
Toby Ord
@tobyordoxford
·
272d
@GaryMarcus I don't think it is a major scandal, but I do think it is a bad look. My guess is that the December results were a mixture of lots of fine-tuning on the ARC-AGI training set and an early version of o3 that was easier to inference-scale (ability that was traded off for price).
Gary Marcus
@GaryMarcus
·
272d
MAJOR SCANDAL: OpenAI’s wildly hyped December o3 ARC demo can’t readily be replicated with the released model.

If that doesn’t give you Theranos vibes, I am not sure what will. Especially in conjunction with the fact that they had privileged access to FrontierMath that they
Toby Ord
@tobyordoxford
·
272d
@arcprize @OpenAI 2. The @arcprize team tried to test a high-compute version of o3, but it kept failing to answer. They spent >$50,000 trying to get it to work, but couldn't, so those December results can't realistically be replicated with the released model.

The conversation involves a discussion about the replicability and performance of OpenAI's o3 model, with some participants suggesting a potential scandal due to discrepancies between the demo and the released model. The tone is critical and investigative, focusing on transparency and accountability in AI development.

  1. Principle 1:
    I will strive to do no harm with my words and actions.
    The statement aims to highlight potential issues with transparency and replicability, which could be seen as striving to do no harm by ensuring accountability. [+1]
  2. Principle 2:
    I will respect the privacy and dignity of others and will not engage in cyberbullying, harassment, or hate speech.
    The discussion respects privacy and dignity, focusing on the technical aspects rather than personal attacks. [+1]
  3. Principle 3:
    I will use my words and actions to promote understanding, empathy, and compassion.
    The conversation promotes understanding by discussing technical details and potential discrepancies. [+1]
  4. Principle 4:
    I will engage in constructive criticism and dialogue with those in disagreement and will not engage in personal attacks or ad hominem arguments.
    The dialogue engages in constructive criticism, questioning the claims made by OpenAI without resorting to personal attacks. [+1]