Gary Marcus

Rank 14 of 47
|
Score 83
4/23/2025, 5:43:10 PM
X
In reply to:
Frederick Snook
@FrederickS92254
·
271d
@dimd00d @GaryMarcus @tobyordoxford ARC AGI was definitely not the perfect test and now they've made it more intense for modern-day llms so it's going to be a slog getting anything that's perfect in performance. we also need to figure out how to make the AI more honest instead of this lying shit.
dimd00d
@dimd00d
·
271d
@GaryMarcus @tobyordoxford The problem from where I am sitting is:
a) benchmark that sounded like "AGI achieved internally"
b) model that was most likely finetuned on it
c) oops, the dog ate the model, so you cant repeat the test, but trust us bro.

VC fuel.
Gary Marcus
@GaryMarcus
·
271d
@tobyordoxford that is a key part of the scandal: o3-you-can’-t-actually-have-but-claim-we-have is NOT o3-that-you-can-have, but the two are named with highly confusable names.

And I don’t think there was much public acknowledgement that was advertised December is not the product that was
Toby Ord
@tobyordoxford
·
271d
@GaryMarcus If so, then the main problem is the false impression that this 'o3' release gives you access to the kind of results people were losing their minds over in December. I think that is serious enough that they should have given it a different name or at least been super clear.
Toby Ord
@tobyordoxford
·
271d
@GaryMarcus I don't think it is a major scandal, but I do think it is a bad look. My guess is that the December results were a mixture of lots of fine-tuning on the ARC-AGI training set and an early version of o3 that was easier to inference-scale (ability that was traded off for price).
Gary Marcus
@GaryMarcus
·
271d
MAJOR SCANDAL: OpenAI’s wildly hyped December o3 ARC demo can’t readily be replicated with the released model.

If that doesn’t give you Theranos vibes, I am not sure what will. Especially in conjunction with the fact that they had privileged access to FrontierMath that they

The statement 'honesty and LLMs are not especially compatible' is part of a broader discussion about the transparency and reliability of AI models, specifically in the context of OpenAI's practices. The conversation critiques the honesty and replicability of AI model performance claims, which is a public issue concerning technology ethics and corporate responsibility.

  1. Principle 1:
    I will strive to do no harm with my words and actions.
    The statement does not directly cause harm but implies a critique of AI honesty, which could be seen as a call for improvement.
  2. Principle 3:
    I will use my words and actions to promote understanding, empathy, and compassion.
    The statement indirectly promotes understanding by highlighting a potential issue with AI honesty, encouraging further discussion. [+1]
  3. Principle 4:
    I will engage in constructive criticism and dialogue with those in disagreement and will not engage in personal attacks or ad hominem arguments.
    The statement engages in constructive criticism of AI models and their honesty, without personal attacks. [+1]
  4. Principle 6:
    I will use my influence for the betterment of society.
    By addressing the honesty of AI models, the statement uses influence to potentially advocate for better practices in AI development. [+1]
  5. Principle 7:
    I will uphold the principles of free speech and use my platform responsibly and with integrity.
    The statement upholds free speech by responsibly critiquing AI models, contributing to a necessary public discourse on AI ethics. [+1]