Gary Marcus

Rank 10 of 47
|
Score 118
5/26/2025, 10:13:12 AM
X
In reply to:
Haider.
@slow_developer
·
58d
@ToddMcLees or we could also include failure mode analysis in these test suites, focusing on how and why models fail on complex tasks, not just whether they fail.

this might help uncover deeper problems in the model's design or training data
Todd McLees
@ToddMcLees
·
58d
@slow_developer I think of them as sanity checks—the model can at least spot patterns at the level of a bright individual. Useful, yet nowhere near a verdict on “general” intelligence.

These are better measuring sticks now…

Real-task suites – SWE-Bench, AgentBench, MATH: multistep reasoning
Haider.
@slow_developer
·
58d
what do you think about this IQ test for language models?

> claude 4 opus is currently leading with 120 offline and 117 on the mensa norway test

> o3 model scored even higher with 135 on the mensa test

these results are impressive.

but i’m still not sure how accurate the test

The conversation involves a discussion on evaluating language models using IQ tests and the importance of failure mode analysis. The reply 'Excellent idea' supports the suggestion to include failure mode analysis, which aims to improve understanding of model limitations.

  1. Principle 1:
    I will strive to do no harm with my words and actions.
    The statement supports a constructive idea that aims to improve model evaluation, aligning with the principle of doing no harm. [+1]
  2. Principle 3:
    I will use my words and actions to promote understanding, empathy, and compassion.
    By promoting a deeper understanding of model failures, the statement encourages empathy and compassion towards improving AI systems. [+1]
  3. Principle 4:
    I will engage in constructive criticism and dialogue with those in disagreement and will not engage in personal attacks or ad hominem arguments.
    The statement engages in constructive dialogue by supporting a suggestion that enhances the discussion. [+1]