Gary Marcus

Rank 20 of 47
|
Score 55
6/9/2025, 2:46:37 PM
X
In reply to:
VraserX e/acc
@VraserX
·
21d
@GaryMarcus These problems were written on-site, after model weights were frozen, and given under NDA, so no, they weren’t in the training data. That’s kind of the whole point.

Math is just where reasoning is easiest to measure. The same toolchain already shows strong results in code,
Gary Marcus
@GaryMarcus
·
21d
@VraserX 1. Those specific problems may not have been seen in the training set, but then again we have NO IDEA what math-specific augmentation OpenAI has done. Similar problems may have in the training set/fine tuning/RL etc; only OpenAI knows how much that might or might not be true. (We
VraserX e/acc
@VraserX
·
21d
@GaryMarcus https://www.scientificamerican.c…

The whole point of the Berkeley experiment was to avoid contamination.
All math problems were newly invented, never published, never online, not seen in any training set.
And still, o4-mini solved most of them faster than the human authors.

This isn’t
Gary Marcus
@GaryMarcus
·
21d
@VraserX link?

the main issue here is that the results on math don’t seem to generalize well to many domains, suggesting data contamination and/or heavy math-specific data augmentation (which is in easier in math than other domains because of verifiability etc).

by the way, computers
VraserX e/acc
@VraserX
·
21d
🚨 At a secret math summit, top researchers tried to outsmart AI with never-before-seen math problems, unpublished, never on the web, not in any training set.

Result?

OpenAI’s o4-mini solved most of them faster than the mathematicians themselves.

One researcher spent months

The statement is part of a discussion about AI's capabilities in solving math problems, specifically addressing concerns about data augmentation and training. The tone is assertive, indicating frustration with the perceived lack of engagement in the discussion. The intent is to emphasize the importance of addressing the augmentation issue and to signal a potential withdrawal from the conversation if it remains unaddressed.

  1. Principle 1:
    I will strive to do no harm with my words and actions.
    The statement does not cause harm but expresses frustration, which could be seen as a minor negative impact on the dialogue.
  2. Principle 4:
    I will engage in constructive criticism and dialogue with those in disagreement and will not engage in personal attacks or ad hominem arguments.
    The statement threatens to disengage rather than constructively addressing the disagreement, which could be seen as a minor violation of engaging in constructive dialogue. [-1]