Gary Marcus

Rank 16 of 47

Score 76

@VraserX 1. Those specific problems may not have been seen in the training set, but then again we have NO IDEA what math-specific augmentation OpenAI has done. Similar problems may have in the training set/fine tuning/RL etc; only OpenAI knows how much that might or might not be true. (We

6/9/2025, 2:36:59 PM

In reply to:

VraserX e/acc

@VraserX

@GaryMarcus https://www.scientificamerican.c…

The whole point of the Berkeley experiment was to avoid contamination.
All math problems were newly invented, never published, never online, not seen in any training set.
And still, o4-mini solved most of them faster than the human authors.

This isn’t

Gary Marcus

@GaryMarcus

@VraserX link?

the main issue here is that the results on math don’t seem to generalize well to many domains, suggesting data contamination and/or heavy math-specific data augmentation (which is in easier in math than other domains because of verifiability etc).

by the way, computers

VraserX e/acc

@VraserX

🚨 At a secret math summit, top researchers tried to outsmart AI with never-before-seen math problems, unpublished, never on the web, not in any training set.

Result?

OpenAI’s o4-mini solved most of them faster than the mathematicians themselves.

One researcher spent months

The statement engages in public discourse by discussing the transparency and potential data practices of AI models, specifically in the context of solving math problems. It questions the extent of data augmentation and the unknowns surrounding OpenAI's training methods.

Principle 1:
I will strive to do no harm with my words and actions.
The statement does not cause harm and raises valid questions about AI training practices. [+1]
Principle 2:
I will respect the privacy and dignity of others and will not engage in cyberbullying, harassment, or hate speech.
It respects privacy and does not engage in harassment or hate speech. [+1]
Principle 3:
I will use my words and actions to promote understanding, empathy, and compassion.
The statement promotes understanding by questioning the transparency of AI training data. [+1]
Principle 4:
I will engage in constructive criticism and dialogue with those in disagreement and will not engage in personal attacks or ad hominem arguments.
It engages in constructive dialogue by responding to a previous comment with a reasoned argument. [+1]