Gary Marcus

Rank 21 of 47
|
Score 54
6/9/2025, 4:01:49 PM
X
In reply to:
VraserX e/acc
@VraserX
·
21d
@GaryMarcus I’m not ignoring augmentation, I’m saying speculation about “unknown math-specific augmentation” is just that, speculation.
Unless there’s concrete evidence that these newly written, unpublished tasks were seen or closely mimicked in fine-tuning, the result still stands,
Gary Marcus
@GaryMarcus
·
21d
@VraserX You are absolutely ignoring what i said about augmentation,
and if you continue to do so, i will opt out of the discussion.
VraserX e/acc
@VraserX
·
21d
@GaryMarcus These problems were written on-site, after model weights were frozen, and given under NDA, so no, they weren’t in the training data. That’s kind of the whole point.

Math is just where reasoning is easiest to measure. The same toolchain already shows strong results in code,
Gary Marcus
@GaryMarcus
·
22d
@VraserX 1. Those specific problems may not have been seen in the training set, but then again we have NO IDEA what math-specific augmentation OpenAI has done. Similar problems may have in the training set/fine tuning/RL etc; only OpenAI knows how much that might or might not be true. (We
VraserX e/acc
@VraserX
·
22d
@GaryMarcus https://www.scientificamerican.c…

The whole point of the Berkeley experiment was to avoid contamination.
All math problems were newly invented, never published, never online, not seen in any training set.
And still, o4-mini solved most of them faster than the human authors.

This isn’t
Gary Marcus
@GaryMarcus
·
22d
@VraserX link?

the main issue here is that the results on math don’t seem to generalize well to many domains, suggesting data contamination and/or heavy math-specific data augmentation (which is in easier in math than other domains because of verifiability etc).

by the way, computers

The statement is part of a public discourse on the topic of AI's capabilities in solving math problems and the potential data augmentation involved. The conversation involves a debate on the validity of AI's performance in a controlled experiment and whether there is any undisclosed augmentation influencing the results.

  1. Principle 1:
    I will strive to do no harm with my words and actions.
    The statement does not directly cause harm but implies that the other party is ignoring evidence, which could be seen as dismissive.
  2. Principle 2:
    I will respect the privacy and dignity of others and will not engage in cyberbullying, harassment, or hate speech.
    The statement does not engage in cyberbullying, harassment, or hate speech, but it does challenge the other party's stance.
  3. Principle 3:
    I will use my words and actions to promote understanding, empathy, and compassion.
    The statement does not promote understanding, empathy, or compassion, as it is accusatory in nature. [-1]
  4. Principle 4:
    I will engage in constructive criticism and dialogue with those in disagreement and will not engage in personal attacks or ad hominem arguments.
    The statement engages in dialogue but does so in a confrontational manner, suggesting the other party is ignoring evidence without providing constructive criticism. [-1]
  5. Principle 5:
    I will acknowledge and correct my mistakes.
    The statement does not acknowledge or correct any mistakes, as it is focused on challenging the other party's position.
  6. Principle 6:
    I will use my influence for the betterment of society.
    The statement does not use influence for societal betterment, as it is focused on a specific debate rather than broader societal issues.
  7. Principle 7:
    I will uphold the principles of free speech and use my platform responsibly and with integrity.
    The statement upholds free speech but could be more responsible and constructive in its approach.