Gary Marcus

Rank 12 of 47
|
Score 83
9/18/2024, 10:03:57 AM
X
In reply to:
Colin Fraser
@colin_fraser
·
494d
@GaryMarcus I think this is kind of contrary to that result. These examples don’t appear in the training data.
Gary Marcus
@GaryMarcus
·
494d
@colin_fraser https://arxiv.org/abs/2202.07206
Colin Fraser
@colin_fraser
·
494d
@GaryMarcus This is 4o so no CoT, it’s just blurting out the words. There’s certainly some relevant training data but the probability that these particular addition problems are in the training data is basically zero. Somehow it’s sort of adding.
Gary Marcus
@GaryMarcus
·
494d
@colin_fraser There were multiple previous similar studies and there are obviously some relevant data in the corpus. And for 01 there may be synthetic data as well.
Colin Fraser
@colin_fraser
·
494d
Here's a similar experiment I just tried. The fact that this works even a little bit completely blows my mind and confuses me greatly. If you asked me if this would work at all I would say definitely not.
Yuntian Deng
@yuntiandeng
·
495d
Is OpenAI's o1 a good calculator? We tested it on up to 20x20 multiplication—o1 solves up to 9x9 multiplication with decent accuracy, while gpt-4o struggles beyond 4x4. For context, this task is solvable by a small LM using implicit CoT with stepwise internalization. 1/4

The statement engages in a technical discussion about the training data and augmentation methods used in machine learning models, specifically in the context of numerical reasoning capabilities. The tone is analytical and seeks to clarify uncertainties regarding the data used in the training process.

  1. Principle 1:
    I will strive to do no harm with my words and actions.
    The statement strives to do no harm and is focused on a technical discussion, thus adhering to this principle. [+1]
  2. Principle 2:
    I will respect the privacy and dignity of others and will not engage in cyberbullying, harassment, or hate speech.
    The statement respects the privacy and dignity of others, as it does not engage in cyberbullying, harassment, or hate speech. [+1]
  3. Principle 3:
    I will use my words and actions to promote understanding, empathy, and compassion.
    The statement promotes understanding by addressing uncertainties and providing insights into the training data and augmentation methods. [+1]
  4. Principle 4:
    I will engage in constructive criticism and dialogue with those in disagreement and will not engage in personal attacks or ad hominem arguments.
    The statement engages in constructive dialogue, responding to a previous comment with additional information and clarification. [+1]