Gary Marcus

Rank 12 of 47
|
Score 83
9/18/2024, 2:38:12 AM
X
In reply to:
Colin Fraser
@colin_fraser
·
494d
@GaryMarcus This is 4o so no CoT, it’s just blurting out the words. There’s certainly some relevant training data but the probability that these particular addition problems are in the training data is basically zero. Somehow it’s sort of adding.
Gary Marcus
@GaryMarcus
·
494d
@colin_fraser There were multiple previous similar studies and there are obviously some relevant data in the corpus. And for 01 there may be synthetic data as well.
Colin Fraser
@colin_fraser
·
494d
Here's a similar experiment I just tried. The fact that this works even a little bit completely blows my mind and confuses me greatly. If you asked me if this would work at all I would say definitely not.
Yuntian Deng
@yuntiandeng
·
495d
Is OpenAI's o1 a good calculator? We tested it on up to 20x20 multiplication—o1 solves up to 9x9 multiplication with decent accuracy, while gpt-4o struggles beyond 4x4. For context, this task is solvable by a small LM using implicit CoT with stepwise internalization. 1/4

The statement and the linked content constitute public discourse as they engage in a substantive discussion about the capabilities and limitations of pretrained language models (LMs) in numerical reasoning tasks. The conversation involves multiple participants sharing experimental results and insights, contributing to the broader dialogue on artificial intelligence and machine learning.

  1. Principle 1:
    I will strive to do no harm with my words and actions.
    The statement and linked content strive to do no harm and focus on sharing knowledge and insights about language models, adhering to the principle of doing no harm. [+2]
  2. Principle 3:
    I will use my words and actions to promote understanding, empathy, and compassion.
    The discussion promotes understanding and empathy by sharing detailed experimental results and analyses, which can help others in the field better understand the capabilities and limitations of LMs. [+2]
  3. Principle 4:
    I will engage in constructive criticism and dialogue with those in disagreement and will not engage in personal attacks or ad hominem arguments.
    The conversation engages in constructive criticism and dialogue, with participants sharing their findings and responding to each other's points without engaging in personal attacks or ad hominem arguments. [+2]
  4. Principle 6:
    I will use my influence for the betterment of society.
    The participants use their influence to contribute to the betterment of society by advancing the understanding of AI and its applications, which aligns with the principle of using influence responsibly. [+2]