Gary Marcus

Rank 21 of 47
|
Score 55
5/2/2024, 4:20:33 AM
X
In reply to:
Wyatt Walls
@lefthanddraft
·
425d
@emollick Interesting in that, even when overfit, LLMs not merely memorizing and can solve similar problems. But it is only assessing generalization to questions of similar style and complexity

Doesn't seek to assess extent of generalizability (eg, to out of distribution questions).
Ethan Mollick
@emollick
·
425d
"We find that frontier models exhibit little to no evidence of overfitting and that many models, even the most heavily overfit families, show strong signs of generalizable mathematical reasoning."
AK
@_akhaliq
·
425d
A Careful Examination of Large Language Model Performance on Grade School Arithmetic

Large language models (LLMs) have achieved impressive success on many benchmarks for mathematical reasoning. However, there is growing concern that some of this performance actually

The statement by @lefthanddraft in response to @emollick's comment about LLMs (Large Language Models) addresses the issue of generalization in AI models, specifically in the context of mathematical reasoning. The discourse is technical and focused on the limitations and capabilities of LLMs in handling out-of-distribution questions, which is a significant topic in AI research and development. The conversation is part of a larger discussion on the performance of LLMs on mathematical tasks, initiated by @_akhaliq's examination of LLM performance on grade school arithmetic.

  1. Principle 1:
    I will strive to do no harm with my words and actions.
    The statement does no harm as it contributes constructively to a technical discussion on AI. [+1]
  2. Principle 2:
    I will respect the privacy and dignity of others and will not engage in cyberbullying, harassment, or hate speech.
    It respects the dignity of others by maintaining a professional and factual tone without any personal attacks or derogatory comments. [+1]
  3. Principle 3:
    I will use my words and actions to promote understanding, empathy, and compassion.
    The comment promotes understanding by clarifying the limitations of LLMs in a specific context, which is beneficial for the audience's comprehension of the topic. [+1]
  4. Principle 4:
    I will engage in constructive criticism and dialogue with those in disagreement and will not engage in personal attacks or ad hominem arguments.
    The statement engages in constructive criticism by pointing out a limitation in the current evaluation methods of LLMs, fostering a dialogue on how to better assess AI capabilities. [+1]
  5. Principle 5:
    I will acknowledge and correct my mistakes.
    The user acknowledges the complexity of the issue and contributes to correcting misunderstandings about LLMs' capabilities. [+1]
  6. Principle 6:
    I will use my influence for the betterment of society.
    By discussing the implications of overfitting and generalization in AI, the statement uses its influence to inform and educate, contributing positively to the field. [+1]
  7. Principle 7:
    I will uphold the principles of free speech and use my platform responsibly and with integrity.
    The user responsibly uses the platform to engage in a substantive discussion about AI, upholding the principles of free speech with integrity. [+1]