Gary Marcus

Rank 17 of 47
|
Score 71
9/13/2024, 10:19:09 PM
X
In reply to:
Julian
@mealreplacer
·
343d
@Wasgo @GaryMarcus @shadbush @colin_fraser You really can’t just merely assert that failing on weird little problems is something that is strictly necessary to be solve before we get “real AGI”
Cameron Williams
@Wasgo
·
343d
@mealreplacer @GaryMarcus @shadbush @colin_fraser The failure points on weird little problems isn't about benchmarking at all though. It's about understanding how the model works, what the limitations are, and why we need to move past generative AI if we want to see real AGI.
Julian
@mealreplacer
·
343d
@GaryMarcus @shadbush @colin_fraser I don’t feel like this is really answering the question I posed.

Would you change your mind if there were difficult benchmarks that are testing LLMs on consequential, real-world tasks, and each generation improved upon the previous?
Gary Marcus
@GaryMarcus
·
343d
@shadbush @mealreplacer @colin_fraser @shadbush is exactly right. I would add that building benchmarks is hard, nobody has ever solved it, but success in bad benchmark doesn’t equate to a solution to the extrapolation problem.

Progress on AI benchmarks has been great, but progress on getting reliable reasoning in…
Aleksandr Tiulkanov
@shadbush
·
344d
@mealreplacer @GaryMarcus @colin_fraser Waiting and hoping for enough examples to be caught in the next training dataset isn't really solving the problem. The fundamental problem is in poor if not inexistent ability to generalise beyond the training dataset.
Julian
@mealreplacer
·
344d
@GaryMarcus @colin_fraser I mean preferably we'd have quantitative benchmarks that don't saturate with each successive generation, yeah?

I'm not sure exactly what I'm supposed to conclude about the trajectory and speed of AI progress by looking at binary examples of LLMs failing at weird little tasks…

The statement constitutes public discourse as it engages in a substantive discussion about the limitations and challenges of artificial general intelligence (AGI) and the reliability of AI systems. The tone is critical and aims to highlight the importance of addressing fundamental issues in AI before achieving AGI.

  1. Principle 1:
    I will strive to do no harm with my words and actions.
    The statement does not appear to cause harm with words and actions. It is a critique of AI systems rather than individuals. [+1]
  2. Principle 2:
    I will respect the privacy and dignity of others and will not engage in cyberbullying, harassment, or hate speech.
    The statement respects the privacy and dignity of others. It does not engage in cyberbullying, harassment, or hate speech. [+1]
  3. Principle 3:
    I will use my words and actions to promote understanding, empathy, and compassion.
    The statement aims to promote understanding of the limitations of AI systems, which can be seen as an attempt to foster a more informed discussion. [+1]
  4. Principle 4:
    I will engage in constructive criticism and dialogue with those in disagreement and will not engage in personal attacks or ad hominem arguments.
    The statement engages in constructive criticism of AI systems and does not resort to personal attacks or ad hominem arguments. [+1]
  5. Principle 7:
    I will uphold the principles of free speech and use my platform responsibly and with integrity.
    The statement upholds the principles of free speech and uses the platform responsibly to discuss a significant issue in AI. [+1]