Gary Marcus

Rank 17 of 47
|
Score 69
9/13/2024, 10:11:02 PM
X
In reply to:
Cameron Williams
@Wasgo
·
343d
@mealreplacer @GaryMarcus @shadbush @colin_fraser The failure points on weird little problems isn't about benchmarking at all though. It's about understanding how the model works, what the limitations are, and why we need to move past generative AI if we want to see real AGI.
Julian
@mealreplacer
·
343d
@GaryMarcus @shadbush @colin_fraser I don’t feel like this is really answering the question I posed.

Would you change your mind if there were difficult benchmarks that are testing LLMs on consequential, real-world tasks, and each generation improved upon the previous?
Gary Marcus
@GaryMarcus
·
343d
@shadbush @mealreplacer @colin_fraser @shadbush is exactly right. I would add that building benchmarks is hard, nobody has ever solved it, but success in bad benchmark doesn’t equate to a solution to the extrapolation problem.

Progress on AI benchmarks has been great, but progress on getting reliable reasoning in…
Aleksandr Tiulkanov
@shadbush
·
344d
@mealreplacer @GaryMarcus @colin_fraser Waiting and hoping for enough examples to be caught in the next training dataset isn't really solving the problem. The fundamental problem is in poor if not inexistent ability to generalise beyond the training dataset.
Julian
@mealreplacer
·
344d
@GaryMarcus @colin_fraser I mean preferably we'd have quantitative benchmarks that don't saturate with each successive generation, yeah?

I'm not sure exactly what I'm supposed to conclude about the trajectory and speed of AI progress by looking at binary examples of LLMs failing at weird little tasks…
Gary Marcus
@GaryMarcus
·
344d
@mealreplacer @colin_fraser because negative examples are very informative about underling mechanism, when you are trying to understand cognitive systems.

The statement '@Wasgo @mealreplacer @shadbush @colin_fraser 💯' is a brief affirmation or agreement with the preceding conversation. It does not substantively engage with the public issues being discussed, such as the limitations of generative AI, the need for better benchmarks, or the trajectory of AI progress. Therefore, it does not constitute public discourse.