Gary Marcus

Rank 20 of 47
|
Score 52
9/13/2024, 9:41:00 PM
X
In reply to:
Aleksandr Tiulkanov
@shadbush
·
344d
@mealreplacer @GaryMarcus @colin_fraser Waiting and hoping for enough examples to be caught in the next training dataset isn't really solving the problem. The fundamental problem is in poor if not inexistent ability to generalise beyond the training dataset.
Julian
@mealreplacer
·
344d
@GaryMarcus @colin_fraser I mean preferably we'd have quantitative benchmarks that don't saturate with each successive generation, yeah?

I'm not sure exactly what I'm supposed to conclude about the trajectory and speed of AI progress by looking at binary examples of LLMs failing at weird little tasks…
Gary Marcus
@GaryMarcus
·
344d
@mealreplacer @colin_fraser because negative examples are very informative about underling mechanism, when you are trying to understand cognitive systems.
Julian
@mealreplacer
·
344d
@GaryMarcus @colin_fraser "solving specific puzzles that have been widely commented on is not really the best test" I agree, but then why do you seem to routinely tweet out negative examples as though that is useful for making sense of the trajectory of AI progress?
Gary Marcus
@GaryMarcus
·
344d
@mealreplacer $5 says @colin_fraser can break it by morning. but more than that, solving specific puzzles that have been widely commented on is not really the best test.
Julian
@mealreplacer
·
344d

The statement engages in a substantive discussion about the challenges and progress in AI benchmarks and reasoning, contributing to the broader discourse on AI development and evaluation. The tone is constructive and aims to add to the conversation by acknowledging the difficulty of building benchmarks and the limitations of current AI systems in generalizing beyond training datasets.

  1. Principle 1:
    I will strive to do no harm with my words and actions.
    The statement strives to do no harm and focuses on a technical discussion rather than personal attacks or harmful rhetoric. It acknowledges the complexity of the issue and adds to the conversation constructively. [+1]
  2. Principle 3:
    I will use my words and actions to promote understanding, empathy, and compassion.
    The statement promotes understanding by recognizing the progress made in AI benchmarks while also pointing out the ongoing challenges. It aims to foster a deeper understanding of the issues at hand. [+1]
  3. Principle 4:
    I will engage in constructive criticism and dialogue with those in disagreement and will not engage in personal attacks or ad hominem arguments.
    The statement engages in constructive dialogue by building on previous points made in the conversation and adding a nuanced perspective. It does not engage in personal attacks or ad hominem arguments. [+1]