Gary Marcus

Rank 17 of 47
|
Score 61
6/18/2024, 2:37:14 PM
X
In reply to:
AI Notkilleveryoneism Memes ⏸️
@AISafetyMemes
·
502d
@GaryMarcus You don't see data showing models have rapidly been getting better?
Gary Marcus
@GaryMarcus
·
502d
@AISafetyMemes I don’t really see the data for that, and I still tons of hallucinations, documented here, in empirical papers, reports in corporate use etc. Certainly they have not reduced to expert/levels as Reid Hoffman promised, and we are still waiting on GPT-5 level models.
AI Notkilleveryoneism Memes ⏸️
@AISafetyMemes
·
502d
@GaryMarcus Thanks, but this doesn't address my core argument that models are improving rapidly at everything, including hallucinations.
Gary Marcus
@GaryMarcus
·
502d
@AISafetyMemes Yes there are more tools; integration with the tools has been problematic though. I certainly think that integrating w symbolic tools would be good if it can be done. But so far that has been shaky.

You can’t just say you personally don’t care about hallucinations; that is a…
AI Notkilleveryoneism Memes ⏸️
@AISafetyMemes
·
502d
@GaryMarcus 1: Yes, but the trend is clear that more compute = smarter models. GPT4 is far more capable than a high school student in the ways that I care about.

I don't care that you can trick it with logic puzzles more often (which I'm not even entirely sure is true) or that it…
Gary Marcus
@GaryMarcus
·
503d
Soooo very wrong, in so many ways
1. That chart is make-believe, with wacky y-axes and totally hypothetical data. And no GPT-4 is not really a substitute for a smart high school student, c’mon.
2. Not much progress on raw capabilities since GPT-4 (trained Aug 2022, almost two…

The statement by @AISafetyMemes critiques the validity and reliability of benchmarks used to measure AI performance. It argues that benchmarks are often gamed, suffer from data contamination, and do not represent valid constructs in a psychological sense. The tone is critical and aims to highlight perceived flaws in the current evaluation methods for AI models.

  1. Principle 1:
    I will strive to do no harm with my words and actions.
    The statement strives to do no harm by focusing on the technical aspects of AI benchmarks rather than attacking individuals. [+1]
  2. Principle 2:
    I will respect the privacy and dignity of others and will not engage in cyberbullying, harassment, or hate speech.
    The statement respects the privacy and dignity of others by not engaging in personal attacks or harassment. [+1]
  3. Principle 3:
    I will use my words and actions to promote understanding, empathy, and compassion.
    The statement promotes understanding by bringing attention to potential issues with benchmarks, encouraging a more nuanced view of AI performance. [+1]
  4. Principle 4:
    I will engage in constructive criticism and dialogue with those in disagreement and will not engage in personal attacks or ad hominem arguments.
    The statement engages in constructive criticism by addressing specific problems with benchmarks, although it could be more balanced by acknowledging any strengths or improvements in the field.