Gary Marcus

Rank 14 of 47
|
Score 83
5/18/2025, 5:27:19 AM
X
In reply to:
Josh You
@justjoshinyou13
·
245d
@GaryMarcus @pli_cachete without qualifying whether or not they can solve that problem with a text answer
Josh You
@justjoshinyou13
·
245d
@GaryMarcus @pli_cachete It's fine to test image generation if you want to understand the capabilities or deficiencies of image generation.

It's misleading to test image generation to answer questions that LLMs can answer perfectly fine via text, and then say LLMs don't understand that concept.
Gary Marcus
@GaryMarcus
·
245d
@justjoshinyou13 @pli_cachete I have tried multiple models and presented data (my own and others).

There is nothing wrong with testing multimodal image generation.
Josh You
@justjoshinyou13
·
245d
@pli_cachete @GaryMarcus testing LLMs on knowledge by asking them to draw pictures is insane.

GPT-4o image gen is probably a lot worse than 2022 LLMs at answering your questions through drawing. according to Gary, 4o therefore experienced a huge regression when OpenAI added image capabilities
Rota
@pli_cachete
·
245d
@GaryMarcus o3 doesn’t generate images, it makes a call to a different model. With how much you discuss this don’t you think it’s worth spending a few minutes learning about them?
Gary Marcus
@GaryMarcus
·
245d
You have got to be kidding me.

I first wrote about how (certain kinds) of neural nets had trouble with concepts like a pair and three of kind in … 1998.

Here’s o3, today:

The statement questions the rationale behind not testing the alleged capabilities of multimodal models, suggesting skepticism about giving them a 'free pass.' It engages in a technical discussion about the evaluation of AI models, specifically regarding their multimodal capabilities.

  1. Principle 1:
    I will strive to do no harm with my words and actions.
    The statement does not cause harm and encourages scrutiny and testing, which is a constructive approach. [+1]
  2. Principle 3:
    I will use my words and actions to promote understanding, empathy, and compassion.
    The statement promotes understanding by questioning the evaluation process of AI models, encouraging a deeper look into their capabilities. [+1]
  3. Principle 4:
    I will engage in constructive criticism and dialogue with those in disagreement and will not engage in personal attacks or ad hominem arguments.
    The statement engages in constructive criticism by questioning the approach without resorting to personal attacks. [+1]