Gary Marcus

Rank 14 of 47
|
Score 83
Gary Marcus
@GaryMarcus
·
246d
Another abject fail at math and world modeling, with ChatGPT (o3 might do better but it is still shocking).
5/18/2025, 5:43:18 AM
X
In reply to:
Keiran Harris
@KeiranJHarris
·
246d
@GaryMarcus @justjoshinyou13 @pli_cachete This is a totally reasonable question, but surely you'd agree it's a completely different message / tone vs.

"You have got to be kidding me. I first wrote about how (certain kinds) of neural nets had trouble with concepts like a pair and three of kind in … 1998. Here’s o3,
Gary Marcus
@GaryMarcus
·
246d
@justjoshinyou13 @pli_cachete Fine but why can’t render what it says?

And it certainly makes illegal moves in chess.
Josh You
@justjoshinyou13
·
246d
@GaryMarcus @pli_cachete 4o can list accurate poker hands
Josh You
@justjoshinyou13
·
246d
@GaryMarcus @pli_cachete without qualifying whether or not they can solve that problem with a text answer
Josh You
@justjoshinyou13
·
246d
@GaryMarcus @pli_cachete It's fine to test image generation if you want to understand the capabilities or deficiencies of image generation.

It's misleading to test image generation to answer questions that LLMs can answer perfectly fine via text, and then say LLMs don't understand that concept.
Gary Marcus
@GaryMarcus
·
246d
@justjoshinyou13 @pli_cachete I have tried multiple models and presented data (my own and others).

There is nothing wrong with testing multimodal image generation.

The statement critiques the performance of a multimodal model in understanding and rendering poker hands, specifically a straight flush. It suggests that the model's failure is being excused rather than addressed.

  1. Principle 1:
    I will strive to do no harm with my words and actions.
    The statement is critical but does not appear to be harmful. It focuses on the model's performance rather than attacking individuals.
  2. Principle 4:
    I will engage in constructive criticism and dialogue with those in disagreement and will not engage in personal attacks or ad hominem arguments.
    The statement engages in constructive criticism by pointing out perceived shortcomings in the model's capabilities, though it could be more diplomatic. [+1]