Gary Marcus

Rank 12 of 47
|
Score 83
Gary Marcus
@GaryMarcus
·
223d
o3 failing my new benchmark. See how many errors you can spot.
6/25/2025, 4:23:43 PM
X
In reply to:
Lawrence Chan
@justanotherlaw
·
223d
@GaryMarcus OpenAI released integrated image generation capabilities in March of this year, and it was a pretty big leap forward in image generation capabilities:
https://openai.com/index/introdu…
Lawrence Chan
@justanotherlaw
·
223d
@GaryMarcus My theory for why o3/4o can do it but Gemini + Imagen cannot is because Imagen 4 and Gemini are separate models -- Gemini understands compositionality but can only access Imagen via a function call API, and thus cannot pass its understanding to Imagen 4 via a prompt.
Lawrence Chan
@justanotherlaw
·
223d
@GaryMarcus When I ask Gemini about the horse/astronaut image with the request in context, it seems to double down that it followed instructions correctly, even though when I ask about the image in a new chat it correct identifies the problem.
Lawrence Chan
@justanotherlaw
·
223d
@GaryMarcus I think this is evidence that the Gemini + Imagen 4 combo isn't great at following instructions for some reason. (In the same way that Grok 3 image generation failures are more a sign that Grok 3 wasn't as great as people were hyping it out to be.)
Lawrence Chan
@justanotherlaw
·
223d
@GaryMarcus First try with o3:
Gary Marcus
@GaryMarcus
·
223d
“A horse rides an astronaut”, three years later.

Y’all owe me an apology.

The conversation revolves around the capabilities and limitations of AI models in generating images based on prompts. It discusses the differences between models like GPT-4o, Gemini, and Imagen, highlighting their strengths and weaknesses in understanding and executing instructions. The tone is analytical and focuses on technical evaluation.

  1. Principle 1:
    I will strive to do no harm with my words and actions.
    The discussion is technical and does not cause harm, focusing on AI capabilities. [+1]
  2. Principle 3:
    I will use my words and actions to promote understanding, empathy, and compassion.
    The conversation promotes understanding of AI model differences and their functionalities. [+1]
  3. Principle 4:
    I will engage in constructive criticism and dialogue with those in disagreement and will not engage in personal attacks or ad hominem arguments.
    Engages in constructive criticism of AI models without personal attacks. [+1]