Gary Marcus

Rank 18 of 47

Score 55

@bob_mankoff With almost two decades of freely available archived puzzles (and no test of whether this specific problem was in the training set, or of reliability across problems, or tests of slightly different problems etc), I am not sure we should be impressed.

https://puzzlemadness.co.uk/arch…

1/5/2025, 6:55:17 PM

In reply to:

Robert Mankoff

@bob_mankoff

321d

@GaryMarcus It turns out all the ChatGPT models can do this now, and 3.5 Sonnet and the Geminis can't. Interestingly, 4o seems to be taking time"thinking" a la o1 and and o1 mini

Robert Mankoff

@bob_mankoff

321d

@GaryMarcus o1 can now solve Suko puzzles through logical steps, not brute force.

Gary Marcus

@GaryMarcus

321d

All this is incredible - but does any of it work reliably?

Wre are going to see a lot of amazing demos this year. Few of them will be trustworthy.

Be careful of enthusiasts that never probe these applications to see if they can safely be used in the real world. (Anyone remember…

Min Choi

@minchoi

321d

Gemini 2.0 has really changed the AI game.

People are using it in wild ways beyond ChatGPT. There's a shift.

8 examples:

The statement questions the reliability and impressiveness of AI models solving puzzles, suggesting skepticism about their capabilities due to lack of testing and potential overfitting. It engages in public discourse by addressing the broader issue of AI reliability and trustworthiness.

Principle 1:
I will strive to do no harm with my words and actions.
The statement does not cause harm and encourages critical evaluation of AI capabilities. [+1]
Principle 4:
I will engage in constructive criticism and dialogue with those in disagreement and will not engage in personal attacks or ad hominem arguments.
It engages in constructive criticism by questioning the reliability of AI models without resorting to personal attacks. [+1]