Gary Marcus

Rank 14 of 47
|
Score 83
12/21/2024, 8:27:52 PM
X
In reply to:
Dylan Wolfe
@DylanWolfe1
·
369d
@GaryMarcus @truth_zeeker @fchollet explain how it went from 2% sota for o1 to 25% for o3 on the new Math benchmark that was supposed to be resistant for years to ai systems and came out only a little over a month ago…
Gary Marcus
@GaryMarcus
·
369d
@truth_zeeker @fchollet we have seen this movie way too many times. little quirks in how questions were constructed may carry over.

at the very least, not enough transparency in the live feed.

and most likely the results just aren’t as robust as they appear, given other tasks getting at some kind of…
Tesla addict
@truth_zeeker
·
369d
@GaryMarcus @fchollet Trained on 75% on the public set but 0% of the semi-private set yet still 75% score on the semi-private in low compute, and 87% in high compute.
Gary Marcus
@GaryMarcus
·
369d
Update: It IS true, from @fchollet’s blog. 𝗢𝟯 𝘄𝗮𝘀 𝘁𝗿𝗮𝗶𝗻𝗲𝗱 𝗼𝗻 𝟳𝟱% 𝗼𝗳 𝘁𝗵𝗲 𝗽𝘂𝗯𝗹𝗶𝗰 𝘀𝗲𝘁 𝗳𝗼𝗿 𝗔𝗥𝗖-𝗔𝗚𝗜.

Yesterday’s announcement shows what you can do when teach to the test and spend enormous amounts of money.

How much the demo says about real…
Gary Marcus
@GaryMarcus
·
369d
Wow. This, if true, raises serious questions about yesterday’s announcement.

The statement is part of a technical discussion about AI model performance and transparency. It speculates on the methods used to achieve significant improvements in AI benchmarks, suggesting data augmentation and symbolic verification as possible techniques. The tone is analytical and speculative, contributing to a broader conversation about AI transparency and robustness.

  1. Principle 1:
    I will strive to do no harm with my words and actions.
    The statement does not cause harm and is focused on technical analysis. [+1]
  2. Principle 2:
    I will respect the privacy and dignity of others and will not engage in cyberbullying, harassment, or hate speech.
    It respects privacy and dignity, focusing on technical aspects rather than personal attacks. [+1]
  3. Principle 4:
    I will engage in constructive criticism and dialogue with those in disagreement and will not engage in personal attacks or ad hominem arguments.
    The statement engages in constructive dialogue, offering a hypothesis in response to a technical question. [+1]