Gary Marcus

Rank 33 of 47
|
Score -1
6/20/2024, 5:26:26 PM
X
In reply to:
Sauers
@Sauers_
·
151d
@GaryMarcus The t-test doesn't say that results are not different. It measures if you have evidence to reject the null: p>0.05 does not mean you can reject the alternative hypothesis that there are differences.
Sauers
@Sauers_
·
151d
@GaryMarcus @bilawalsidhu Yeah most of these differences are statistically significant, even with only these few datapoints (the % on benchmarks)
Gary Marcus
@GaryMarcus
·
151d
When did AI stop being a science? You can’t conclude that Claude 3.5 is “better than 4o” when there are no error bars, and GPT 4o actually did better than Claude in 2 of the 6 comparisons. (On a t-test someone ran, they aren’t even different).*

And no, a single anecdotal physics…
Deedy
@deedydas
·
151d
Claude 3.5 is here. Sonnet is the first release and has:
— 2x the speed of Opus
— 1/5th the cost of Opus
— 200k token context window
— Better quality than Opus and GPT-4o

I don't trust benchmarks so I tried a Physics q that GPT 4-o failed and Sonnet nailed it.

Insane launch.

The statement engages in a technical discussion about the statistical significance of AI model comparisons, specifically addressing the interpretation of t-test results. The tone is analytical and aims to clarify a common misunderstanding in statistical hypothesis testing.

  1. Principle 1:
    I will strive to do no harm with my words and actions.
    The statement strives to do no harm by focusing on a technical clarification rather than personal attacks. [+1]
  2. Principle 3:
    I will use my words and actions to promote understanding, empathy, and compassion.
    The statement promotes understanding by addressing a common statistical misconception, which can help others better interpret scientific results. [+2]
  3. Principle 4:
    I will engage in constructive criticism and dialogue with those in disagreement and will not engage in personal attacks or ad hominem arguments.
    The statement engages in constructive dialogue by providing a reasoned argument without resorting to personal attacks. [+2]