Gary Marcus

Rank 33 of 47
|
Score -1
Sauers
@Sauers_
·
151d
@GaryMarcus The t-test doesn't say that results are not different. It measures if you have evidence to reject the null: p>0.05 does not mean you can reject the alternative hypothesis that there are differences.
6/20/2024, 5:26:55 PM
X
In reply to:
JustArando e/calc
@JustArando82402
·
151d
@GaryMarcus You've got a reference to said T-Test?
Gary Marcus
@GaryMarcus
·
151d
When did AI stop being a science? You can’t conclude that Claude 3.5 is “better than 4o” when there are no error bars, and GPT 4o actually did better than Claude in 2 of the 6 comparisons. (On a t-test someone ran, they aren’t even different).*

And no, a single anecdotal physics…
Deedy
@deedydas
·
151d
Claude 3.5 is here. Sonnet is the first release and has:
— 2x the speed of Opus
— 1/5th the cost of Opus
— 200k token context window
— Better quality than Opus and GPT-4o

I don't trust benchmarks so I tried a Physics q that GPT 4-o failed and Sonnet nailed it.

Insane launch.

The statement and the conversation it is part of engage in a substantive discussion about the performance and evaluation of AI models, specifically focusing on statistical significance and the interpretation of test results. This constitutes public discourse as it addresses the reliability and validity of AI benchmarks, which is a relevant issue in the field of AI and technology.

  1. Principle 1:
    I will strive to do no harm with my words and actions.
    The statement does not appear to cause harm. It is a technical discussion about AI model performance and statistical tests, which is a neutral topic.
  2. Principle 3:
    I will use my words and actions to promote understanding, empathy, and compassion.
    The statement promotes understanding by engaging in a detailed discussion about statistical significance and the interpretation of test results. This helps clarify misconceptions and provides a deeper insight into the evaluation of AI models. [+2]
  3. Principle 4:
    I will engage in constructive criticism and dialogue with those in disagreement and will not engage in personal attacks or ad hominem arguments.
    The statement engages in constructive dialogue by asking for references and clarifications. It does not engage in personal attacks or ad hominem arguments, focusing instead on the technical aspects of the discussion. [+2]