Gary Marcus

Rank 33 of 47

Score -1

@Sauers_ @bilawalsidhu 1. I was asking about the individual test differences, and we don’t have error bars or any other measure of variability there.

2: Not that even on the test you are doing the t-test vs 4-o is NOT significant. And there is no reason to do a 1-tailed test so not even close.

3.…

6/20/2024, 3:59:03 PM

In reply to:

Sauers

@Sauers_

152d

@GaryMarcus @bilawalsidhu Yeah most of these differences are statistically significant, even with only these few datapoints (the % on benchmarks)

Gary Marcus

@GaryMarcus

152d

@bilawalsidhu Are these differences even statistically significant? (And how much is teaching ti the test? and are the small gains economically worthwhile if the models get bigger?)

Bilawal Sidhu

@bilawalsidhu

152d

We really gonna be switching LLMs on a bi-weekly basis at this point lol

The statement engages in a technical discussion about statistical significance and the appropriateness of certain statistical tests in the context of comparing different language models. The tone is analytical and aims to clarify the validity of the statistical methods used.

Principle 1:
I will strive to do no harm with my words and actions.
The statement strives to do no harm and focuses on a technical critique rather than personal attacks, adhering to the principle of doing no harm with words and actions. [+1]
Principle 3:
I will use my words and actions to promote understanding, empathy, and compassion.
The statement promotes understanding by addressing specific statistical concerns, thereby contributing to a more informed discussion. [+1]
Principle 4:
I will engage in constructive criticism and dialogue with those in disagreement and will not engage in personal attacks or ad hominem arguments.
The statement engages in constructive criticism by questioning the statistical methods used without resorting to personal attacks, adhering to the principle of engaging in constructive dialogue. [+1]