Gary Marcus

Rank 33 of 47

Score -1

@Sauers_ @bilawalsidhu I don’t buy your argument on 2; in advance of running the experiment (build model and test) you don’t really know which way to run the hypothesis.
3. You can have either Wilcoxon or t-test but not both, and you need to decide in advance and surely they aren’t fully independent…

6/20/2024, 5:29:25 PM

In reply to:

Sauers

@Sauers_

151d

@GaryMarcus @bilawalsidhu 1. Yes, it would be great if we had that. I'll run it again if it's released.
2. I am testing if 3.5 is better than 4o, so a 2-tailed would be incorrect for that hypothesis.
3. Both Wilcoxon and t-test are reasonable. I did not test assumptions, so I provided both. Not p-hacking.

Gary Marcus

@GaryMarcus

151d

@Sauers_ @bilawalsidhu 1. I was asking about the individual test differences, and we don’t have error bars or any other measure of variability there.

2: Not that even on the test you are doing the t-test vs 4-o is NOT significant. And there is no reason to do a 1-tailed test so not even close.

3.…

Sauers

@Sauers_

151d

@GaryMarcus @bilawalsidhu Yeah most of these differences are statistically significant, even with only these few datapoints (the % on benchmarks)

Gary Marcus

@GaryMarcus

151d

@bilawalsidhu Are these differences even statistically significant? (And how much is teaching ti the test? and are the small gains economically worthwhile if the models get bigger?)

Bilawal Sidhu

@bilawalsidhu

151d

We really gonna be switching LLMs on a bi-weekly basis at this point lol

The statement engages in a technical discussion about statistical methods and hypothesis testing in the context of evaluating machine learning models. The tone is critical but focused on the substance of the argument, aiming to clarify and correct methodological points.

Principle 1:
I will strive to do no harm with my words and actions.
The statement strives to do no harm and focuses on the technical aspects of the argument rather than personal attacks, adhering to the principle of doing no harm with words and actions. [+1]
Principle 3:
I will use my words and actions to promote understanding, empathy, and compassion.
The statement promotes understanding by addressing specific methodological concerns, which can help others in the discussion better understand the nuances of statistical testing. [+1]
Principle 4:
I will engage in constructive criticism and dialogue with those in disagreement and will not engage in personal attacks or ad hominem arguments.
The statement engages in constructive criticism by challenging the methodological choices made by others, without resorting to personal attacks or ad hominem arguments. [+1]