Gary Marcus

Rank 33 of 47
|
Score -1
6/20/2024, 4:55:13 PM
X
In reply to:
Adan Z Becerra, PhD
@AdanBecerraPhD
·
151d
@GaryMarcus @bilawalsidhu Statistical significance and the null hypothesis significance framework needs to be retired and you don’t need to use this flawed framework to note that LLMs are plateauing, which they clearly are.
Gary Marcus
@GaryMarcus
·
151d
@AdanBecerraPhD @bilawalsidhu am very well aware of the issues, and have written about them myself, but in the limit complaining them to the point of nihilism just becomes sophistry.
Adan Z Becerra, PhD
@AdanBecerraPhD
·
151d
@GaryMarcus @bilawalsidhu If you want a thorough understanding of challenges with statistical significance, see the petition we signed in Nature below:

And yes, I wasn’t implying that this is decisively better than 4o. Was just noting limitations with p values.

https://www.nature.com/articles/…
Gary Marcus
@GaryMarcus
·
151d
@AdanBecerraPhD @bilawalsidhu that’s a wild and misleading oversimplification of the (real) challenges with p thresholds. but you really can’t make the argument that this is decisively better than 4o, regardless of your philosophy of statistics.
Adan Z Becerra, PhD
@AdanBecerraPhD
·
151d
@GaryMarcus @bilawalsidhu Statistical significance is irrelevant since it is based on an arbitrary threshold of p<0.05.

But I see your point. Are the differences technologically significant is what I would want to know?
Gary Marcus
@GaryMarcus
·
151d
@bilawalsidhu Are these differences even statistically significant? (And how much is teaching ti the test? and are the small gains economically worthwhile if the models get bigger?)

The statement constitutes public discourse as it engages in a substantive discussion about statistical significance, the null hypothesis significance framework, and their implications for understanding the performance of large language models (LLMs). The tone is critical but aims to clarify and improve the discourse by suggesting alternative tools for analysis.

  1. Principle 1:
    I will strive to do no harm with my words and actions.
    The statement does not appear to cause harm with words and actions. It is critical but not harmful. [+1]
  2. Principle 2:
    I will respect the privacy and dignity of others and will not engage in cyberbullying, harassment, or hate speech.
    The statement respects the privacy and dignity of others and does not engage in cyberbullying, harassment, or hate speech. [+1]
  3. Principle 3:
    I will use my words and actions to promote understanding, empathy, and compassion.
    The statement aims to promote understanding by suggesting the use of Bayesian tools, which can be seen as an effort to improve the discourse. [+1]
  4. Principle 4:
    I will engage in constructive criticism and dialogue with those in disagreement and will not engage in personal attacks or ad hominem arguments.
    The statement engages in constructive criticism by pointing out potential misinterpretations and suggesting alternatives. It does not engage in personal attacks or ad hominem arguments. [+2]
  5. Principle 7:
    I will uphold the principles of free speech and use my platform responsibly and with integrity.
    The statement upholds the principles of free speech and uses the platform responsibly by contributing to a meaningful discussion on statistical methods. [+1]