Gary Marcus

Rank 13 of 47
|
Score 83
12/22/2024, 3:41:16 AM
X
In reply to:
Olcan
@olcan
·
386d
@GaryMarcus @mikeknoop @doomie @circlerotator @jpineau1 @fchollet to be clear i would have no issue with this being on the openai blog. that's a marketing blog and of course they want to present their work in the best possible light. it just does not belong on the arc foundation blog imo
Olcan
@olcan
·
386d
@GaryMarcus @mikeknoop @doomie @circlerotator @jpineau1 my main issue is that the chart published by @fchollet on the arc foundation blog greatly exaggerates the jump ("breakthrough") to o3 series. we already know tuning takes llama 3 8b model from 9% to 45%, see figures 1 and 7 in https://arxiv.org/pdf/2411.07279
Gary Marcus
@GaryMarcus
·
386d
@olcan @mikeknoop @doomie @circlerotator My own thoughts below, referring to this thread. I appreciate people like @doomie speaking up from within the ML community to insist on transparency and proper science.

Think for a minute about @jpineau1’s Replicability Checklist. There is a reason why thrown together…
Gary Marcus
@GaryMarcus
·
386d
The replies in this thread shows that there are lots of people *within the ML community* raising the same concerns re how OpenAI discussed ARC that I was making. (And more concerns, e.g, in important comments about how the graph was labeled).

Importantly, this an issue about…
Olcan
@olcan
·
386d
@mikeknoop @doomie @circlerotator it just seems odd to put tuned methods and untuned models (or even differently tuned models) on the same chart since tuning makes such a huge difference; i mean there is a very similar looking chart in the TTT paper based on tuning ablations alone
Mike Knoop
@mikeknoop
·
386d
@doomie @circlerotator ARC Prize foundation is a nonprofit
Dumitru Erhan
@doomie
·
386d
@circlerotator @mikeknoop didn't realize the ARC folks are a business

The statement is part of a broader discussion about transparency and representation in scientific communication, specifically regarding the presentation of data in the machine learning community. It critiques the omission of legitimate competitors in a chart, suggesting a lack of transparency or fairness.

  1. Principle 1:
    I will strive to do no harm with my words and actions.
    The statement does not use harmful language and focuses on critiquing the presentation of data rather than individuals. [+1]
  2. Principle 2:
    I will respect the privacy and dignity of others and will not engage in cyberbullying, harassment, or hate speech.
    The statement respects the privacy and dignity of others, focusing on the issue rather than personal attacks. [+1]
  3. Principle 3:
    I will use my words and actions to promote understanding, empathy, and compassion.
    The statement aims to promote understanding by highlighting the importance of including all relevant competitors in scientific communication. [+1]
  4. Principle 4:
    I will engage in constructive criticism and dialogue with those in disagreement and will not engage in personal attacks or ad hominem arguments.
    The statement engages in constructive criticism by addressing the issue of data presentation without resorting to personal attacks. [+1]
  5. Principle 7:
    I will uphold the principles of free speech and use my platform responsibly and with integrity.
    The statement upholds free speech by responsibly critiquing the presentation of data, contributing to a transparent and informed public discourse. [+1]