Gary Marcus

Rank 13 of 47
|
Score 83
12/22/2024, 4:38:25 AM
X
In reply to:
Olcan
@olcan
·
387d
@GaryMarcus @mikeknoop @doomie @circlerotator @jpineau1 my main issue is that the chart published by @fchollet on the arc foundation blog greatly exaggerates the jump ("breakthrough") to o3 series. we already know tuning takes llama 3 8b model from 9% to 45%, see figures 1 and 7 in https://arxiv.org/pdf/2411.07279
Gary Marcus
@GaryMarcus
·
387d
@olcan @mikeknoop @doomie @circlerotator My own thoughts below, referring to this thread. I appreciate people like @doomie speaking up from within the ML community to insist on transparency and proper science.

Think for a minute about @jpineau1’s Replicability Checklist. There is a reason why thrown together…
Gary Marcus
@GaryMarcus
·
387d
The replies in this thread shows that there are lots of people *within the ML community* raising the same concerns re how OpenAI discussed ARC that I was making. (And more concerns, e.g, in important comments about how the graph was labeled).

Importantly, this an issue about…
Olcan
@olcan
·
387d
@mikeknoop @doomie @circlerotator it just seems odd to put tuned methods and untuned models (or even differently tuned models) on the same chart since tuning makes such a huge difference; i mean there is a very similar looking chart in the TTT paper based on tuning ablations alone
Mike Knoop
@mikeknoop
·
387d
@doomie @circlerotator ARC Prize foundation is a nonprofit
Dumitru Erhan
@doomie
·
387d
@circlerotator @mikeknoop didn't realize the ARC folks are a business
shaperotator (e/λ)
@circlerotator
·
387d
@doomie @mikeknoop they just wanted to release in time for christmas, it's a business not a college social club

The statement is a brief acknowledgment or notification directed at specific individuals, likely to draw their attention to the ongoing discussion about the accuracy and representation of data in machine learning models. It does not provide substantive engagement with the public issue itself, but rather serves as a part of a larger conversation about transparency and accuracy in the field of machine learning.

  1. Principle 1:
    I will strive to do no harm with my words and actions.
    The statement itself does not cause harm, but it is neutral and does not actively promote understanding or empathy.
  2. Principle 2:
    I will respect the privacy and dignity of others and will not engage in cyberbullying, harassment, or hate speech.
    The statement respects privacy and dignity, as it is a simple notification without any personal attacks or hate speech.
  3. Principle 3:
    I will use my words and actions to promote understanding, empathy, and compassion.
    The statement does not actively promote understanding, empathy, or compassion, as it is merely a notification.
  4. Principle 4:
    I will engage in constructive criticism and dialogue with those in disagreement and will not engage in personal attacks or ad hominem arguments.
    The statement does not engage in constructive criticism or dialogue, as it is a simple notification without any argumentative content.
  5. Principle 5:
    I will acknowledge and correct my mistakes.
    The statement does not involve any acknowledgment or correction of mistakes.
  6. Principle 6:
    I will use my influence for the betterment of society.
    The statement does not use influence for societal betterment, as it is a neutral notification.
  7. Principle 7:
    I will uphold the principles of free speech and use my platform responsibly and with integrity.
    The statement upholds free speech principles by participating in a public discussion, but it does not use the platform to substantively address the issue.