Gary Marcus

Rank 14 of 47
|
Score 83
Gary Marcus
@GaryMarcus
·
376d
The replies in this thread shows that there are lots of people *within the ML community* raising the same concerns re how OpenAI discussed ARC that I was making. (And more concerns, e.g, in important comments about how the graph was labeled).

Importantly, this an issue about…
12/22/2024, 2:57:37 AM
X
In reply to:
Brydon Eastman
@brhydon
·
376d
@GaryMarcus I saw the poll and was surprised at how biased the framing was. That poll wouldnt have made it past my undergrad survey review. How many of those respondents have read the paper? Do you really think your audienceship is unbiased? Or a group of people familiar with Tr/Te splits?
Gary Marcus
@GaryMarcus
·
376d
@brhydon I am upset with what Sam said. I did a poll and many viewers did not come away with an accurate impression of what actually transpired.
Brydon Eastman
@brhydon
·
376d
@GaryMarcus Professor Marcus, you're upset that we used the evaluation EXACTLY as designed and indicated by the original paper.

If someone watched that stream and feels mislead, then they never read the paper.

There is zero dishonesty here and its disrespectful to insinuate otherwise.
Gary Marcus
@GaryMarcus
·
376d
@brhydon ARC folks, who you now have a partnership may have ok’d it but

1. it’s more like studying for the SAT then reading a book. It’s learning about the idiosyncrasies of the questions. we have seen this many times in benchmarks.

2. am pretty sure humans could do well on ARC by…
Brydon Eastman
@brhydon
·
376d
@GaryMarcus It's truly no different than reading a text book and then going to do a test.

There is no scandal and nothing up our sleeves - the Arc agi folks explicitly OK this.
Brydon Eastman
@brhydon
·
376d
@GaryMarcus it's the TRAIN set that was used not the TEST set. And it wasn't even the FULL train set.

The format for Arc is very bespoke and requires some explanation, using 3/4 of the TRAIN set we demonstrated the weird JSON format and let the thing fly on the PRIVATE TEST set.

The statement by @GaryMarcus constitutes public discourse as it engages in a substantive discussion about the presentation and interpretation of data within the machine learning community, specifically regarding the ARC (AI Readiness Challenge) and its implications. The tone is one of urging for understanding and dialogue, suggesting a desire to clarify misunderstandings and promote a more accurate interpretation of the events discussed.

  1. Principle 1:
    I will strive to do no harm with my words and actions.
    The statement does not appear to cause harm and instead seeks to clarify misunderstandings, aligning with the principle of doing no harm. [+1]
  2. Principle 3:
    I will use my words and actions to promote understanding, empathy, and compassion.
    The statement encourages reading comments from others to gain a broader understanding, promoting empathy and understanding. [+1]
  3. Principle 4:
    I will engage in constructive criticism and dialogue with those in disagreement and will not engage in personal attacks or ad hominem arguments.
    The statement engages in constructive dialogue by urging others to consider additional perspectives, avoiding personal attacks. [+1]