Gary Marcus

Rank 14 of 47

Score 83

@brhydon ARC folks, who you now have a partnership may have ok’d it but

1. it’s more like studying for the SAT then reading a book. It’s learning about the idiosyncrasies of the questions. we have seen this many times in benchmarks.

2. am pretty sure humans could do well on ARC by…

12/21/2024, 10:29:54 PM

In reply to:

Brydon Eastman

@brhydon

372d

@GaryMarcus It's truly no different than reading a text book and then going to do a test.

There is no scandal and nothing up our sleeves - the Arc agi folks explicitly OK this.

Brydon Eastman

@brhydon

372d

@GaryMarcus it's the TRAIN set that was used not the TEST set. And it wasn't even the FULL train set.

The format for Arc is very bespoke and requires some explanation, using 3/4 of the TRAIN set we demonstrated the weird JSON format and let the thing fly on the PRIVATE TEST set.

Gary Marcus

@GaryMarcus

372d

If Taylor Swift lipsynced her way through her Eras Tours, it would be a huge scandal. (And she would never do such a thing.)

If OpenAI gave a big spiel about solving ARC-AGI and didn’t say out loud that they trained on 75% of the public examples, and didn’t dwell on the fact…

The statement engages in a discussion about the transparency and methodology of AI training, specifically related to the ARC dataset. It critiques the approach taken by a company in training AI models, comparing it to test preparation rather than genuine learning. This constitutes public discourse as it addresses issues of transparency and integrity in AI development, which are significant public concerns.

Principle 1:
I will strive to do no harm with my words and actions.
The statement does not appear to cause harm directly, but it does critique a practice, which is a form of constructive criticism rather than harmful speech.
Principle 4:
I will engage in constructive criticism and dialogue with those in disagreement and will not engage in personal attacks or ad hominem arguments.
The statement engages in constructive criticism by questioning the methodology used in AI training. It does not resort to personal attacks or ad hominem arguments, focusing instead on the process and implications. [+1]
Principle 7:
I will uphold the principles of free speech and use my platform responsibly and with integrity.
The statement uses the platform to discuss transparency in AI, which is a responsible use of free speech. It contributes to a broader dialogue on ethical AI practices. [+1]