Gary Marcus

Rank 13 of 47
|
Score 83
12/22/2024, 4:12:16 AM
X
In reply to:
Radhika Sharma
@radhika5kh
·
387d
@GaryMarcus The original clip claims (aside from including the training set) they haven't targeted the benchmark using xyz techniques. The arc foundation probably has verified that these results aren't gamed

They're on par with humans for arguably the most challenging reasoning benchmark
Gary Marcus
@GaryMarcus
·
387d
@radhika5kh maybe it, maybe there is a lot of data augmentation, or built-in verification tools etc that aren’t disclosed.

maybe it will carry over to other domains, maybe it won’t.

to the extent that there is a breakthrough, we don’t really know if it pertains to data, or architecture,…
Radhika Sharma
@radhika5kh
·
387d
@GaryMarcus François Chollet's blog states (in the first line) that they used the training set. https://arcprize.org/blog/oai-o3…

Yes "open"ai has never released a paper and is the most opaque lab; this is a huge breakthrough, irrespective of including the training set.
Gary Marcus
@GaryMarcus
·
387d
@radhika5kh see discussion here
Gary Marcus
@GaryMarcus
·
387d
The replies in this thread shows that there are lots of people *within the ML community* raising the same concerns re how OpenAI discussed ARC that I was making. (And more concerns, e.g, in important comments about how the graph was labeled).

Importantly, this an issue about…
Radhika Sharma
@radhika5kh
·
387d
@GaryMarcus What's your point? There's no cross contamination. The results have been verified by the arc foundation 🤔
Gary Marcus
@GaryMarcus
·
387d
@radhika5kh I am aware, thank you.

The statement is part of a public discourse on the performance of AI models compared to human capabilities, specifically in the context of a reasoning benchmark. The tone is analytical and seeks to clarify or challenge claims made about AI performance.

  1. Principle 1:
    I will strive to do no harm with my words and actions.
    The statement does not use harmful language and focuses on factual analysis, adhering to the principle of doing no harm. [+1]
  2. Principle 3:
    I will use my words and actions to promote understanding, empathy, and compassion.
    The statement aims to promote understanding by providing specific points about the AI model's training and performance, aligning with the principle of promoting understanding. [+1]
  3. Principle 4:
    I will engage in constructive criticism and dialogue with those in disagreement and will not engage in personal attacks or ad hominem arguments.
    The statement engages in constructive criticism by questioning the claims about AI performance without resorting to personal attacks, adhering to the principle of constructive dialogue. [+1]