Gary Marcus

Rank 13 of 47
|
Score 115
9/1/2025, 5:58:53 PM
X
In reply to:
Jon Bateman
@JonKBateman
·
13d
@GaryMarcus @peterwildeford Three questions about the famous METR graph, which I haven't seen discussed at all:

1. It measures task length at 50% accuracy. For economically valuable work, we'd often be more interested in 90-99% accuracy, but METR has said this is methodologically difficult to assess. So we
Gary Marcus
@GaryMarcus
·
13d
@peterwildeford https://garymarcus.substack.com/…
Peter Wildeford🇺🇸🚀
@peterwildeford
·
13d
@GaryMarcus do you have a link to a particular article on it? would be happy to check it out

I agree misinterpretations are common, I tried to put this front and center in my post
Gary Marcus
@GaryMarcus
·
13d
@peterwildeford and see my own substack on this benchmark. i am really not fond of it, and even less fond of a lot of the interpretations i see (esp the ones that extend beyond software which is of course a specific focus of the LLM companies).
Peter Wildeford🇺🇸🚀
@peterwildeford
·
13d
@GaryMarcus Hi Gary! There's an awful lot of nuance to unpack here that doesn't fit easily in the character limit. I have so much to say about it!

But some quick replies:

* the time horizon was ~2hrs actually, I realize the graph is small so that's hard to read

* Definitely far from doing
Gary Marcus
@GaryMarcus
·
13d
@peterwildeford do you really think gpt-5 can do anything a human can in 4 hours with 50% accuracy?

and what would the curve be for 100% accuracy?

and what is the value of 50% accuracy?

The statement 'yep' by @JonKBateman is a brief affirmation in a conversation about the interpretation and implications of AI scaling graphs. The conversation involves a discussion on the accuracy and economic value of AI tasks, which is a substantive public issue related to technology and its societal impact. However, the single-word response 'yep' does not provide substantive engagement or analysis on the topic, thus it does not constitute public discourse.