Gary Marcus

Rank 13 of 47
|
Score 84
11/13/2024, 3:27:06 AM
X
In reply to:
Garrison Lovely
@GarrisonLovely
·
457d
There's a lot more in the full piece: https://garrisonlovely.substack.…

Please check it out and subscribe!
Garrison Lovely
@GarrisonLovely
·
457d
After all, the Department of Defense has something in common with venture-backed AI startups: both can receive billions without turning a profit. The main difference is the DoD can tack on a few extra zeroes.
Garrison Lovely
@GarrisonLovely
·
457d
What does this mean for AI progress? Companies like OpenAI are burning billions with an uncertain path to profitability. But governments are getting interested, and they don't need to turn a profit.
Garrison Lovely
@GarrisonLovely
·
457d
The signs were there: marginal improvements since GPT-4, running out of quality training data, and limits to what next-word prediction can achieve, as I wrote back in June:
Garrison Lovely
@GarrisonLovely
·
610d
1. We haven't seen anything more than marginal improvements in the year+ since GPT-4
2. Rumors from people in the labs
3. Running out of good training data
4. Maybe some intrinsic limits in how far next-word prediction + RLHF can get you (though RL w/ self-play may totally...
Garrison Lovely
@GarrisonLovely
·
457d
Even so, we can see that the costs increase exponentially while performance improves linearly i.e. we can build smarter AI, but like quantum computers or fusion reactors, making it economically viable is another story.
Garrison Lovely
@GarrisonLovely
·
457d
OpenAI's o1 model offers a potential way forward: having models "think longer" on tasks. But there's a catch — the economics might not work.

One big tell is that OpenAI hid the absolute values on the x-axis of their key chart!

The statement engages in public discourse by discussing the progress and limitations of large language models (LLMs) and deep learning, which are significant topics in AI development. It critiques the notion of exponential advancement in AI, suggesting diminishing returns and economic challenges.

  1. Principle 1:
    I will strive to do no harm with my words and actions.
    The statement strives to do no harm by focusing on technical and economic aspects of AI development rather than personal attacks. [+1]
  2. Principle 3:
    I will use my words and actions to promote understanding, empathy, and compassion.
    It promotes understanding by providing evidence and analysis on the current state of AI, encouraging informed discussion. [+2]
  3. Principle 4:
    I will engage in constructive criticism and dialogue with those in disagreement and will not engage in personal attacks or ad hominem arguments.
    The statement engages in constructive criticism by questioning prevailing assumptions about AI progress and offering a reasoned argument. [+2]