Gary Marcus

Rank 19 of 47
|
Score 64
4/20/2024, 2:32:40 AM
X
In reply to:
Nathan Benaich
@nathanbenaich
·
442d
Of course, we believe frontier models will continue to play an important role, but as part of a split ecosystem … provided it’s allowed to happen.
Nathan Benaich
@nathanbenaich
·
442d
This work’s in its early days, but its potential implications could be seismic.

A world of on-device, small models for most commercial implications could shrink demand for high-end GPUs significantly, while ending the world of aggressive dilution to pay for capex.
Nathan Benaich
@nathanbenaich
·
442d
Microsoft has taken this further, with its BitNet paper.

This work demonstrates the potential of low precision 1-bit weights and 8-bit activations, which drive significant efficiencies in memory usage and energy consumption, while preserving scaling laws.
Nathan Benaich
@nathanbenaich
·
442d
We’re already seeing the power of some of these smaller models, some of which can already be run on-device:
Nathan Benaich
@nathanbenaich
·
442d
Ultimately, we believe that most economically useful foundation model applications likely don’t require multi-trillion parameter frontier models - we need to use the tools that are right-sized for the job.

cc @ssankar re: "cargo culture"
Nathan Benaich
@nathanbenaich
·
442d
The economics somehow manage to be bad for both parties: model builders (huge capex, low margins) and users (expensive services)

The statement engages in public discourse by critically evaluating the progress and investment in AI models, specifically comparing the advancements to GPT-4. It challenges the optimistic view of rapid progression in AI technology, suggesting a more cautious interpretation of current developments.

  1. Principle 1:
    I will strive to do no harm with my words and actions.
    The statement aims to correct potentially misleading optimism about AI progress, contributing to a more informed public understanding. [+1]
  2. Principle 2:
    I will respect the privacy and dignity of others and will not engage in cyberbullying, harassment, or hate speech.
    It respects the dignity of others by maintaining a factual and non-personal tone in the critique. [+1]
  3. Principle 3:
    I will use my words and actions to promote understanding, empathy, and compassion.
    The critique promotes understanding by highlighting the gap between investment and tangible advancements in AI models. [+1]
  4. Principle 4:
    I will engage in constructive criticism and dialogue with those in disagreement and will not engage in personal attacks or ad hominem arguments.
    Engages in constructive criticism without resorting to personal attacks, focusing on the issue of AI development and its perceived stagnation. [+1]
  5. Principle 5:
    I will acknowledge and correct my mistakes.
    Acknowledges the current state of AI advancements, providing a basis for correction if future developments prove the statement wrong. [+1]
  6. Principle 6:
    I will use my influence for the betterment of society.
    By fostering a realistic perspective on AI progress, the statement potentially helps in shaping more grounded expectations and policies. [+1]
  7. Principle 7:
    I will uphold the principles of free speech and use my platform responsibly and with integrity.
    Uses the platform to responsibly discuss and critique significant technological developments, contributing to the broader discourse on AI. [+1]