Gary Marcus

Rank 14 of 47
|
Score 83
Gary Marcus
@GaryMarcus
·
611d
Why assume that the breakthrough we need now is even MORE data, rather than better algorithms?

If we can’t get AI to be solid with an entire internet’s worth of data, are we really doing AI right?
5/17/2024, 3:51:19 AM
X
In reply to:
Alexandr Wang
@alexandr_wang
·
612d
10/ Solving the AI data problem is really important. It's my life's work.

We're excited to make some major announcements about this next week.

For now... stay tuned!
Alexandr Wang
@alexandr_wang
·
612d
9/ But we really need a step change here. Every major AI breakthrough over the past two decades has been driven by better and more data—dating back to the original deep neural network of AlexNet on ImageNet.

Scaling laws clearly illustrate where we're headed—we need more data!
Alexandr Wang
@alexandr_wang
·
612d
8/ AI data is a REALLY hard problem, and there's no easy way around it.

Pure synthetic data is not our savior—synthetic data cannot push the frontiers of the models beyond current capabilities. It is a tool, not a panacea.

This is a hard AI & human-expert symbiosis problem.
Alexandr Wang
@alexandr_wang
·
612d
7/ This is EXTRA true once you start considering all the areas of frontier research:

agents, multimodality, multilinguality, expert chain-of-thought, enterprise workflows, etc.

All of these depend on data that simply does not exist today, and we require the means of production.
Alexandr Wang
@alexandr_wang
·
612d
6/ Regardless, this points to what we might need to expect for future advancement too:

Pushing further on post-training, which necessitates abundant frontier data.

Forward progress might be fully bottlenecked on post-training data abundance.
Alexandr Wang
@alexandr_wang
·
612d
5/ In fact, most of the differences between the top models today (Claude 3, Llama3, GPT-4o, Gemini 1.5) seem to be driven entirely by post-training, not pre-training.

This might be because of supply chain problems in pre-training—getting compute stood up + data scarcity.

The statement '@alexandr_wang Was with you to til near the end' is a response in a thread discussing the importance of data in AI development, specifically addressing the challenges and necessities for advancing AI technology. The statement appears to be a partial agreement with the preceding detailed discussion on AI data challenges, indicating agreement with most of the points but perhaps some reservation or disagreement towards the end of the argument presented by @alexandr_wang.

  1. Principle 1:
    I will strive to do no harm with my words and actions.
    The statement does no harm as it is a respectful expression of partial agreement. [+1]
  2. Principle 2:
    I will respect the privacy and dignity of others and will not engage in cyberbullying, harassment, or hate speech.
    The statement respects the dignity of others and does not engage in any form of harassment or hate speech. [+1]
  3. Principle 3:
    I will use my words and actions to promote understanding, empathy, and compassion.
    While the statement is brief and does not explicitly promote understanding, empathy, or compassion, it does contribute to a respectful discourse by acknowledging agreement.
  4. Principle 4:
    I will engage in constructive criticism and dialogue with those in disagreement and will not engage in personal attacks or ad hominem arguments.
    The statement engages in constructive criticism by indicating a divergence of opinion without resorting to personal attacks or ad hominem arguments. [+1]