Gary Marcus

Rank 13 of 47
|
Score 83
9/14/2024, 4:40:55 PM
X
In reply to:
Cameron Williams
@Wasgo
·
486d
@GaryMarcus I think it still is a pure LLM though.

Based on their answers yesterday, the changes made to do CoT output happened through changes to the model.

It sounds like once the hidden generative output happens, it then uses a different generative model to summarize it and answer.
Gary Marcus
@GaryMarcus
·
486d
@Wasgo it’s not a pure LLM, and it doesn’t bear on the claims I made about pure scaling of LLMs.

it also doesn’t really solve the problems that I keep harping on, in any kind of general way.
Cameron Williams
@Wasgo
·
486d
@GaryMarcus Which makes it different than the incremental progress you had predicted for their next model.

It’s an interesting thing as o1 isn’t a paradigm shift as it’s built over the existing 4o model, but it’s not an incremental improvement that had been expected either.
Cameron Williams
@Wasgo
·
486d
@GaryMarcus Because o1 has improved through structured output, it masks hallucinations and overfitting in ways that makes it look like progress is happening on those problems in the generative model.

So people now wrongly think those issues are more solvable than you previously claimed.
Gary Marcus
@GaryMarcus
·
486d
@Wasgo to help me calibrate better, can you spell out more the things where I have been wrong?
Cameron Williams
@Wasgo
·
486d
@GaryMarcus I think people get confused between your predictions of how growth will happen (where you’ve been occasionally wrong) and the limits of the current model (where you haven’t).

Because of that, people think you’re changing your goalposts to claim you’re not wrong (you haven’t).

The statement in question is part of a technical discussion about the nature of a language model (LLM) and its underlying mechanisms. It engages substantively with public issues related to artificial intelligence and machine learning, making it a form of public discourse.

  1. Principle 1:
    I will strive to do no harm with my words and actions.
    The statement does not appear to cause harm with words or actions. It is a technical clarification. [+1]
  2. Principle 2:
    I will respect the privacy and dignity of others and will not engage in cyberbullying, harassment, or hate speech.
    The statement respects the privacy and dignity of others. It does not engage in cyberbullying, harassment, or hate speech. [+1]
  3. Principle 3:
    I will use my words and actions to promote understanding, empathy, and compassion.
    The statement aims to promote understanding by clarifying a technical point, which can be seen as promoting understanding and empathy among those interested in the topic. [+1]
  4. Principle 4:
    I will engage in constructive criticism and dialogue with those in disagreement and will not engage in personal attacks or ad hominem arguments.
    The statement engages in constructive dialogue by addressing a technical point without resorting to personal attacks or ad hominem arguments. [+1]
  5. Principle 7:
    I will uphold the principles of free speech and use my platform responsibly and with integrity.
    The statement upholds the principles of free speech and uses the platform responsibly by contributing to a technical discussion with integrity. [+1]