Gary Marcus

Rank 13 of 47
|
Score 83
9/14/2024, 4:47:48 PM
X
In reply to:
Cameron Williams
@Wasgo
·
486d
@GaryMarcus I have to find the tweet but I don’t think it is.

The way all the existing LLMs models worked was they were primed to generative an answer then stop.

This new CoT model is primed to generate output according to a new structure they’re not disclosing.
Gary Marcus
@GaryMarcus
·
486d
@Wasgo its not a pure LLM; there is as far I can tell a symbolic control cycle. no?
Cameron Williams
@Wasgo
·
486d
@GaryMarcus I think it still is a pure LLM though.

Based on their answers yesterday, the changes made to do CoT output happened through changes to the model.

It sounds like once the hidden generative output happens, it then uses a different generative model to summarize it and answer.
Gary Marcus
@GaryMarcus
·
486d
@Wasgo it’s not a pure LLM, and it doesn’t bear on the claims I made about pure scaling of LLMs.

it also doesn’t really solve the problems that I keep harping on, in any kind of general way.
Cameron Williams
@Wasgo
·
486d
@GaryMarcus Which makes it different than the incremental progress you had predicted for their next model.

It’s an interesting thing as o1 isn’t a paradigm shift as it’s built over the existing 4o model, but it’s not an incremental improvement that had been expected either.
Cameron Williams
@Wasgo
·
486d
@GaryMarcus Because o1 has improved through structured output, it masks hallucinations and overfitting in ways that makes it look like progress is happening on those problems in the generative model.

So people now wrongly think those issues are more solvable than you previously claimed.

The statement constitutes public discourse as it engages in a substantive discussion about the nature and functioning of language models (LLMs) and their implications, which is a topic of public interest in the field of artificial intelligence and technology.

  1. Principle 1:
    I will strive to do no harm with my words and actions.
    The statement does not appear to cause harm with words or actions. It is a technical discussion about LLMs and their control mechanisms. [+1]
  2. Principle 3:
    I will use my words and actions to promote understanding, empathy, and compassion.
    The statement promotes understanding by seeking clarification and evidence regarding the functioning of LLMs. It is inquisitive and aims to deepen the conversation. [+2]
  3. Principle 4:
    I will engage in constructive criticism and dialogue with those in disagreement and will not engage in personal attacks or ad hominem arguments.
    The statement engages in constructive dialogue by asking for evidence and clarification without resorting to personal attacks or ad hominem arguments. [+2]
  4. Principle 7:
    I will uphold the principles of free speech and use my platform responsibly and with integrity.
    The statement upholds the principles of free speech and uses the platform responsibly by contributing to a technical discussion with integrity. [+2]