Gary Marcus

Rank 11 of 47
|
Score 83
4/12/2024, 2:09:49 PM
X
In reply to:
Ryan
@RyanMorey
·
657d
@GaryMarcus I think it is pretty well established now that the original GPT-4 is ~1T parameters (Jensen Huang has said as much and he would know!) and my understanding is that the Turbo model is either a distilled/quantized model. But for certain, it’s not an OOM larger
Gary Marcus
@GaryMarcus
·
657d
@RyanMorey Do we actually know, either model size or data amount? I thought none of this was disclosed anymore.
Ryan
@RyanMorey
·
657d
@GaryMarcus I don’t think this is quite fair, 3 to 4 spans an order of magnitude in model size, but 4 to 4-Turbo is either the same or smaller!
Gary Marcus
@GaryMarcus
·
657d
What happens when you plot GPT-2, 3, 4, and Turbo side-by-side?

Below I have plotted one common measure, MMLU, where there are easy to find data going back to GPT-2. (There may be others with data going back that far; this is just a first quick attempt.)

What I see is an…
Gary Marcus
@GaryMarcus
·
657d
Could we see GPT 3 and 3.5 and GPT 4 on the same plot? And Gemini Pro 1.5 and Claude Opus?

The statement is part of a technical discussion on AI model sizes and capabilities, seeking to clarify understanding.

  1. Principle 1:
    I will strive to do no harm with my words and actions.
    The statement contributes to constructive dialogue by seeking clarification. [+1]
  2. Principle 2:
    I will respect the privacy and dignity of others and will not engage in cyberbullying, harassment, or hate speech.
    The statement respects the privacy and dignity of others. [+1]
  3. Principle 3:
    I will use my words and actions to promote understanding, empathy, and compassion.
    The statement promotes understanding and seeks to clarify information. [+1]
  4. Principle 4:
    I will engage in constructive criticism and dialogue with those in disagreement and will not engage in personal attacks or ad hominem arguments.
    The statement engages in constructive criticism and dialogue. [+1]
  5. Principle 5:
    I will acknowledge and correct my mistakes.
    The statement avoids personal attacks or ad hominem arguments. [+1]
  6. Principle 6:
    I will use my influence for the betterment of society.
    The statement does not acknowledge or correct mistakes but is a question seeking information.
  7. Principle 7:
    I will uphold the principles of free speech and use my platform responsibly and with integrity.
    The statement uses the platform responsibly to engage in a technical discussion. [+1]