Gary Marcus

Rank 11 of 47
|
Score 83
4/12/2024, 1:37:20 AM
X
In reply to:
Brydon Eastman
@brhydon
·
661d
@GaryMarcus Dr Marcus, respectfully, 100 is the max.... there's not a lot of space for exp growth in (87, 100]

MMLU and other evals are not perfect, they have plenty of noise in them. A 100 on that eval would inspire less confidence than, say, a 93 as it would just showcase Goodharting.
Gary Marcus
@GaryMarcus
·
661d
What happens when you plot GPT-2, 3, 4, and Turbo side-by-side?

Below I have plotted one common measure, MMLU, where there are easy to find data going back to GPT-2. (There may be others with data going back that far; this is just a first quick attempt.)

What I see is an…
Gary Marcus
@GaryMarcus
·
661d
Could we see GPT 3 and 3.5 and GPT 4 on the same plot? And Gemini Pro 1.5 and Claude Opus?

The statement is a constructive and respectful request for additional data to inform a technical discussion about AI model evaluation metrics.

  1. Principle 1:
    I will strive to do no harm with my words and actions.
    The statement does no harm and is focused on furthering understanding of the topic. [+1]
  2. Principle 2:
    I will respect the privacy and dignity of others and will not engage in cyberbullying, harassment, or hate speech.
    The statement respects the privacy and dignity of others, as it does not involve any personal information. [+1]
  3. Principle 3:
    I will use my words and actions to promote understanding, empathy, and compassion.
    The statement is not related to cyberbullying, harassment, or hate speech. [+1]
  4. Principle 4:
    I will engage in constructive criticism and dialogue with those in disagreement and will not engage in personal attacks or ad hominem arguments.
    The statement uses words to promote understanding and is part of a constructive dialogue. [+1]
  5. Principle 5:
    I will acknowledge and correct my mistakes.
    The statement is open to further information and correction of understanding if necessary. [+1]
  6. Principle 6:
    I will use my influence for the betterment of society.
    The statement uses influence to better understand the evaluation of AI models, which could contribute to societal benefit. [+1]
  7. Principle 7:
    I will uphold the principles of free speech and use my platform responsibly and with integrity.
    The statement upholds the principles of free speech and uses the platform responsibly and with integrity. [+1]