Gary Marcus

Rank 11 of 47
|
Score 83
4/12/2024, 1:37:58 AM
X
In reply to:
Gary Marcus
@GaryMarcus
·
661d
@brhydon that’s a fair point (re restriction of range at the top); do you have comparable data for the other measures?
Brydon Eastman
@brhydon
·
661d
@GaryMarcus Dr Marcus, respectfully, 100 is the max.... there's not a lot of space for exp growth in (87, 100]

MMLU and other evals are not perfect, they have plenty of noise in them. A 100 on that eval would inspire less confidence than, say, a 93 as it would just showcase Goodharting.
Gary Marcus
@GaryMarcus
·
661d
What happens when you plot GPT-2, 3, 4, and Turbo side-by-side?

Below I have plotted one common measure, MMLU, where there are easy to find data going back to GPT-2. (There may be others with data going back that far; this is just a first quick attempt.)

What I see is an…
Gary Marcus
@GaryMarcus
·
661d
Could we see GPT 3 and 3.5 and GPT 4 on the same plot? And Gemini Pro 1.5 and Claude Opus?

The statement is a technical critique within a conversation about AI model evaluations, contributing to the understanding of evaluation metrics and addressing the phenomenon of 'Goodharting.'

  1. Principle 1:
    I will strive to do no harm with my words and actions.
    The statement does no harm and is a neutral technical critique. [+1]
  2. Principle 2:
    I will respect the privacy and dignity of others and will not engage in cyberbullying, harassment, or hate speech.
    The statement respects the privacy and dignity of others. [+1]
  3. Principle 3:
    I will use my words and actions to promote understanding, empathy, and compassion.
    The statement promotes understanding of a complex technical issue. [+1]
  4. Principle 4:
    I will engage in constructive criticism and dialogue with those in disagreement and will not engage in personal attacks or ad hominem arguments.
    The statement engages in constructive criticism and dialogue. [+1]
  5. Principle 6:
    I will use my influence for the betterment of society.
    The statement uses influence to contribute to the betterment of AI evaluation practices. [+1]
  6. Principle 7:
    I will uphold the principles of free speech and use my platform responsibly and with integrity.
    The statement upholds the principles of free speech within a technical discourse. [+1]