Gary Marcus

Rank 22 of 47
|
Score 35
6/18/2024, 4:09:06 PM
X
In reply to:
Adam Karvonen
@a_karvonen
·
454d
@acherm @GaryMarcus @emollick @HaileyStormC I think 50M parameter models plateau around 1500 Elon, but larger models like gpt-3.5-turbo-instruct play around 1800 Elo.

Given that they don't use search, I'm curious how that compares to people playing the ultra bullet format.
Mathieu Acher
@acherm
·
454d
@GaryMarcus @emollick interestingly, language models exclusively trained on chess data seem plateauing around 1500 Elo... at least based on experiments of https://haileystorm.github.io/ma… @HaileyStormC @a_karvonen and the paper seems to reuse similar settings
Gary Marcus
@GaryMarcus
·
454d
@emollick I think you are overstating the results; they aren’t really “expert” human players. The result was strongest with weak amateurs, and they got no positive result even for the condition with max “expert” = 1500. They are NOT showing this works for grandmasters (eg 2600), let alone…
Ethan Mollick
@emollick
·
454d
Awesome result on a way in which AIs can outperform all the individual human experts whose data it is trained on: the AI is effectively aggregating the wisdom of the expert crowd.

When trained on diverse expert knowledge, the AI acts like a majority vote, which beats individuals

The statement engages in a technical discussion about the performance of AI models in chess, specifically their Elo ratings and potential improvements. It suggests that mixing players to achieve a higher rating would lend more credibility to the results. The tone is analytical and aims to contribute constructively to the ongoing conversation about AI capabilities in chess.

  1. Principle 1:
    I will strive to do no harm with my words and actions.
    The statement strives to do no harm and focuses on a technical analysis rather than personal attacks or harmful rhetoric. It adheres to this principle well. [+1]
  2. Principle 3:
    I will use my words and actions to promote understanding, empathy, and compassion.
    The statement promotes understanding by contributing a thoughtful perspective on how AI performance could be improved. It aligns with this principle by fostering a deeper discussion on the topic. [+1]
  3. Principle 4:
    I will engage in constructive criticism and dialogue with those in disagreement and will not engage in personal attacks or ad hominem arguments.
    The statement engages in constructive dialogue, offering a suggestion to improve the credibility of the results. It does not engage in personal attacks or ad hominem arguments, adhering well to this principle. [+1]