Gary Marcus

Rank 14 of 47
|
Score 83
Arvind Narayanan
@random_walker
·
247d
Devastating takedown of Chatbot Arena. It's one thing for leaderboards to suck because they try to quantify the unquantifiable but quite another thing to actively choose flagrantly unscientific and nontransparent practices that benefit the big dogs. https://arxiv.org/abs/2504.20879
4/30/2025, 10:47:36 PM
X

The statement suggests that the issues with Chatbot Arena could be as significant as the Theranos scandal, implying potential deception or malpractice. It critiques the lack of transparency and scientific rigor in the leaderboard practices, which could mislead stakeholders.

  1. Principle 1:
    I will strive to do no harm with my words and actions.
    The statement aims to highlight potential harm caused by nontransparent practices, aligning with the principle of doing no harm. [+1]
  2. Principle 2:
    I will respect the privacy and dignity of others and will not engage in cyberbullying, harassment, or hate speech.
    It respects privacy but critiques organizational practices, not individuals, aligning with respecting dignity. [+1]
  3. Principle 3:
    I will use my words and actions to promote understanding, empathy, and compassion.
    The statement promotes understanding by drawing attention to issues in AI evaluation, fostering awareness. [+1]
  4. Principle 4:
    I will engage in constructive criticism and dialogue with those in disagreement and will not engage in personal attacks or ad hominem arguments.
    It engages in constructive criticism by addressing specific practices and suggesting the need for reform. [+1]