Gary Marcus

Rank 18 of 47

Score 95

Reposted by Gary Marcus

We need a Consumer Reports or Underwriters Lab for AI testing. All the public benchmarks are game-able and mostly not useful measures of things LLMs do. We need secret test batteries for subject areas (coding, reasoning, human conversation, writing) and secret red team tests, too

4/24/2024, 5:13:19 AM

The statement addresses the need for more robust and secretive testing methods for AI, particularly large language models (LLMs), to ensure their effectiveness and reliability. The speaker suggests the establishment of an organization similar to Consumer Reports or Underwriters Lab specifically for AI testing. This addresses a public concern about the transparency and accountability of AI technologies.

Principle 1:
I will strive to do no harm with my words and actions.
The statement aims to improve AI testing, which indirectly strives to do no harm by ensuring AI systems are reliable and safe before they are deployed. [+1]
Principle 3:
I will use my words and actions to promote understanding, empathy, and compassion.
By advocating for better testing mechanisms, the statement promotes understanding and potentially increases trust in AI technologies among the general public. [+1]
Principle 4:
I will engage in constructive criticism and dialogue with those in disagreement and will not engage in personal attacks or ad hominem arguments.
The statement engages in a constructive suggestion rather than criticism, aiming to initiate dialogue on improving AI testing methodologies. [+1]
Principle 6:
I will use my influence for the betterment of society.
The suggestion to create a dedicated entity for AI testing uses influence to potentially better society by enhancing the safety and reliability of AI systems. [+2]