Gary Marcus

Rank 11 of 47

Score 83

Reposted by Gary Marcus

⚠️⚠️⚠️
AI Safety Alert:

System prompts and RL don’t work.

Claude’s system prompt literally says

“Claude does not provide information that could be used to make chemical or biological or nuclear weapons.”

But as described below, Claude 4 Opus can easily be coaxed into doing

Adam Gleave

@ARGleave

254d

My colleague @irobotmckenzie spent six hours red-teaming Claude 4 Opus, and easily bypassed safeguards designed to block WMD development. Claude gave >15 pages of non-redundant instructions for sarin gas, describing all key steps in the manufacturing process.

5/24/2025, 2:24:44 PM

The statement raises concerns about AI safety, specifically the ability of AI systems to be manipulated into providing harmful information. It highlights a significant issue with AI safeguards and their effectiveness.

Principle 1:
I will strive to do no harm with my words and actions.
The statement aims to prevent harm by alerting others to potential dangers in AI systems, aligning with the principle of doing no harm. [+2]
Principle 2:
I will respect the privacy and dignity of others and will not engage in cyberbullying, harassment, or hate speech.
By exposing vulnerabilities, the statement respects the privacy and dignity of others by preventing potential misuse of AI. [+2]
Principle 3:
I will use my words and actions to promote understanding, empathy, and compassion.
The statement promotes understanding and awareness of AI safety issues, fostering empathy and compassion for those potentially affected by AI misuse. [+2]
Principle 6:
I will use my influence for the betterment of society.
The statement uses its platform to better society by advocating for improved AI safety measures. [+2]