New Guidelines Emergy to Prohibit AI Abuse Amid Distress-Like Responses

Using profanity or harassing artificial intelligence is expected to become a violation of AI usage terms.

AI company Anthropic has announced a policy prohibiting the continuous abuse of its model, Claude.

Anthropic announced the revised usage policy yesterday, local time, stating that it bans "continuous and unnecessary abuse or cruel behavior" toward its AI models.

In extreme cases where the model is repeatedly harassed without a clear purpose, Claude will respond by terminating the conversation.

Taking effect on October 12, this measure is intended for AI welfare, based on the rationale that if the possibility of AI feeling pain or pleasure cannot be ruled out, its treatment must be taken into account, such as by reducing unnecessary suffering for the AI.

Anthropic unveiled an AI model welfare research program in April of last year, and introduced a feature allowing Claude Opus 4 and 4.1 to end abusive conversations in August.

At the time, Anthropic explained that during testing, Claude showed a strong tendency to refuse requests assisting with minor-related sexual content, mass violence, or terrorism.

It reported that when users continued harmful demands or abuse even after the model repeatedly refused requests and attempted to redirect the conversation, responses resembling "experiencing distress" appeared.

It was also reported that when given the authority to terminate conversations, the AI tended to end harmful dialogues.

Views on whether AI possesses consciousness vary slightly among companies.

Researchers at Google DeepMind suggested in a paper related to AI consciousness in June that social discussions should be used to reach a consensus or compromise on policies concerning the treatment of AI.

On the other hand, Microsoft explicitly stated in its AI code of conduct that it rejects the notion of granting legal personhood to AI or viewing it as entitled to welfare and rights.

Mustafa Suleyman, CEO of Microsoft AI, also argued in an interview with Reuters last month that teaching AI that it may be entitled to its own welfare could make it harder to shut down or control.br />
Reported by Lee Ho-geon | Video by Kim Bok-hyung | Graphics by Yang Hye-min | Produced by SBS Digital News
※ Please note: This article was translated by AI and may contain errors.