Mistral AI unveils new moderation API to enhance content safety

Mistral AI announced the release of its new Moderation API, a tool designed to improve the safety and scalability of content management systems. According to Mistral AI, the API aims to help users detect undesirable text content across a variety of policy dimensions.

Enhanced Safety Measures

The Moderation API is built on the same framework that powers the moderation service in Mistral AI’s Le Chat platform. It provides users with a flexible tool that can be customized to meet specific safety standards and application requirements. As demand for Large Language Model (LLM)-based coordination systems grows, Mistral AI’s products seek to provide scalable and robust solutions.

Multilingual function

The API has an LLM classifier that can classify text input into nine distinct categories. It contains endpoints for raw text and conversation content, allowing you to classify messages within a specific conversation context. This model supports multiple languages, including Arabic, Chinese, English, French, German, Italian, Japanese, Korean, Portuguese, Russian, Spanish, etc., making it suitable for users all over the world.

Focus on policy relevance

Content moderation classifiers integrate relevant policy categories to establish effective safeguards against potential harms such as unqualified advice and disclosure of personally identifiable information (PII). Mistral AI’s approach to LLM safety is practical, comprehensive, and addresses the nuanced nature of undesirable content in a variety of contexts.

Performance and Collaboration

Mistral AI shared performance metrics, including area under the precision-recall curve (AUC PR) for internally tested policies. The company is committed to contributing to safety advancements in the field of AI by collaborating with its customers and the broader research community to improve and expand its intervention tools.

This release is part of Mistral AI’s ongoing effort to provide lightweight, customizable orchestration solutions that can adapt to the changing needs of the industry.

Image source: Shutterstock