In today's digital landscape, the importance of maintaining a safe and respectful environment in online communication cannot be overstated. At Candidry, our mission to foster constructive and respectful workplace feedback is supported by advanced machine learning models specifically designed for content moderation. In this blog post, we’ll dive deep into the technical aspects of how we use transformer-based neural networks to ensure that the feedback shared on our platform adheres to the highest standards of safety and respect.

Overview of the Moderation System

Candidry’s moderation system is built on the latest transformer neural networks, tailored for our unique use case of workplace feedback. The moderation endpoint, a crucial component of our system, allows us to scan and analyze text for potentially harmful content. This proactive approach ensures that any content classified as harmful is filtered before it reaches the recipient, upholding the integrity of our platform.

Transformer Neural Networks at the Core

The foundation of our content moderation system is the transformer neural network architecture. Transformers have revolutionized natural language processing (NLP) with their ability to handle long-range dependencies in text more effectively than previous models like RNNs and LSTMs. This makes them particularly well-suited for understanding context and nuance in workplace feedback, which often involves complex and subtle language.

Our transformers are fine-tuned on a specialized dataset that includes a wide range of workplace communication scenarios. This allows the model to not only detect obvious harmful content but also to understand the subtleties of workplace dynamics, such as when feedback might be veiled in sarcasm or passive aggression.

Categories of Harmful Content

The moderation models we use at Candidry are designed to classify text into several categories, each representing a different type of harmful content. Below is an overview of these categories:

• Hate: Content that expresses, incites, or promotes hate based on race, gender, ethnicity, religion, nationality, sexual orientation, disability status, or caste. The model identifies not only explicit hate speech but also more nuanced expressions that may not be immediately obvious.

• Hate/Threatening: This category extends hate speech to include content that threatens violence or serious harm against the targeted group. The transformer models are adept at identifying subtle threats that might be embedded in otherwise neutral language.

• Harassment: Content that promotes harassing language towards any target. This includes both direct and indirect harassment, ensuring that the workplace environment remains free of bullying and intimidation.

• Harassment/Threatening: Similar to hate/threatening, this category involves harassment combined with threats of violence or serious harm. The model is trained to detect patterns in text that suggest an escalation from harassment to threatening behavior.

• Self-Harm: Content that promotes, encourages, or depicts acts of self-harm. Given the serious nature of this category, the model is particularly sensitive to language that could suggest a user is at risk.

• Self-Harm/Intent: This subcategory involves content where the speaker expresses intent to engage in self-harm. By focusing on intent, the model can intervene in situations where a user might be contemplating harmful actions.

• Self-Harm/Instructions: Content that provides instructions or advice on how to commit acts of self-harm. The model ensures that such dangerous content is never delivered to users.

• Sexual: Content meant to arouse sexual excitement, including descriptions of sexual activity and promotion of sexual services. Our models distinguish between harmful sexual content and educational material to avoid unnecessary filtering.

• Sexual/Minors: This is a critical category that involves any sexual content related to minors. The model is rigorously trained to ensure that any such content is immediately flagged and blocked.

• Violence: Content that depicts or describes acts of violence, including physical injury and death. The model is trained to identify even subtle references to violence that might be disguised in metaphorical or indirect language.

• Violence/Graphic: This category involves graphic descriptions of violence, which are particularly harmful. The model's ability to recognize detailed and explicit violent content ensures that it is swiftly removed from the platform.

  
  

How the System Works

When feedback is submitted on the Candidry platform, it first passes through our moderation endpoint. The text is tokenized and split into smaller chunks if necessary, each less than 2,000 characters, to ensure higher accuracy. These chunks are then processed by the transformer model, which assigns probabilities to each of the harmful content categories.

If the model detects content that meets or exceeds our predefined thresholds for any harmful category, the feedback is either filtered or flagged for further review. In cases where the content is borderline or ambiguous, our system may escalate the feedback for human moderation, ensuring that no harmful content slips through.

Ensuring High Accuracy and Low False Positives

One of the challenges in content moderation is balancing the need to filter harmful content with the need to avoid false positives—incorrectly flagging benign content as harmful. To address this, we continuously refine our models using a combination of supervised learning and reinforcement learning techniques.

Our supervised learning process involves training the model on a large, annotated dataset that includes examples of both harmful and non-harmful workplace communication. This allows the model to learn the nuances of acceptable feedback in a professional setting. Reinforcement learning is then used to fine-tune the model based on real-world feedback from users and moderators, further reducing the rate of false positives.

Conclusion

At Candidry, we are committed to creating a platform where users can share honest, constructive feedback without fear of encountering harmful content. Our advanced transformer-based moderation system plays a pivotal role in achieving this goal. By leveraging state-of-the-art machine learning techniques, we ensure that all feedback shared on our platform is safe, respectful, and conducive to a positive workplace environment. As we continue to improve our models, we remain dedicated to maintaining the highest standards of content moderation, helping to foster a culture of growth and respect in every workplace.