Originally published on The Daily Flare.
OpenAI announced a new image classifier and tougher ChatGPT safeguards on Monday, November 16, aiming to detect child sexual abuse material and deter users from seeking or creating it.
Classifier aims to reduce review workload
The classifier is designed to spot visual signs of suspected child sexual abuse material, including images generated by AI and images the system has not encountered before. It assigns each image a score to help human reviewers prioritize cases. OpenAI developed the tool with the National Center for Missing & Exploited Children (NCMEC), the US organization that receives reports of suspected child exploitation.
OpenAI is offering select organizations access to the classifier through its API at no cost. The company is also working with the Tech Coalition and ROOST to share access. The tool is intended to identify suspected abuse in new images, rather than relying only on matching a file against a known digital fingerprint.
In early partner testing, OpenAI said the classifier flagged 48% fewer images for review while identifying the same number of abusive images as the comparison system. Images it flagged were twice as likely to be abusive, the company said. Those results suggest the tool could reduce the volume of material reviewers need to examine without lowering the number of abusive images identified.
Fox News also reported results shared by participating organizations. AI platform Fal said reviewers confirmed 72% of its high-confidence flags as previously unidentified CSAM. SafetyKit reported that the classifier detected 90% of cases involving drawings or animations that sexualize children in its evaluation. Those figures came from the organizations’ evaluations, rather than a single combined test.
ChatGPT warnings and developer guidance
OpenAI is adding warning screens for people who try prompts to access or create material that sexualizes children. The alert will explain why the content is prohibited and direct users toward specialized support intended to help stop harmful behavior. OpenAI said the account will be suspended for 24 hours after flagged activity, while repeated violations could lead to permanent removal.
ChatGPT already blocks CSAM and related requests, according to OpenAI. The new warnings add an intervention and temporary account restriction to those existing blocks. The company developed the warning language and guidance with organizations including Protect Children and the Lucy Faithfull Foundation, as well as University of Tasmania researcher Joel Scanlan, who independently evaluated a trial of deterrent messages.
OpenAI is also publishing guidance for developers using its tools on how to prevent and detect CSAM and help users report it. The company said early testing of the guidance helped a number of developers adopt child-safety tools and register to report suspected CSAM to NCMEC.
For law-enforcement agencies working on child safety, OpenAI is providing AI-powered features that include dedicated plugins, investigative tools and complex data analysis. Access is managed by program administrators. The company is also offering investigators training and a field guide on using AI.
The announcement comes amid wider scrutiny of AI safety for children. TechCrunch reported that OpenAI has faced several lawsuits alleging its technology harmed users, particularly children. Safety-focused nonprofit Common Sense Media has labeled OpenAI’s technology an “unacceptable risk” for teens. Over the past 12 months, OpenAI has also released a safety blueprint, updated content guidelines, retuned models to prevent harmful outputs and introduced a version of ChatGPT for teens.
Top comments (1)
This feels like one of the more practical AI safety announcements I've seen recently. A lot of the discussion around AI safety focuses on hypothetical future risks, while CSAM detection addresses a very real and immediate problem.
I'm particularly interested in the reported reduction in review workload without reducing detection rates. If those results hold up in broader deployment, that could significantly improve moderator efficiency while reducing exposure to harmful material.