Back

OpenAI offers approved groups a free tool to flag likely child sexual abuse images, including AI-made ones

Two ChatGPT phone screens from OpenAI: a warning that content sexualizing children is harmful and the account is suspended for 24 hours, and a suspension notice saying the conversation was removed and offering free, specialized support
Image: OpenAI

OpenAI is giving approved organizations a free tool that scores images for possible child sexual abuse, including whether a picture may be AI-made, so reviewers can check the likeliest ones first. Unlike tools that only match images already on file, it can help flag pictures no one has confirmed before. In OpenAI's own test on ChatGPT images, though, only about 3 in 10 of the images it flagged were abuse, so trained people still have to check them.

OpenAI on Monday, Nov. 16, released a tool that checks images for possible child sexual abuse material, including pictures made with AI, and is offering it free to approved organizations. OpenAI says it is the first widely available tool of its kind that also signals whether such an image may be AI-made. OpenAI built the tool with help from the National Center for Missing and Exploited Children, a nonprofit that runs a central tipline where people report the online exploitation of children. The tool gives each image a score for how likely it is to show abuse, and safety teams can use the scores to decide which images their reviewers look at first.

Matching systems compare images against a list of abuse images that have already been confirmed, so they can miss a picture that isn't on that list. OpenAI's tool instead looks for visual signs of abuse in the image itself, which can help it flag new pictures, including AI-made ones. OpenAI's head of child safety public policy, John Buckley, told reporters at a briefing that telling a fictional child from a real one is "critical" when cases are passed on to police, according to a news report. OpenAI says the AI-made score is a warning sign, not proof of where a picture came from.

OpenAI says the tool should sit beside matching systems, not replace them, and it is not the only detection tool on offer: OpenAI itself points to Thorn's Safer, a child-safety group's detection system. A score alone does not prove an image is abuse, OpenAI adds, and trained people must review flagged images before one is reported to the tipline or a hotline.

Organizations apply through an online form and use the tool through OpenAI's developer platform, and OpenAI says approved ones pay nothing. Applicants need an OpenAI developer account, may have to show a government ID, and in the U.S. must be registered with the National Center for Missing and Exploited Children. OpenAI names safety teams at online platforms, child-protection groups and police as the intended users, and says each should test the tool on its own images first. The Tech Coalition, an industry group, and ROOST, which makes free, open-source online safety tools, are helping share access.

In a test on images users had uploaded, OpenAI estimates the new tool could flag 48% fewer images than its current system while finding about the same number of abusive ones. Images the new tool flagged were about twice as likely to be abuse as those its current system flagged, OpenAI says. OpenAI also shared results from two early partners. At fal, reviewers confirmed 72% of the tool's most confident flags as abuse images not seen before. In SafetyKit's tests, the tool caught 90% of drawings or animations that sexualize children.

OpenAI's own report on the tool sets out its limits. On a separate sample of ChatGPT images, only about 3 in 10 images flagged at the standard setting turned out to be abuse, because such images are rare in everyday ChatGPT use. The tool can also miss abuse images. On OpenAI's set of test images, the tool caught about 8 in 10 abuse images overall, but fewer than half of those tagged as showing infants. It also did worse on images tagged as showing Black children or boys. Another AI assigned those age, race and sex tags, and OpenAI says these results are only a rough guide. It also cautions that some test pictures may show the same children as pictures the tool learned from, which could make the results look better than they are. The tool has not been tested on video, and OpenAI warns that results on ChatGPT's images are no guarantee for other services.

ChatGPT is changing too. When someone tries to find such material, it now shows a warning, removes the conversation and pauses the person's access for 24 hours, while pointing them to specialist help. Repeat attempts may lead to longer limits, a closed account or a report to authorities.

OpenAI also published guidance for developers, and it is offering police child-safety units new tools and training, starting with one that helps investigators sort through long chat logs and records handed over under legal orders.

North Carolina Attorney General Jeff Jackson, who spoke at OpenAI's briefing, told reporters that working with a tech company to prevent this harm is "still basically an experiment," according to a news report. OpenAI has not said how many organizations it will approve.

More on OpenAI