For the best part of a decade, abusive deepfake “pornography” websites have published thousands of nonconsensual videos depicting women, including high-profile celebrities, politicians, and public figures, into sexual situations. Now, in what is likely the largest takedown of explicit deepfake websites to date, officials in New York have seized more than a dozen of the harmful sites.
On Monday, the Manhattan District Attorney’s Office announced the seizure of 12 domains which it says have been used for “unlawfully” sharing, publishing, and selling celebrity deepfake videos. Around 1,200 people—overwhelmingly women—were depicted in nonconsensual sexual images and videos on the websites, the office says. These included social media influencers, actors, activists, athletes, musicians, and politicians.
“These horrific violations of privacy follow victims into their careers and personal lives and take an immense toll on emotional and mental well-being,” district attorney Alvin Bragg said in a statement, urging potential victims to come forward and contact his office’s cybercrime bureau. The office says its investigations into who is responsible for the websites, plus those who uploaded damaging videos of victims, are ongoing.
The seizures, carried out under New York’s criminal procedure laws, mark perhaps one of the most significant and sweeping actions against websites hosting explicit deepfake videos since the technology first emerged at the end of 2017. As artificial intelligence technology has improved, a dark industry of websites, apps, and bots has appeared: They are designed to “undress” photos of women or create graphic sexual videos that include them. The images and videos are increasingly realistic and can be generated in seconds.
While deepfake technology is widely used against non-famous women and girls—often being shared in private social media groups—celebrities have spoken out about the many harms of being depicted in the content and the challenges of getting it removed. While laws against deepfakes have been slow to emerge, the US Take It Down Act increasingly allows images to be removed, with part of the law allowing law enforcement officials to take down and seize websites.
The Manhattan District Attorney’s Office has not publicly named the websites it has seized. However, ahead of Bragg’s announcement, WIRED independently observed several of the sites’ homepages being replaced by takedown notices—“THIS DOMAIN HAS BEEN SEIZED”—late last week, with the sites now saying they have been taken over following a seizure warrant issued by a New York State Supreme Court.
“The takedowns are a significant win in the fight against synthetic nonconsensual intimate imagery (NCII),” says Leonie Oehmig, a researcher at the Institute for Strategic Dialogue (ISD), who has studied the deepfake creation ecosystem. Oehmig says that enforcement of state laws shows that the harmful content can be tackled and that the seized sites don’t appear to be accessible if using VPNs to try and access them from other locations. “This shows that meaningful, coordinated enforcement is achievable even where regulation itself remains fragmented across jurisdictions.”
The websites were likely viewed millions of times. Many used names that included phrases such as “sex celebrity” or promised “real deepfakes.” They hosted false videos of some of the world’s most famous celebrities, with one saying, “various actresses’ faces are used for deepfakes.”
Videos could be several minutes in length and include fully sexually explicit scenes—often swapping the faces of celebrities or public figures into existing pornographic videos. Archived versions of the websites show some videos had been watched tens of thousands of times, with one of the websites claiming to have been online since 2018.
“Most videos took the shape of actual pornographic movies, manipulated to instead show targets’ faces instead of the original actors,” says Benjamin Shultz, the lead researcher at the American Sunlight Project, a nonprofit. Shultz, who has been documenting deepfake abuse against political figures and had come across some of the websites during this research, says that many videos appeared to be reposts of content that had originally be shared on the MrDeepFakes website, a notorious deepfake platform that was taken offline after its creators were unmasked by reporters.
“On at least two of the seized websites, elected officials were given profiles containing their name and position, along with a grid of manipulated videos hosting their likeness,” Shultz says. Through recent research, Shultz says that 56 European politicians had been mentioned or appeared on the websites, including 16 US politicians and two based in the UK.
While Oehmig, the ISD researcher, says the takedowns are encouraging, there are many other parts of the deepfake industry that are causing serious harm. “Effective intervention needs to target the ecosystem at multiple points, including the tools used for the creation of synthetic NCII, the discovery and promotion of these, the hosting and technical infrastructure that keeps them online, and the payment and affiliate systems that enable operators to monetize them,” Oehmig says.
