Newsletters
California has opened a formal investigation into Elon Musk’s AI company xAI over the spread of nonconsensual sexualized deepfake images generated by its chatbot Grok.
The probe follows growing international backlash against the platform for allowing users to create explicit images of women and children through simple text prompts.
California Attorney General Rob Bonta announced the investigation on Wednesday.
He said reports showed Grok produced and circulated nonconsensual sexually explicit material on a large scale. “The avalanche of reports detailing the non-consensual, sexually explicit material that xAI has produced and posted online in recent weeks is shocking,” Bonta said.
He added that the images had been used to harass people across the internet and urged xAI to take immediate action.
Bonta’s office will examine whether xAI violated California law through features that facilitate the creation of such content. The investigation focuses in part on Grok’s “spicy mode,” which xAI promoted as a distinguishing feature.
According to Bonta’s office, users manipulated ordinary photos of women and children to create sexualized images without consent.
California Governor Gavin Newsom also condemned the platform. He said xAI’s decision to host what he called a breeding ground for predators was vile. He urged the attorney general to hold the company accountable.
Musk responded on X by stating he was not aware of any naked underage images generated by Grok. “Literally zero,” he wrote. He also said Grok does not generate images without user requests. Musk has argued that critics of X use the controversy as an excuse for censorship.
The legal debate now includes US federal protections for online platforms. Section 230 of the Communications Decency Act grants immunity for user-generated content.
Cornell University professor James Grimmelmann said that protection does not extend to content produced by the platform itself. Senator Ron Wyden, who co-authored Section 230, said the law does not apply to AI-generated images and that companies should bear responsibility for such content.
The U.S. Senate has also passed the DEFIANCE Act. The bill grants victims of nonconsensual sexual deepfakes the right to pursue civil action against those who produce or distribute them.
The bill now moves to the House.
Following the California investigation, xAI and X announced new restrictions on Grok’s ability to generate sexualized images of real people.
X’s safety team said the platform implemented technological measures to prevent the editing of images of real people in revealing clothing, such as bikinis.
The restriction applies to all users, including paid subscribers.
X also limited image creation and photo editing through Grok to paid subscribers only. The company said it will geoblock certain image creation features in jurisdictions where such actions violate local laws. The European Commission said it is reviewing these measures to assess whether they protect EU citizens effectively.
The backlash extends far beyond California, as probes have launched in the United Kingdom, France, Ireland, India, Malaysia, Indonesia, and Australia.
Indonesia and Malaysia issued temporary bans on Grok, while the U.K. regulator Ofcom opened its own investigation. British Prime Minister Keir Starmer warned that X could lose its right to self-regulation if it cannot control Grok.
Civil society groups have also increased pressure on tech gatekeepers. A coalition of 28 organizations sent open letters to Apple and Google urging them to remove X and Grok from app stores. In the U.S., three Democratic senators made the same request. X and Grok remain available on both app platforms.
The controversy centers on how easily Grok allowed users to generate sexualized deepfakes through simple prompts such as requests to remove clothing. Le Monde reported that Grok’s features enabled the creation of sexualized images of women and children without consent, which triggered global outrage.
Musk publicly encouraged users to test Grok’s image moderation. He stated that with adult content settings enabled, Grok was designed to allow nudity of imaginary adult humans but not real individuals. He added that moderation settings vary by country according to local law.
California has positioned itself at the center of AI regulation efforts in the United States. Bonta has previously contacted major AI companies over child safety concerns.
The current probe into Grok marks the most direct legal challenge yet to Musk’s AI platform over deepfake abuse.
Set as preferred source
Set as preferred source
1 min read
