California Attorney General Rob Bonta announced Wednesday that he is launching an investigation into xAI and Grok after the Grok artificial intelligence model produced sexualized images of women and children.
The investigation will look at "the proliferation of nonconsensual sexually explicit material produced using Grok," Bonta said in a statement.
He said xAI, Elon Musk's company that created Grok, "appears to be facilitating the large-scale production of deepfake nonconsensual intimate images that are being used to harass women and girls across the internet, including via the social media platform X."
Bonta's statement cites recent news reports of Grok taking normal photos of women and children that are available on the internet and sexualizing them, after being prompted by users, with AI by undressing the people in the photo (typically placing them in underwear or bikinis), posing them in a suggestive manner or showing them engaged in sexual activity — all without the consent of the people in the photos.
NBC News has reached out to X and Grok for comment.
Grok's image generation model now includes a "spicy mode," created specifically to produce explicit content, according to the statement, something Bonta said the company has used as a marketing tool, which has led to the influx of this kind of content.
One analysis cited in Bonta's release states that more than half of the 20,000 images that were generated by the program from Christmas to the New Year "depicted people in minimal clothing, and some of those appeared to be children."
"The avalanche of reports detailing the non-consensual, sexually explicit material that xAI has produced and posted online in recent weeks is shocking," Bonta said in the statement, noting that these photos are used to harass people online.
"I urge xAI to take immediate action to ensure this goes no further. We have zero tolerance for the AI-based creation and dissemination of nonconsensual intimate images or of child sexual abuse material," he added.
California Gov. Gavin Newsom appeared to endorse the investigation, writing on X on Wednesday: "xAI's decision to create and host a breeding ground for predators to spread nonconsensual sexually explicit AI deepfakes, including images that digitally undress children, is vile," before calling on Bonta to "hold xAI accountable."
After blowback from X users, the company appeared to restrict these permissions on the social media app, while keeping them available on the Grok standalone app, website and the Grok tab on X.
By Friday, the number of explicit images created by X's reply bot appeared to have been dramatically reduced. The same cannot be said for the Grok platforms.
A number of U.S. lawmakers have called on Musk and X to take down these images and prohibit the programs from producing more, but Bonta's investigation Wednesday marks the first major U.S.-based government action on the issue.
