A Chinese law enforcement official inadvertently revealed a sprawling, state-sponsored operation to intimidate Chinese dissidents living abroad by using ChatGPT to document the campaign, according to a report released Wednesday by OpenAI, the creator of the AI chatbot.
The official, described as a member of China’s security apparatus, treated ChatGPT as a digital diary, detailing a systematic effort to harass and silence critics of the Chinese government, OpenAI stated. The discovery prompted the company to ban the user and publicly disclose the findings, highlighting a novel way authoritarian regimes may leverage artificial intelligence.
The operation, which involved hundreds of operators and thousands of fake online accounts across multiple social media platforms, extended beyond simple online harassment. According to the report, operatives impersonated U.S. Immigration officials to warn dissidents residing in the United States that their public statements allegedly violated American law. In one instance, the operation attempted to fabricate a court document from a U.S. County court to request the removal of a dissident’s social media profile, CNN reported.
Researchers at OpenAI noticed the unusual activity not because of the content generated *by* ChatGPT, but because of how the platform was being *used* – as a log for covert operations. The content itself, including disinformation, was created using other tools and disseminated through the network of false accounts, according to the report.
One documented episode involved a fabricated obituary and images of a gravestone created in an attempt to simulate the death of a Chinese dissident, which circulated online in 2023, as reported by Voice of America’s Chinese language service. The OpenAI report cross-referenced these details with real-world activity, confirming the operation’s scope.
“This is how modern Chinese transnational repression looks,” Ben Nimmo, OpenAI’s lead researcher, told reporters prior to the report’s publication. The incident underscores the potential for AI tools to be exploited for censorship and repression, even inadvertently revealing such activities.
CNN International sought comment from the Chinese Embassy in Washington regarding the report but did not receive a response. The escalating competition between the U.S. And China in the field of artificial intelligence adds another layer of complexity to the situation, as both nations vie for technological dominance.
OpenAI has not disclosed details about the specific methods used to identify the Chinese official or the extent to which the operation may have been successful. The company stated it is continuing to investigate and refine its security measures to prevent similar incidents in the future.
