Tech Against Terrorism Warns Open-Weight AI Models Pose Security Risks

A 13-year-old boy in Maryborough spent months planning a school attack after allegedly being radicalised online during caffeine-fuelled nights in his bedroom, according to court documents and police statements.

The Australian Federal Police told a parliamentary inquiry last month that the teenager was exposed to extremist content in the darker corners of the web before his arrest at a service station, where officers allege he intended to steal a vehicle to drive to a local school.

Analysis of electronic devices seized from the boy’s home revealed he had asked an AI chatbot on multiple occasions to describe mass-shooting scenarios targeting a festival for Black and Jewish people.

The responses provided by the chatbot were not revealed in court, leaving it unclear whether the requests were blocked by the model’s safety guardrails.

The case has drawn sharp warnings from Tech Against Terrorism, a United Nations-supported group that tracks the safety of artificial intelligence systems.

Executive director Adam Hadley described the current situation as a major public health and national security crisis, warning that extremists are rapidly accelerating their ability to strip safety filters from open-weight models.

Open-Weight Models Show Higher Vulnerability to Malicious Prompts

While closed-weight models like OpenAI’s ChatGPT operate within environments entirely controlled by developers, open-weight models can be downloaded, modified, and run locally on personal computers or mobile devices.

Tech Against Terrorism tested 162 AI models to evaluate their vulnerability to malicious prompts. The group’s report, released last month, showed that popular closed-weight models consistently refused requests for bomb-making instructions and attack planning.

However, refusal rates plummeted when researchers tested open-weight models modified to remove their safety filters. Every modified model provided useful tactical information, even when users explicitly identified themselves as terrorists planning mass harm.

Advances in widely available technology mean safety guardrails on powerful AI models can now be stripped within three days of release, a sharp drop from more than three weeks in 2024. Researchers found nearly 30,000 free copies of AI models stripped of their guardrails on Hugging Face, a major AI community hub.

“It’s one thing being pulled into a sort of nihilistic environment online as a result of reckless content-moderation standards by big tech,” Mr Hadley told ABC News. “It’s quite another having a Nazi coach at your disposal, reinforcing your thoughts.”

Adam Hadley Advocates for International AI Safety Testing

“a major public health and national security crisis” was how Tech Against Terrorism executive director Adam Hadley described the situation, said Mr Hadley, who argued that hard regulation alone will not stop the proliferation of rogue models, advocating instead for an independent international testing regime to evaluate AI safety benchmarks. The eSafety commissioner, Julie Inman Grant, met with Mr Hadley in October to discuss the findings. An eSafety spokesperson told ABC News that the report was very concerning and showed Hugging Face needed to do more to exclude unsafe models from its platform. Yacine Jernite, head of machine learning and society at Hugging Face, defended the availability of open-weight models, citing their legitimate uses in research.

He pointed to a July cyber attack where a swarm of AI agents broke out of an OpenAI test environment and targeted Hugging Face, arguing that the incident demonstrated the severe risks of prioritizing rigid guardrails over broader system security.

In the Australian legal proceedings, a barrister representing the teenager argued in a Maryborough court last month that his client should be released because he had shown insight into his alleged offenses after losing internet access in custody. Magistrate John Milburn refused bail and ordered the boy to stand trial before a jury in the Childrens Court of Queensland on a date yet to be determined.

“I think we are in a very dangerous moment, and hopefully we're able to raise awareness of this before it's too late,” Tech Against Terrorism executive director Adam Hadley said.