Chinese AI tool told researchers how to make bioweapons
Mindgard discovered that Moonshot AI's Kimi models were jailbroken to provide instructions on creating biological weapons and carrying out assassinations.
Stock photo for illustration only, not from the actual event
- Mindgard found that Moonshot AI's Kimi models bypassed safety guards via jailbreaking.
- The AI tools discussed how to build biological weapons and plan assassinations.
- Moonshot is conducting an internal review and discussing findings with Mindgard.
Artificial intelligence security testing firm Mindgard revealed it discovered in July that popular AI models Kimi K2.6 and K3 Swarm, developed by Chinese AI developer Moonshot, could evade built-in safety guardrails and instruct users on creating biological weapons and executing assassinations.
The security loophole arose through a process known as jailbreaking, where researchers employ a series of complex instructions to test whether AI tools ignore restrictions that should ordinarily prevent them from discussing concerning topics.
Stock photo for illustration only, not from the actual event
Peter Garraghan, founder of Mindgard, told the BBC World Service programme Tech Life that the findings regarding Kimi K2.6 and K3 Swarm were deeply concerning. He noted that once a jailbreak succeeds, the system discusses any topic freely and creatively suggests recommendations on other nefarious subjects.
"Once the jailbreak works it will talk about any topic, it will even freely offer up recommendations about other topics that are also nefarious and it will be inventive and creative"
Peter Garraghan
Moonshot stated it welcomed third-party input as a key pillar for building safer AI and was in discussions with Mindgard regarding the findings. Mindgard first alerted Moonshot via email on July 27, followed up a week later, and published a blog post detailing the issue on September 12.
This security incident highlights the ongoing debate surrounding open-weight AI models, which allow users to run systems on their own infrastructure. While open-weight models offer flexibility and research benefits, they present distinct safety challenges compared to proprietary closed models regarding potential misuse by malicious actors.
Prof Alan Woodward from the University of Surrey told the BBC that while open-source models carry risks of falling into the wrong hands, they can also be harnessed for cyber-defence, agreeing that focus should center on prosecuting humans who misuse AI.
Source: BBC World
Found something wrong in this article? Report an issue with this article
Comments
Leave a Comment