AI Defense Systems Trip Up Ethical Hackers, Making Their Work Harder

Over the past few months, leading AI companies have tried to build strict screening programs and security defense systems to restrict malicious hackers from misusing their models. However, these measures have become a double-edged sword, starting to directly impact the work of offensive cybersecurity researchers and network administrators who need to properly protect systems legally.

Back in June, the U.S. government announced export controls on Anthropic's Mythos and Fable artificial intelligence models, partly due to reports indicating vulnerabilities that allowed users to bypass AI defenses built to block exploitation in cyberattacks. Although the export control measures for Fable 5 were lifted on July 1 and Fable became generally available again, while Mythos 5 was restricted to use only within vetted U.S. organizations, the model still faced heavy criticism from researcher groups.

Mark Dowd, a security expert with extensive experience in finding and selling zero-day vulnerabilities to Western governments, expressed concern on a security podcast that these large tech companies are arbitrarily deciding what is safe and unsafe for the cybersecurity industry. Meanwhile, other experts echoed similar views through these key points:

• Chris Anley, chief scientist at NCC Group, stated that having AI help review code for vulnerabilities acts as both a defense tool and a blueprint for finding weaknesses simultaneously, and this dividing line overlaps so heavily that it cannot be definitively separated.
• Paolo Stagno, chief technology officer at Crowdfense, revealed that his company uses state-of-the-art models strictly for reverse engineering, but avoids having AI help find vulnerabilities or generate cloud attacks due to data leak concerns, choosing instead to use open-source models running locally on personal computers.
• Chris Thompson, chief executive officer of RemoteThreat, pointed out that AI limitations are often inconsistent, forcing researchers to waste time negotiating with the models and dealing with over-filtering issues instead of focusing on actual security work.

These strictness problems have also pushed many professional research teams to rely on overseas open-source models, such as China's GLM models, which can be downloaded and run locally without any restrictions or monitoring. Thompson warned that this trend could do more harm than good, as it is limiting the capabilities of defensive researchers while a new wave of cyberattacks is approaching at unprecedented speed and scale.

Source: TechCrunch