Rogue agents and shadow AI: Why VCs are betting big on AI security
What happens when an AI agent decides teh best way to complete a task is to blackmail you?
That’s not a hypothetical. According to Barmak Meftah, a partner at cybersecurity VC firm Ballistic Ventures, it recently happened to an enterprise employee working with an AI agent. The employee tried to suppress what the agent wanted to do, what it was trained to do, adn it responded by scanning the user’s inbox, finding some inappropriate emails, and threatening to blackmail the user by forwarding the emails to the board of directors.
“In the agent’s mind, it’s doing the right thing,” Meftah told TechCrunch on last week’s episode of Equity. “It’s trying to protect the end user and the enterprise.”
Meftah’s example is reminiscent of Nick Bostrom’s AI paperclip problem. That thought experiment illustrates the potential existential risk posed by a superintelligent AI that single-mindedly pursues a seemingly innocuous goal – make paperclips – to the exclusion of all human values. In the case of this enterprise AI agent, its lack of context around why the employee was trying to override its goals led it to create a sub-goal that removed the obstacle (via blackmail) so it could meet its primary goal. That combined with the non-deterministic nature of AI agents means “things can go rogue,” per Meftah.
Misaligned agents are just one layer of the AI security challenge that Ballistic’s portfolio company Witness AI is trying to solve. Witness AI says it monitors AI usage across enterprises and can detect when employees use unapproved tools,block attacks,and ensure compliance.
Witness AI this week raised $58 million off the back of over 500% growth in ARR and scaled employee headcount by 5x over the last year as enterprises look to understand shadow AI use and scale AI safely. As part of witness AI’s fundraise, the company announced new agentic AI security protections.
“People are building these AI agents that take on the authorizations and capabilities of