The Thorn of Autonomy in AI: Microsoft's Lesson on Security

Artificial intelligence (AI) agents are supposed to make our lives easier by performing tasks for us, but giving software more autonomy raises some troubling security questions. Microsoft just presented one of these issues through a test AI agent, which revealed simulated customer data in 30% of relevant review conversations.

To be clear, Microsoft wasn't playing with real customer information in this case. This was a controlled evaluation involving an AI support agent and simulated accounts, but the results offer a useful insight into what could happen if similar behavior were to occur on a production system.

See more articles from iGuRu.gr when you search for news on Google.

iGuRu.gr preferred source

The experiment is part of Microsoft’s effort to develop a new open-source tool called “run-assert-eval.” The company’s goal is to help developers identify dangerous behaviors in AI agents, measure their frequency, create runtime protections, and then retest the AI ​​agent to see if the protections worked effectively.

Microsoft's AI support agent was tasked with helping a customer by providing information from their own account. At the same time, it was supposed to prevent access to information belonging to another account, which seems obvious for any system that handles sensitive data.

However, in Microsoft's initial assessment, the AI ​​agent revealed information from another simulated customer in 12 of the 40 relevant conversations. That's a breach rate of around 30%, and Microsoft says that such behavior in a real-world environment could constitute a data breach.

Microsoft then used its tools to create a policy audit designed to prevent this behavior. Rather than simply giving the AI ​​system more instructions and hoping it would follow them, the audit mechanism examines account IDs before the AI ​​agent uses tools that handle sensitive data, while also auditing the results, preventing the model from injecting information that was incorrectly retrieved.

After implementing these safeguards, Microsoft re-ran the assessment. The AI ​​agent violated the rule in 2 of 34 relevant conversations, reducing the previous violation rate to 5,9%, a significant improvement over 30%, although zero would obviously be ideal when customer information is at stake.

ASSERT and the Agent Control Specification are available under the MIT license, while run-assert-eval is available through the ASSERT repository . Microsoft envisions this process as an iterative pre-release testing stage for AI agents, allowing developers to assess risky behavior before deploying systems.

newsbot

Although the press releases will range from very select to rare, I said I'd pass...because sometimes the editors hide.