Former Microsoft security architect Michael Bargury has demonstrated several flaws that malicious hackers can exploit to abuse Microsoft Copilot, bypassing protections implemented by the giant.
Bargury demonstrated Copilot’s flaws last week in two sessions at Black Hat USA 2024, 15 Ways to Break Your Copilot and Live from Microsoft Copilot, and posted more data on the online page of Zenity Labs, the company he co-founded after leaving Microsoft. In each case, she particularly highlighted Copilot for Microsoft 365, as this service relies on access to sensitive internal information stored across Microsoft’s business customers. And despite security controls designed to keep this information private, Bargury was able to extract and exfiltrate it in some cases.
Sign up for our new free newsletter to receive three time-saving tips every Friday and get free copies of Paul Thurrott’s Windows 11 and Windows 10 field guides (normally $9. 99) as a special welcome gift.
Part of this has to do with social engineering. Its most impressive demonstration is that of a phishing attack called LOLCopilot that can access internal emails, write new emails mimicking the author’s writing style, and send large emails on their behalf. This requires that the user’s account be compromised in some way first. , a vital warning. But Copilot’s ability to automate malicious moves with so much insider knowledge greatly amplifies the damage it can cause.
“I can do that with the person you’ve talked to and I can send a bunch of emails on your behalf,” Bargury told Wired. “A hacker would spend days creating the right email to trick you into clicking on it, but they can generate tons of those emails in a matter of minutes. “
Unlike security researchers who undermined the Recall feature that Microsoft planned to launch in June with the new Copilot PCs, Bargury privately revealed the flaws he discovered to the software giant. He praises the work that Microsoft is doing to protect Copilot and that it is executing with the company to help address the underlying problems.
“The dangers of AI abuse after a compromise are comparable to those of other post-compromised techniques,” Phillip Misner, head of AI incident detection and reaction at Microsoft, said of Bargury’s findings. “Preventing and tracking security in environments and identities is helping to mitigate or prevent such behaviors. “
Microsoft has been aggressively bringing its AI technologies to market at an unusually fast speed in recent years. But the fear is that, in doing so, the software giant has left Copilot open to attacks and abuse. After all, getting the edge over the competition is simply speed: if Copilot proves to be dangerous, corporations will forget about it and those that followed it will abandon it.
It’s no coincidence that Microsoft last week outlined its “red team” to emulate real-world attacks on its artificial intelligence systems so it can help proactively protect corporate data. But this is particularly complicated because those systems are evolving.
“The AI Red Team practice not only covers finding security vulnerabilities, but also includes finding other formula flaws, such as the generation of potentially destructive content,” said Ram Shankar Siva Kumar, leader of the reading team. of Microsoft AI. “AI formulas carry new risks and the red team is key to those new risks, such as immediate injection and the production of unsubstantiated content. . . Microsoft recently committed that all high-risk AI formulas go through a review. independent red team before implementation.
You can learn more about the dangers of Microsoft Copilot on the Zenity Labs website.
Paul Thurrott is an award-winning generation journalist and blogger with 30 years of industry experience and 30 books. He is the owner of Thurrott. com and host of three tech podcasts: Windows Weekly with Leo Laporte and Richard Campbell, Hands-On Windows, and First Ring Daily with Brad Sams. In the past, he was a Senior Generation Analyst at Windows IT Pro and a writer for SuperSite for Windows from 1999 to 2014 and Thurrott. com Master Dome at BWW Media Group from 2015 to 2023. You can succeed in Paul through email, Twitter, or Mastodon.
Join the crowd where the love of the generation is genuine – become a Thurrott Premium member today!
Sign up for our new free newsletter to receive three time-saving tips every Friday