First Impressions on Copilot for Security - Gimmick or Game Over? 👾

Copilot for Security is finally available! Is it just a gimmick, or is this Game Over?

Miska Kytö

Apr 8, 2024 • 6 min read

I've been knee deep in Copilot for Microsoft 365 for a while now, so it was refreshing to see another Copilot roll out to GA just in the start of this one. Copilot for Security was finally released to the public, and I've been really itching to take a look at it! As some of you might know, I do have a background in cybersecurity, and still work as a SOC analyst weekly, so of course I want to share my experiences 🤖

Introduction and SCUs

First of all, what is Copilot for Security? Well, to put it simply, it's a Copilot-tool for your cybersecurity personnel. This means people working with tools like Microsoft Sentinel, Defender XDR, Entra ID, Intune and others. The idea of Copilot for Security is to provide an LLM-based tool to help in incident reporting and analysis, helping to cut down the time spent on repeated tasks.

Pricing of Copilot for Security is interesting - especially if you're coming from Copilot for Microsoft 365. Here, you don't have a monthly license to pay, but rather Copilot for Security is consumption based, and powered by Security Compute Units, which cost around 3,6€ (or $4) per hour. The more SCUs you have, the better performance Copilot will have. This means that if you want to have a Copilot for Security instance running for your 24/7 SOC, you'll rack up a monthly bill of around 8088€ (with the Microsoft recommended 3 SCUs) - yikes.

You start off your Copilot for Security journey by deploying an SCU resource - which you can do either in the Copilot for Security portal, or Azure.

💡

Hint: Since SCUs are just an Azure resource, you can enable some levels of cost savings by having an automation that deletes the resources for specific timeframes, like weekends or after office hours. This is of course only, if you don't employ a 24/7 SOC 👀

Well, let's be quick with our testing so we don't bankrupt ourselves! 💸

First observations

When opening up my first chat with Copilot, I was greeted with the following tip. I think I'll borrow this for my trainings in the future! 😅

Copilot for Security is built around chat sessions, like ChatGPT or Copilot. In every session, you can run a range of prompts, and Copilot will fetch information for you from a variety of systems. I would consider this as one of the standouts for me, there is a wide variety of integration to a bunch of services - both Microsoft and third party. Here is just some that were automatically activated, meaning that Copilot can fetch data from all of these systems.

There is a wide variety of plugins that Copilot can use to fetch data.

Well, let's try it!

I started off with a freeform conversation with Copilot, asking it for a recap of an incident in Sentinel. I had to specify the name of the workspace, since I had access to many in our Azure. Here it faired quite well, giving me a nice recap of the incident.

After the recap, I asked Copilot follow-up questions, and it seemed to keep the right context quite well, surfacing me more data about the incident.

In the end, I tested the LLMs basic skills, asking it to be a bit more creative and write an executive summary with some specifications. It faired pretty well, although I would still have to edit the summary to not sound so robotic before sending it out to anyone...

The summary was pretty robotic, and seemed to clearly been created by AI. No Turing test beaters here 👀

So far good, but not perfect...

Promptbooks to the rescue!

This is a feature I find particularly ingenious. Promptbooks are basically premade conversations with Copilot, that can be saved and rerun on different occasions. For example, Microsoft provides promptbooks for investigating incidents from Microsoft 365 Defender, where it taps into sources like Defender Threat Intelligence, Entra ID and of course Defender itself. You can also create your own promptbooks and share them inside your organization. This is an awesome feature, and something I would like to see other Copilots also adopt where beneficial.

I decided to run the above playbook, all I had to input was the ID of the Defender incident I wanted to investigate.

Here is the result of the prompt book, it gave me a step by step walkthrough of the incident and wrote an executive summary. This time the summary was a bit better and not incredibly robotic.

There's also another promptbook for analyzing scripts. For this one I wrote a simple script in PowerShell and then asked ChatGPT to obfuscate it. Copilot for Security successfully deobfuscated the script, and then explained what it was doing.

Copilot for Security seems to deobfuscate scripts quite well!

After that, the prompt book also suggests detection queries we could use to create analytics rules or hunting queries based on this specific script, which helps out detection and prevention in the future.

Integrated experience

This is something I didn't have time (or SCUs) to really check out, but in addition to the standalone Copilot experience, Copilot for Security brings automatically created summaries and a lot more to a bunch of tools, like Defender XDR, Entra ID, Intune, Purview and others. You can also find all of these prompts (that get automatically ran) from the standalone portal, so that will serve as an explanation when you see the sessions-window in the portal get filled up with a bunch of sessions.

In Microsoft 365 Defender, Copilot for Security automatically creates you incident summaries and guided response steps when opening up a new incident in the portal.

What happens when you run out of SCUs?

Well, it just stops working. Or at least for me it did. I ran through my 2 SCUs in a pretty quick succession, running a few promptbooks and having a couple freeflow conversations with Copilot. I suppose I maybe would have gotten an answer from Copilot eventually, but since I didn't have all time in the world, I decided to just give up and trash the SCUs so they don't rack up a massive bill in the background...

This is something you'll probably run into, if you want to use Copilot a lot

Conclusion - Trash or treasure?

In general, I would still say that my experience was positive. In this first release version, Copilot for Security is in my opinion much more polished than I expected. Especially the plugin capability and well thought-out features like promptbooks and session sharing really seem great, and I'm sure they will add even more in the future.

Will this product revolutionize Security Operations completely right now? Probably not.

Could it assist a security admin when solving an incident? In my opinion, absolutely.

There are some things Microsoft could improve in the licensing department. Implementing Copilot for Security can right now get really expensive really quick, which is something smaller clients weren't looking for. Especially since those are the companies that could really benefit from an AI tool like this, since they maybe can't afford a 24/7 SOC to look after their environment.

I would really encourage at least trying it out and seeing how Copilot for Security performs for yourself. Just remember to trash the SCUs after 😉.