Security Reports and Identity Protection features available in Azure AD, Azure AD Premium P1 and P2

Azure AD Premium P1 is included with Enterprise Mobility and Security (EMS) E3. I have been experimenting with numerous aspects of this subscription, since security is such a high priority these days, especially for the SMB (small businesses are statistically far more more likely to be targeted than large enterprises).

As an aside: the EMS E3 subscription is now being given away to Nonprofit customers who apply for the annual donation (up to 50 seats which is a 1,500 dollar value). And, outside of that donation it is still very affordable for them at only $2.50 USD/user/month. For-profit clients will pay a hefty $8.74 USD /user/month for the same, or $6.00 for Azure AD Premium P1 only. I enjoy this as an add-on to Microsoft 365 Business as well as Office 365 E3.

Today I want to look at security reports that are available to you with an Azure AD Premium (P1) subscription. Also, we will look at an advanced “Identity Protection” feature, which is related.

If you’re in a hurry and just want the quick-and-dirty, this post could be summarized as follows:

Sign-ins, audit logs and the Risky sign-ins report

First thing to understand: a lot of security event data is included for you to view even at the basic level of Azure AD, which is included with Office 365 subscriptions. Go to Azure Active Directory, find Activity and click Audit logs.

Here you can browse and filter many different event types for activity taking place in your tenant, such as those related to policy changes or provisioning users, etc.

Next, click Sign-ins to see a log of all sign-ins (successes and failures).

Again, this is available even in tenants without Azure AD Premium–it’s just good to know that the data is available to you. Under Security, check out Users flagged for risk and Risky sign-ins. Again, although data regarding “risky” events are available in every edition, the differences are in the level of detail. Check out the basic Risky sign-ins report:

(Yeah, I shamelessly stole some of these screenshots from Microsoft because I was being lazy. So what?)

We have a few data points here like the User, IP, location, sign-in time, etc. We are also able to mark the incidents after our investigation (Resolve, false positive, etc.). Now look at the same report using Azure AD Premium P1:

Way more interesting, right? We have a breakdown by risky event types, and we can see the underlying detail to each by drilling down further. Now I can verify WHY a user is getting flagged for risk–was it because of an infected device, or because of leaked credentials, or just unfamiliar locations and impossible travel events?

Impossible travel is when an account is authenticated in one geographical location, then soon after in a very different location (impossibly far away from the first). Sometimes, it turns out to be a false positive; for example: if a user is traveling and they stay in a hotel or work from an airplane, then they may be relaying through who-knows-what hub location in transit. When they switch to their next hot spot or wifi access point in a conference room, it can appear they made an impossible travel event. Nevertheless, it is better to be safe than sorry about these things–so check them out to be sure whenever you see them.

You might want to force a user with risk events like this to change their password, and consider enforcing MFA using Conditional access.

But here is the disappointing part: Azure AD Premium P1 will only allow you to see these events and take action on them from the portal: manually marking events as resolved, false positive, etc.  If you want anything automated, like the ability to receive a digest of events, or notifications when high risk events take place, or the ability to implement policies based on certain types of events, then you have to go all the way to Azure AD Premium P2 in order to obtain Azure AD Identity Protection.  Sadly, this feature used to be included with EMS before they split it out into P1 and P2. Sigh. Ah well, let us take a gander at Identity Protection.

Azure AD Identity Protection within Azure AD Premium P2

To get access to this tool set, it is necessary to visit the Azure admin portal, and click on Marketplace.

From there, you will be able to find the Identity Protection app, and add it to your portal.

Just click Create.

Now you should be able to access the app within the Azure portal, or even just from the Azure Active Directory admin center (which shows a sub-set of the full Azure portal–just those items pertaining to Azure AD).

You will notice that some of the same reports are available here, as we see in P1.

But, we can start to find more options as we continue scrolling down. In this tenant at least, it seems the only vulnerability is that MFA registration has not been completed for 201 people. So far, I have yet to see another type of vulnerability highlighted by this report (but I also don’t work with that many P2 subscriptions).

To re-mediate this type of vulnerability, we can use policy.

In my opinion, this particular feature is redundant, since we already have Conditional access with P1, and we can easily set a policy to require MFA registration (and enforce it’s use in certain circumstances). With any of these policies, it is possible to see the scope of estimated impact in advance.

Next: you can create user-based policies, and based on how they are classified according to risk, take an action like Require password change (pictured below).

There is also sign-in risk policy, so that a user, even one classified as low-risk, could be challenged in a higher-risk login situation. In this case, we can require MFA.

So at the end of the day, it appears that the whole “magic” of Azure AD Identity Protection, is to drive you toward creating policies that require users to register for and eventually use MFA.  Hm… like I said, I can already do that with Conditional access in P1… so… meh? I guess it’s cool that you can basically create an “advanced” Conditional access policy that is based on risk alone? That’s neat, I suppose. But still, in most small orgs, I think that using P1 to require MFA except from trusted locations / IPs will be sufficient.

Now, the feature that I really, really can’t believe is held hostage in P2–Alerts and a Weekly Digest report. I want to be notified when tricky shit is happening in my environment, don’t you?! The carrot on the end of that stick is pretty tempting, but damn… 9.00/user/month… you’re killing me, Microsoft!

Other considerations

Two more important caveats.

1. Risky events don’t really show up “right away”–it seems like it can take time to register the fact that certain events are “risky.” The sign-in and audit log is basically updated in real time, but I am guessing the heuristics to detect risky business are only run against that data on a schedule of some sort. In some cases it shows up within an hour or so, and I have waited up to a day before some events or user identities filtered into those “risky” reports, after the event timestamp.

But what if I wanted to pull reports from 6 months ago?

2. Also, you should know that you can download reports, but you only get access to certain amount of data, depending on where you’re at. For instance, you can only go back 30 days for sign-in events, and return a maximum of 5,000 records total (this is a limitation of SQL actually). If you want to access more than that–to sift through events for historical reference or forensic purposes, you are going to want to integrate with a Security Information and Event Management (SIEM) tool, which is capable of pulling in data from Azure. Most small businesses cannot and will not afford this kind of technology, so I see this as a huge opportunity for MSP’s, who are willing to take advantage of their scale and build a security and compliance offering around this gap.

Plus, if an MSP were to build their own monitoring and alerts with that SIEM-as-a-service offering, you could plug in other data sources, and you don’t even need an Azure AD Premium P2 in order to get alerts from your Microsoft-hosted items! Most SMB customers will never leap for P2 subscriptions, or go for the EMS E5 and Microsoft 365 E5 licenses that would be required to obtain it bundled anyway–they are far more likely to have a Microsoft 365 Business subscription, for example, or maybe the E3 flavor of EMS layered on top of some other Office 365 subscription.