A Tale of Two Incidents

I have two stories to share today. Both are from individuals who contacted me about recent Cybersecurity incidents that their MSP had suffered. Both incidents were very different, and yet they share some common threads and lessons that I think we would be wise to heed. And this is all applicable beyond MSP’s, but it also highlights just how much Managed Services Providers are being targeted right now, too.

Incident #1: The BYOD/WFH device

In this first incident, which unfolded over the summer, an MSP had about a dozen customers get ransomware on the same day. It was not the customers who were individually compromised but rather the MSP. At first they thought there must be some kind of new global worm like Wannacry, but nope, it was just them–they were the common thread between all these incidents.

What happened? They were so dumbfounded because they believed they had done everything right: MFA on all of their major systems, antivirus and EDR agents on all their endpoints. They have what they feel is a really good (third-party) anti-spam/phish solution in front of their email (which is in turn hosted at O365).

Now they were a little surprised that only a portion of their overall client base got taken down–maybe more were about to get compromised just around the corner? They had to send communications out to their entire customer base about the unfolding attack, just in case. But at the end of the day, they traced the compromised customers back to one single tech who had all of these accounts in common.

During the pandemic, of course all of their technicians are working from home, and, some of them chose to work on personal (rather than work-issued) devices. And in this case, the tech had their personal device compromised (none of the MSP’s owned systems had even been part of the event). On this device was stored the keys to the kingdom: credentials and other access information, as well as details about the customers’ networks. The tech had been keeping “convenience copies” of customer information handy, in personal apps and storage locations.

Hard to say how the initial compromise happened, as this tech ended up wiping/reloading that home machine before they could try to preserve any artifacts as evidence, etc. But, had it been a managed device, certainly the risk of compromise would have been lower (provided the MSP does a decent job of protecting their endpoints–including local admin privilege, application control, and so on).

CIS Critical Security Control #1 states:

Maintain an accurate and up-to-date inventory of all technology assets with the potential to store or process information. This inventory shall
include all assets, whether connected to the organization’s network or not.

Center for Internet Security, CSC 1.4

(Italics added for emphasis).

So it is totally cool that this provider was doing “the right things” as far as MFA and so on–but this incident demonstrates that when you don’t follow the basics of Cybersecurity hygiene, you are undermining any subsequent protection that you may choose to employ. MFA doesn’t protect you against all types of threats–especially when you are allowing unmanaged devices and applications to store and process corporate data.

Biggest lesson here is that you should not allow unmanaged devices to store or process corporate data if you are an MSP–especially if that device has the potential to expose customer information. You should absolutely bake language akin to this into your Acceptable Use Policy; also consider adopting additional technical controls to prevent access to and prevent exfiltration from your systems on unmanaged devices.

Unfortunately, you could still be open to issues on a managed device, and possibly this same outcome, if you are ignoring other critical security controls such as application control, limiting admin privileges, and monitoring the audit logs for example. Once you have limited access to a set of managed devices, you do still have to manage them, after all.

Incident #2: Phishing with an additional OAUTH app angle

In the previous story, it is unfortunate we do not have more details about the initial events that lead to compromise on that unmanaged device. But here is an interesting one where we do have some more detail, and it is interesting how the attack was targeted against a specific cloud service–Microsoft’s Partner Center. If you are a Cloud Service Provider (CSP) and use the Partner Center, you know that this will allow you administrative access to numerous other tenants under your watch.

Last year, Microsoft required all CSP’s to enable Multi-factor (and disable Basic Auth). That’s good. But it sounds like the adversaries are now starting to make adjustments to their approach–they will move around this problem in other ways.

A non-technical user who administers the CSP and billing relationships at this MSP received a phishing email that looked almost identical to other notifications that come from Microsoft regarding Billing, etc. It made some claims about expiring subscriptions that needed immediate attention or else data loss, etc.–and these do also come from MSFT some times, so very hard to tell the difference here. The only tip would have been that the from address is not actually a Microsoft domain.

So this person clicked on the link in the email and was taken to a Microsoft authentication page, which she completed with MFA included, and was then greeted with a consent request for an application called “Office 365 Billing Account.” Upon accepting the prompt, she unknowingly granted access to an outside party through a new application identity that had been added to their tenant. That identity had extremely wide reaching permissions, and was quickly used to establish new accounts in the organization.

Now this provider was extremely lucky. They had read a blog of mine advocating for the addition of Microsoft Cloud App Security. Taking my advice to heart, they had enabled all of the out-of-box alerts, including the OAuth application alerts. So they quickly investigated this anomalous event and removed the app along with the additional footholds it had created–before the attacker had a chance to cause any damage to their environment or that of their customers.

Note that you can also enable Admin consent requests to mitigate the risk of OAuth apps, although sadly in this case the person who was targeted had full global administrator credentials (another no-no for which I scolded them)–so the admin consent requests would not have helped in this case.

Here are some relevant CIS Critical Security Controls:

#2: Inventory and Control of Software Assets

Ensure that unauthorized software is either removed or the inventory is updated in a timely manner.

Center for Internet Security, CSC 2.6

#4: Controlled Use of Administrative Privileges

Ensure that all users with administrative account access use a dedicated or secondary account for elevated activities. This account should only be used for administrative activities and not Internet browsing, email, or similar activities.

Center for Internet Security, CSC 4.3

#14: Controlled Access Based on the Need to Know:

Protect all information stored on systems with file system, network share, claims, application, or database specific access control lists. These controls will enforce the principle that only authorized individuals should have access to the information based on their need to access the information as a part of their responsibilities.

Center for Internet Security, CSC 14.6

Knowing that this kind of email campaign exists, in addition to all of the above, I would also caution anyone who administers Microsoft 365 billing and especially CSP/Partner Center for their customers as follows:

Do not click on the links in these emails, even if they seem legit. Instead, navigate in a browser to the Microsoft Admin Center or Partner Center directly to review the claims that are being made in these emails.

Conclusions

In both cases, the folks I was chatting with believed they were doing “everything” they were supposed to be doing. Yet in just a short conversation I learned that they had not been following some of the Basic CIS Controls, which I often harp on (for obvious reasons). What these individuals meant to say is that they were doing everything that the herd collectively believes they should be doing. And that “common sense” knowledge may share some threads with an actual evidence-based, real-world Cybersecurity Framework like the CIS Controls, but it very often comes up a few fries short of a Happy Meal, if you know what I mean.

Better to make sure you are following and documenting your progress against a formal framework. If you do the basics really well you will be out ahead of many of your peers. But as we can see in the examples above, you also have to stay on your toes. You never know what your adversary’s next move will be, and many folks are surprised to learn where they still have blind spots.

I also enjoy collecting these stories. If you are willing to share them with me, I will share them with the world along with an analysis of what we can do to avoid these same things happening again, to another of our peers. You can choose to remain anonymous, or not. The parties above chose to remain unnamed.