Evaluating the quality of Managed Services Providers (MSP) is not easy. You can review services, SLAs, prices, and capabilites, but how do you know if they are willing to “go the extra mile.” This article is a real case study of an MSP who went above and beyond, and one that did the minimum.
This was the week us mere mortals learned about the Solarwinds Orion vulnerability. While I can’t figure out the full extent of the risk, I know it is bad and, as a virtual CTO to many companies, it is my job to inform executive management and Boards of Directors as to where we stand and what risks exist.
Harvard Partners provides virtual CTO services to many clients. We provide part-time CTO services for small to mid-tier firms requiring high-performce IT services without a full-time CTO resources. A big part of gaining scalability and leverage is utilizing Cloud and Managed Services. If done correctly, reliability and supportability go up and costs start to come down.
This means we interface with many enterprise-class managed services providers and get the opportunity to evaluate the differences.
Managed Services Provide (MSP) #1
Our Client turned over their entire computing environment, from PCs to Citrix (Azure-based) to Azure servers and storage to LAN, WAN, SAN, SIEM, SOC, and Cybersecurity support to a national MSP for 7×24 support. The MSP is not an insignificant company, with many resources and a vast network of clients across many industries.
When I questioned this MSP why they hadn’t alerted me (the CTO) as to the threat and status of this vulnerability they said:
“We are only telling our clients who use SolarWinds.”
I told them I had been told this has impact beyond SolarWinds as SolarWinds was simply a “vehicle” for implemeting other things and I felt my Client was still vulnerable.
The response I got was a phone call (they will not put anything in email) saying the following:
MSP #1 is actively monitoring the fluid situation related to the SolarWinds supply chain attack, and related activities of concern. As information emerges, our collective understanding and decisions are also evolving.
To that point, MSP #1 takes this situation very seriously and recommends all customers do as well.
MSP #1 believes all companies take a fresh look at their security posture, and the posture of any networks they interact with and trust.
Even though MSP #1 does not run, operate, or manage the known vulnerable code, we are proactively implementing additional security measures.
If your organization is running SolarWinds, we highly recommend your team the following guidance from DHS and CISA (URLs included)
Related to this particular set of threats, MSP #1 believes an abundance of caution is advisable and if any customers have any questions or concerns about their Cyber security posture or organizational readiness, please contact MSP #1 for advisory services on next steps.
Managed Services Provider (MSP) #2
We used MSP #2 to migrate a Client from a colocation data center to Microsoft Azure, including servers, storage, networks, desktops-as-a-service, SIEM, SOC, monitoring, and security services. Post migration we continued to use MSP #2 for 7×24 onsite and Cloud support services.
I received a call from the President of this firm on the day the news publicly broke about the SolarWinds breach. He told me his engineering team has been investigating some very strange client systems anomolies and, when his team escalated into Microsoft Engineering, they were told to stick with it and it was not a false positive. It turned out Microsoft was also in the process of diagnosing what was going on and couldn’t disclose anything.
Later that day an email and a phone call from MSP #2 arrived alerting me, and my clients, to the vulnerability and what to watch for. MSP #2’s advice focused on the increased level of Phishing attacks they were witnessing where they thought it was connected to the SolarWinds vulnerability. Useful information to send to users and something on which I can act.
A day later I received this email:
Earlier this week we at MSP #2, a Microsoft premier Cloud Solution Provider, discovered what appeared to be a symptom of the SolarWinds Orion breach. The symptom presented itself as an authentication issue in the Marketplace Portal. After extensive research and after working closely with Microsoft, we don’t believe any of our clients are at a direct risk. To further protect our customers, the MSP #2 team has deployed custom Azure Sentinel Analytics that will alert us to known characteristics of the breach for our Managed Services clients. For those clients not directly managed with our Sentinel deployments, we have enabled log analytics and alerts in those tenants to notify those clients and MSP #2 of possible breaches.
If you have questions or concerns, please contact us at MSP #2 email address
Below are a few links that we have found useful and that we wanted to share.
Highly Evasive Attacker Leverages SolarWinds Supply Chain to Compromise Multiple Global Victims With SUNBURST Backdoor
Cybersecurity & Infrastructure Security Agency: Active Exploitation of SolarWinds Software
National Cyber Security Centre (UK): Dealing with the SolarWinds Orion compromise
GitHub: FireEye Mandiant SunBurst Countermeasures
GitHub: description of arbitrary code execution vulnerability found in SolarWinds Orion
MSP #1 didn’t send an email, gave somewhat of a less than authoritative message in their phone call, and didn’t commit to doing anything for hte client.
In my opinon MSP #2 went way over the top. They not only investigated this further, demonstrated it existed, found a way to monitor it, implemented the monitoring solution, and didn’t charge their clients for performing any of those activites.
Who do you want having your back?
Conclusion
When searching for managed servcies providers, after all the marketing material and paperwork, what you really want is someone who has your back. You want them to go the extra mile when solving problems and you want them to actually solve problems rather than continue to fix symptoms.
Selecting a Managed Services Provider is not easy. Not only do you have to find a company with the right skills, but you need to make sure their culture fits your culture. The relationship between the two firms can’t be as company and vendor. It needs to be as partners with each party making the other better.
At Harvard Partners, we perform a lot of Managed Services RFPs and get to meet many MSPs. One of the tests we give presents the following problem to the prospective MSP:
“The client is an investment management firm that trades stocks on the New York Stock Exchange. It is 9:25 AM, the systems supporting the Traders are down, and trading begins at 9:30 AM. What do we do?”
For those MSPs who tell us to call the Help Desk we politely get up, thank them for their time, and leave the meeting. For those MSPs who recognize this is a business disaster and say we should contact whoever we can reach immediately, including the President of the MSP, they usually get our client’s business.