In July 2024, Managed Service Providers (MSPs) faced a significant challenge when a faulty update from CrowdStrike, a leading cybersecurity firm, disrupted the operations of countless businesses worldwide. This incident highlighted the vulnerability of MSPs to third-party software issues, especially when the software is as critical as endpoint protection. For MSPs, whose reputation and business continuity are deeply intertwined with the reliability of the services they manage, the CrowdStrike incident was a wake-up call. This blog post will discuss the challenges MSPs encountered during this crisis and propose strategies to mitigate similar risks in the future.

The Fallout of the Faulty Update

Operational Disruptions

The faulty CrowdStrike update led to widespread system failures, with many endpoints becoming unresponsive or, worse, entering a continuous reboot loop. MSPs were suddenly inundated with client complaints, facing an overwhelming demand for immediate solutions. The scale of the problem meant that even the most robust support teams were stretched thin, leading to delays in resolution and, in some cases, loss of client trust.

Reputation at Stake

MSPs act as the frontline of defense for their clients’ IT infrastructures. When a critical service like CrowdStrike fails, the immediate assumption from clients is that their MSP is at fault. This perception, fair or not, can severely damage the MSP’s reputation. Clients expect seamless service and immediate solutions; when these expectations aren’t met, they may start questioning their choice of provider.

Financial Impact

Beyond the immediate operational headaches, the financial impact of the CrowdStrike incident was significant. MSPs had to dedicate countless hours to troubleshooting and remediation, often at their own expense. Moreover, the disruption could have led to clients seeking compensation or, in extreme cases, terminating contracts. The financial strain of such events can be crippling, especially for smaller MSPs operating on thin margins.

Lessons Learned: A Path Forward

1) Diversification of Critical Services

One of the key lessons from the CrowdStrike incident is the danger of over-reliance on a single vendor for critical services. While CrowdStrike is a leading name in cybersecurity, this incident underscores the importance of having a diversified portfolio of security tools. MSPs should consider implementing a multi-layered security approach, combining different solutions to ensure that if one fails, others can compensate. This approach not only reduces risk but also provides a broader range of features and capabilities to address various security challenges.

2) Enhanced Vendor Management and Communication

The CrowdStrike incident highlights the need for MSPs to have robust vendor management practices. This includes regular communication with vendors to stay informed about updates, patches, and potential issues. MSPs should establish clear channels of communication with their vendors, ensuring they receive timely information about any potential risks or problems. Additionally, MSPs should demand transparency from their vendors regarding their testing and quality assurance processes. Knowing how rigorously an update has been tested before deployment can help MSPs make informed decisions about whether to implement it immediately or wait for further validation.

3) Proactive Monitoring and Rapid Response Plans

One of the critical challenges MSPs faced during the CrowdStrike incident was the speed at which the problem escalated. To mitigate the impact of similar incidents in the future, MSPs should invest in proactive monitoring tools that can detect unusual behavior or system anomalies early. By identifying potential issues before they become widespread, MSPs can respond more quickly and minimize disruption.

In addition to monitoring, MSPs should develop and regularly update rapid response plans for handling software failures. These plans should outline the steps to take in the event of a critical service disruption, including communication strategies with clients, immediate remediation steps, and escalation procedures. Having a well-defined plan in place ensures that when issues arise, the MSP can act swiftly and effectively.

4) Client Education and Communication

During the CrowdStrike incident, many MSPs found themselves on the defensive, trying to explain the situation to frustrated clients. This experience highlights the importance of client education and communication. MSPs should proactively educate their clients about the complexities of IT environments, including the potential risks associated with third-party software updates. By setting realistic expectations and maintaining open lines of communication, MSPs can build stronger relationships with their clients and reduce the likelihood of reputational damage during a crisis.

Regular status updates and transparency about the steps being taken to resolve issues can also help maintain client trust. In the event of a disruption, MSPs should communicate clearly and frequently with their clients, providing them with as much information as possible about the situation and the expected timeline for resolution.

5) Insurance and Legal Preparedness

The financial repercussions of the CrowdStrike incident have underscored the importance of having appropriate insurance coverage and legal protections in place. MSPs should review their insurance policies to ensure they include coverage for third-party software failures and associated business disruptions. Additionally, MSPs should work with legal counsel to draft or update contracts with clients, clearly outlining the responsibilities and limitations of the MSP in the event of a third-party failure.

Building Resilience for the Future

The CrowdStrike incident of July 2024 was a stark reminder of the vulnerabilities inherent in the MSP business model. However, it also provided valuable lessons on building resilience. By diversifying critical services, enhancing vendor management, investing in proactive monitoring, educating clients, and ensuring financial and legal preparedness, MSPs can better navigate the challenges posed by third-party software failures.

In an industry where trust and reliability are paramount, MSPs must take these lessons to heart. By doing so, they can not only mitigate the risks of future incidents but also strengthen their value proposition to clients. As the IT landscape continues to evolve, the MSPs that adapt and prepare will be the ones that thrive.