Not only are RXT stock traders impatient, MSPs around the world went into crisis mode as Rackspace Hosted Exchange (HEX) servers caused many businesses to come to a complete halt, with outages that can cost millions of dollars a day for large enterprises, and tens or hundreds of thousands for service providers and the SMBs they support.
On December 2, Rackspace identified an issue affecting the company’s Hosted Exchange environments. This appears to have caused problems for many Microsoft (News - Alert) 365 users. RXT stock traders reacted by dumping their shares.
On day one, Rackspace said it was “investigating reports of connectivity issues to [its] Exchange environments,” and noted “users may experience an error upon accessing the Outlook Web App (Webmail) and syncing their email client(s).”
Not until December 3 did Rackspace call it a “security incident.”
And as of December 6, the Rackspace homepage had a banner saying the company had “determined that this was the result of a ransomware incident. We’ve engaged a leading cyber defense firm to investigate alongside our internal security team. If we determine sensitive information was affected, we will notify customers as appropriate. Based on information to date, we believe that this incident was isolated to our Hosted Exchange business and other services remain fully operational. Out of an abundance of caution, we have put additional security measures in place and will continue to actively monitor for any suspicious activity. All available resources have been mobilized to support customers in migrating their users and domains to Microsoft 365.”
Pax8, a fast-growing cloud marketplace and PSA-integrated platform company, immediately began reaching out to its partners and posted this advice in a Pax8 blog post written by Dom Kirby, Director of Cloud Services: “For every professional responsible for ensuring critical cloud infrastructure availability, this is an opportunity to evaluate the use of legacy hosting and to beef up incident response (IR) and disaster recovery (DR) plans to be prepared for potential outages or incidents.
“Your IR and DR plans should include scenarios in which your cloud provider(s) is/are unavailable. Incidents and outages of this magnitude are exceedingly rare and unlikely for hyperscalers like Microsoft and Amazon Web Services (News - Alert), but the hyperscalers are not immune from them. Preparations are crucial for even minor outages and incidents such as normal service interruptions or local compromise of your own tenant.”
Kirby, who is also a cybersecurity expert, recommended MSPs ask themselves immediately:
- What critical applications and services are running on which cloud or clouds (in this case, email was down, which dealt a crushing blow to organizations hosted on HEX)
- How are we backing it up?
- How do we restore it?
- How much downtime can the company survive?
- How much downtime do we tolerate before we enact our disaster recovery process?
- What is our backup provider going to be?
- How are we going to get critical data into that backup provider?
- How are we going to get our users up and running?
“Even if you’re in the cloud, you still need backup,” Kirby wrote in the Pax8s blog post. “Most small and medium-sized businesses (SMBs) and large enterprises leverage Microsoft 365 for their communications, collaboration, and security. However, third-party backups, at a minimum, are non-negotiable. Data needs to be in more than one place, always.”
Down Detector posts came in fast and furious on the Rackspace page, including this from Becca around 3 AM Eastern:
“There's no point in contacting support as you'll get an automatic response. I also had a similar issue and after filing several complaints I didn't get any helpful response and finally got help from a tech who telegraphed @Hacker_smith45... he's a genius…”
Becca got lucky and put a positive spin on things after getting creative.
Others were less positive, with Fitz posting, “They have successfully restored email services to thousands of customers on Microsoft 365 by forcing them to another hosting company and then claiming that as progress! Awesome job! Fanatical Support, Ha!”
According to a 2021 Gartner survey, 86% of I&O leaders self-assessed their recovery capabilities as meeting or exceeding CIO expectations. Yet only 27% of that group consistently undertook three of the most basic elements expected of a DR program — formalizing scope, performing a BIA to acquire business requirements, and creating detailed recovery procedures.
According to the same survey, those with a solid disaster recovery program are 40% more likely to demonstrate a stronger overall resilience posture in other areas of reliability and tolerability.
Arti Loftus is an experienced Information Technology specialist with a demonstrated history of working in the research, writing, and editing industry with many published articles under her belt.
Edited by Erik Linask