Routine maintenance on Facebook’s network stitching together its data centers caused Monday’s collapse of its global system for more than six hours, the company has said.
The outage was the largest that Downdetector, a web monitoring firm, said it had ever seen. It blocked access to apps for billions of users of Facebook, Instagram and WhatsApp, further intensifying weeks of scrutiny for the $1 trillion company.
At a US Senate hearing on Tuesday, a former employee turned whistleblower accused Facebook of putting profits before people’s safety, which the company denies.
In a blog post, Facebook VP of Engineering Santosh Janardhan explained the company’s engineers issued a command that unintentionally disconnected Facebook data centers from the rest of the world.
While users lost access to one of the world’s most popular messaging apps – as WhatsApp has more than 2 billion users – employees were also blocked from internal tools.
The outage knocked out tools that engineers would normally use to investigate and repair such outages, making the task even more difficult, Facebook said.
The company said it sent a team of engineers to the location of its data centers to try to debug and restart the systems.
However, it took the company extra time to get engineers inside to work on the servers due to the high physical and system security in place.
Facebook added that its program audit tool had a bug, and failed to stop the command that caused the outage.
“Every failure like this is an opportunity to learn and get better,” Janardhan wrote.
“From here on out, our job is to … make sure events like this happen as rarely as possible.”
The team at Platform Executive hope you have enjoyed the ‘Maintenance error caused Facebook’s 6-hour outage, company says‘ article. Automatic translation from English to a growing list of languages via Google AI Cloud Translation. Initial reporting via our official content partners at Thomson Reuters. Reporting by Sheila Dang in Dallas. Editing by Sonya Hepinstall and Grant McCool.
You can stay on top of all the latest developments across the platform economy, find solutions to your key challenges and gain access to our problem-solving toolkit and proprietary databases by becoming a member of our growing community. For a limited time, our subscription plans start from just $16 per month.