Australian productivity tools maker Atlassian is having a very bad week.
An outage that began on April 4 – yes, more than 10 days ago – continues to cause problems for the company, affecting around 400 of its customers. Given the size of the company, these customers could be heavy hitters.
In a blog post, Atlassian CTO Sri Viswanath elaborated on what exactly is happening, after several days of silence. As of April 14, approximately 55% of data subjects had regained access to their data and services.
After explaining that there was a “communication gap” between the tweets and the fact that the wrong set of IDs had been provided, Viswanath gets to the heart of the matter.
“[T]The script we used provided both the “mark for deletion” functionality used in normal day-to-day operations (where retrievability is desirable) and the “permanently delete” functionality that is needed to permanently delete data when needed to compliance reasons,” he wrote.
“The script was executed with the wrong execution mode and the wrong list of identifiers. The result was that approximately 400 client sites were incorrectly deleted.”
Update on cloud outage affecting approximately 400 customers. As part of planned maintenance, our team ran a script to remove legacy data from an outdated service. Instead of deleting data, the script mistakenly deleted connected sites and products, users, and third-party applications. (1/5)April 12, 2022
The fix is in
Atlassian, of course, worked hard to rectify this error and restore customer data.
As of April 15, about 55% of customers have recovered this data, the company saidand automation is used to speed up the rest of the recovery.
Atlassian maintains extensive backup and restore systems, and no data loss has been reported for customers who have been restored to date. This incident was not the result of a cyberattack and there was no unauthorized access to customer data. (2/5)April 12, 2022
The fact that it was not a cyberattack and that the company keeps full backups means that any data loss is likely to be minimal. But that’s still a problem for a company whose customers depend on it.
In the hyper-competitive space of cloud services, a simple mistake like this – causing pain for paying customers – could force some to look elsewhere.
“We know incidents like this can erode trust,” Viswanath says. “We are not living up to the high standards we have set for ourselves. This includes our communication efforts, which until now have been entirely focused on directly reaching our affected customers.”