Sudhakar Ramakrishna was sitting at a birthday meal with his family when he got the call: SolarWinds had suffered a large-scale cyberattack. The date was December 12, 2020 and Ramakrishna was expected to become CEO in a few weeks.
The extent and seriousness of the incident were not immediately apparent, but he still had a decision to make. Would he abandon the ship, which had caused a leak under the leadership of the previous captain, or would he grab a bucket and start bailing out?
A number of close confidants advised Ramakrishna to give up the post, while others suggested his skills and experience in cyber security makes him the ideal person to lead the recovery.
Although he took some time to consider his options, the decision to stay the course was ultimately simple, Ramakrishna said. Tech Radar Pro. The board was told it would back down if it was decided that SolarWinds would benefit from continuity, but was otherwise prepared to steer the company through the crisis.
In the weeks that followed, Ramakrishna began collaborating with the management team behind the scenes. The first priority was to find out exactly what happened and how, and the second was to formulate an action plan that SolarWinds could take to its customers, partners and the press.
“The idea that an attack can happen to anyone has become more prevalent, but that doesn’t absolve you of the fact that it happened to you,” he said. “Every business will experience a crisis or two, but what matters is how management reacts.”
A turbulent start
The attack itself had actually started several months earlier, in September 2019, when a sophisticated group of cybercriminals suspected of having ties to the Russian state first gained access to the SolarWinds network.
The threat actors showed remarkable patience, hiding in plain sight while they painted a comprehensive picture of SolarWinds’ infrastructure and the company’s product development process.
Among the various SolarWinds products, the attackers were particularly interested in a computer performance monitoring service called Orion, which needs privileged access to customer systems to function as intended.
After a first test, hackers injected a malware strain known as SUNBURST in an Orion software update sometime between March and June 2020. The Poisoned patch has been shipped to approximately 18,000 SolarWinds customers, giving attackers virtually unlimited access to the networks of government agencies, security companies and multinational corporations.
“The industry is not new to security issues, but each has its own twist and meaning – and that was important in its own way,” Ramakrishna said.
“The craft used to create the breach was not trivial, it was a supply chain attack. It’s a well-known concept in security, but not well practiced.
What makes an attack of this type so difficult to detect, he explained, is that the threat actor only needs to modify one of thousands of files to carry out an attack. which results in the compromise of a large number of targets.
In the end, the group chose to infiltrate only a subset of the compromised organizations – including Microsoft, Cisco, VMware, Intel and a number of US federal agencies – but the attack was nevertheless described as the one of the most important in history.
When SolarWinds was alerted to the incident by security firm FireEye, which had detected unusual activity on its own network, the company went into crisis mode. And it was in this climate that Ramakrishna walked through the doors on his first official day in charge.
However, while staff morale was as expected and conversations with angry customers were often difficult, the crisis at least provided a platform for Ramakrishna to lean on.
“In some ways, it’s easier to make changes in the middle of a crisis,” he told us. “When everything is perfect, there is a lot of resistance, but when a company is in shock, people are receptive to new ideas.”
On January 7, 2021, Ramakrishna posted a blog post which outlined what had been learned about the attack so far, offered immediate steps to help customers navigate the incident, and set out a new framework to prevent a similar attack from happening again in the future.
The supply chain puzzle
Although SolarWinds has managed to recover over the past twelve months, with customer retention now returning to roughly pre-attack levels, the incident has had a severe impact on the company’s bottom line.
Instead of channeling resources into product development, sales and demand generation as a normal company would, the company was forced into recovery mode, with its reputation in tatters.
Ramakrishna and his management team split the list of clients and began meeting with many of them individually, both to apologize and explain what had happened, and to help them determine if their own networks had been hacked.
He described it as a very uncomfortable but essential part of the “healing process” that eventually paved the way for a return to normal business operations.
However, despite the consequences for SolarWinds, there is evidence to suggest that the right lessons have not been learned by the wider cybersecurity industry. Since the attack, a number of similar high-profile incidents have taken place, such as the Kaseya attack, Log4j, and, even more recently, the Okta-Lapsus$ breach.
When asked why he thinks supply chain attacks keep happening, Ramakrishna explained that the disjointed nature of collective defense gives the attacker a significant advantage from the start.
“It’s not just a technology problem, there’s a lot more to it,” he said. “Each of us defends ourselves against an aggressor. But on the one hand, there is a coordinated army with a single objective, to attack, and on the other, a collection of fragmented soldiers.
Ramakrishna also criticized the culture of victim shaming, which he said contributes to companies’ reluctance to share vital information.
“There is still a lot of victim shaming, so companies often end up solving the problems without saying anything about them. There is definitely a reluctance to speak up,” he told us.
“In the event of an incident, it is important to get help from the community. We need to make people aware of the issues faster; this state of mind must prevail in software security.
To prevent a supply chain attack of this magnitude from happening again, Ramakrishna also believes companies need to embrace a new security framework, which he calls “secure by design.”
The model has three components: infrastructure security, building system security, and the design of the building system itself. But the general idea is to keep changing the attack surface, so as not to provide an attacker with a fixed target, and to minimize the window of opportunity.
With this goal in mind, SolarWinds has created a “parallel build system” in which its software is built in three separate locations, which can be changed dynamically. The result of each individual construction is then cross-checked with the others to eliminate inconsistencies that could betray an attack.
To successfully infiltrate a hotfix, an intruder would therefore have to launch three attacks simultaneously, at exactly the same time and using exactly the same technique.
“It’s a very difficult thing to do, even for the most persistent cybercriminal,” Ramakrishna said.
The new look of SolarWinds
Ironically, it has been suggested that SolarWinds could now be considered the safest company in the world. After all, no other organization has faced the same level of scrutiny since the attack was discovered.
Ramakrisha declined to be taken to say whether or not he believed that characterization to be accurate, but said it was something the company was “determined to make true”.
Operating within its secure design framework, SolarWinds will now look to build on its foundations in IT monitoring and evolve into a company that can support customers’ hybrid needs, both in the cloud and on the spot.
Ramakrishna promised an increased level of automation and superior visualization and correction facilities which together will help solve the kinds of problems created by digital transformation. The goal is to “reduce complexity, improve productivity and reduce costs” for customers, we were told.
As a few rays of sunshine now begin to pierce the cloud hanging over the company, Ramakrishna is eager to focus on these central goals. But as our conversation drew to a close, he also took a moment to caution against complacency:
“No business, no matter what it does, should believe it is immune to attack, because that is a mistake,” he said.