The SolarWinds SUNBURST Supply Chain Attack

Security Researchers have dubbed the supply chain attack against SolarWinds’ Orion platform as “SUNBURST.”

SolarWinds’ Orion is a network and infrastructure management platform used by thousands of companies and government agencies across all sectors and missions.  From oil and gas, to retail, manufacturing, finance, technology, critical infrastructure, government services, and national security, this tool is used to simplify the management of large and complex networks, systems, and infrastructure.  It is advertised as a “single pane of glass” solution to allow network products, IT operations products, and security products of all kinds to be operated and managed from a single console.

This consolidation is accomplished by allowing the Orion platform to have full access to the systems it manages, which means that anyone who gains access to the platform essentially has the keys to the kingdom, which is exactly what happened in the case of organizations like FireEye, the US Treasury, the NSA, and even Microsoft.  The threat actors, who are being described as sophisticated nation-state attackers, took control of the SolarWinds Orion platform at more than 50 organizations and potentially every SolarWinds Orion customer: more than 18,000 companies, schools, universities, agencies, etc.

The nation-state attackers were able to compromise so many organizations at once, not because SolarWinds had coded the product poorly or because there was a vulnerability on the systems that allowed the platform to be readily compromised, but because attackers were literally able to build in a backdoor, and every customer walked it right past their security stack as a trusted update, in essence holding the door open and politely inviting the attackers in the front door.

What makes the SolarWinds attack so uniquely troubling is that it did not just compromise the security of the product, it compromised the actual development cycle that produces the product.  All major enterprise applications require maintenance.  Windows, for example, gets automatically updated, or patched, at least once a month.  These updates and patches are how holes and errors in software, that cause instability or make it vulnerable to attack, are closed and corrected.

When a software company produces new versions or patches, they digitally sign their code/package so that the end user, or receiving organization, can trust that the software really did come from the company that wrote it and presumably verified that it is correct and secure.  In a fully mature software development shop the process of signing code could go all the way down to the individual developers and allow for a full tracking of the provenance and work product of any given release or update.  Code signing is currently the best practice in place to validate software and it is a general practice across industries to trust, deploy, and use signed code from known suppliers to update systems and software.  This is done because it would be impractical for an organization to individually test each and every release of every product. Almost all organizations lack the tools, personnel, and skills to even consider an approach where every release of every piece of software is tested end to end.

The whole process from the whiteboard in a design session where new software often begins, to the keyboards of the coders, and the tools of the testers, through sale, distribution, licensing, and delivery, all the way to the deployment of the resulting software at an organization is the digital supply chain that feeds enterprise systems across the globe.  If any stage in the supply chain is compromised, it cascades to the next.  The earlier the stage of compromise, the greater potential for downstream impact.

We now know that SolarWinds’ development cycle was compromised for the Orion platform.  We know that attackers were able to insert their own code into the Orion codebase and include masked backdoors and various other components that would facilitate reconnaissance and compromise of targets downstream.  Once the malicious code became part of the signed update, it was downloaded and deployed by tens of thousands of customers.  Even downloading a free trial of the software likely puts an organization at significant risk.  Even worse, free trials and evaluations often are not part of the organization’s software inventory and as such, there may be organizations out there who think that they are not vulnerable to this attack, when, in fact, they are.

Supply chain attacks represent the worst-case scenario in a cyber-attack, because they are virtually undetectable until the threat actors start to leverage them. SUNBURST is the worst such attack known to date.

Any organization using SolarWinds Orion has some immediate risk decisions to make.  The first being to decide whether or not to shut down the platform immediately.  All federal agencies using SolarWinds were ordered to do exactly that last week, but the nuances of this decision are likely to differ from organization to organization.  When considering the risk, look to understand what the platform is doing for the organization.  If left unmonitored and or uncontrolled by Orion, will business critical systems in the environment become unstable and crash?  How far into the enterprise does SolarWinds reach?  Are business critical systems on the same network segment as SolarWinds?  Are there other controls in place to limit an attacker’s ability to freely move across the enterprise using stored credentials stolen from SolarWinds?

It is critical to ask questions and consider alternatives to normal operations.  Given the nature and depth of this compromise, organizations should operate under the presumption that they are already compromised and work backwards to decide how to stay up and running while improving security to counter the threat.  It is likely that organizations will need to consider augmenting or increasing security staff, and that organizations may need to consider making changes to enterprise architecture and adding additional layers of security tooling and controls.

Ask questions about security fundamentals.  Consider turning on Multi-factor Authentication; examine network and system segmentation; improve visibility and consider adding discovery tools; ensure that vulnerabilities are being tracked and remediated; keep systems up to date (yes, even though the supply chain was compromised); ensure the organization is keeping good backups and test the ability to recover; use this attack to get the business and security together and run through what the organization would do if it were next on the attackers’ list.

There are many unknowns that remain in the aftermath of this attack, and organizations will need to remain vigilant and ready to respond as the fallout continues and the next attacks occur.

Over the coming days, weeks, and months, more details will emerge and new impacts will be discovered.  Currently more than 50 organizations, including some of the largest companies in the world and the most sensitive government agencies, are known to have been actively impacted by this attack.  The final total could be orders of magnitude higher.   Organizations who used or tested an up-to-date version of SolarWinds in their environment since at least March of 2020 should assume they are compromised and respond as such.

Expect the unexpected. It is unknown how far and where else this attack, and others like it, may go.  It is possible that the attackers used their access from this attack to compromise the development cycle at other major organizations.  Consider what your approach will be to handling the next announcement of compromise from another one of your suppliers.

Organizations should also consider how they will handle the scenario where a company they work with is impacted by this attack.  Are the organizations’ systems connected in any way?  What controls are in place?  What processes could the attacker leverage at the other firm to move to other organizations?  Many of the largest breaches in the last 10 years have been the direct result of external, trusted, third parties being compromised and providing the threat actors access to their primary target.  SUNBURST targeted the supply chain to do this and it could have paved the way for more traditional third-party attacks across the board.

Another consideration is that once the attack vector becomes known and the tools used to perpetrate SUNBURST become public, other threat actors will seek to emulate or leverage the same attack elsewhere.

Ask questions!  Ask about the inventory of software in the organization.  Ask about network visibility and any constraints.  Ask about asset inventories, discovery, and tracking.  Ask what has already been done.  Ask what else needs to be done.  Ask when and about how the last time the organization’s security was tested.  Ask what controls are in place.  Ask what controls are missing.  Ask when the last security program assessment was and what was done with the output of that assessment.

Be prepared to invest in security.  Security is generally recognized as being underfunded and under-resourced and even today is still viewed in many organizations as a cost center or a compliance checkbox.  To find the answers to these questions and all of the questions that will inevitably follow, IT, Security, the Business, and Executives will have to work together.

Get started.  This issue may feel entirely daunting, even if most of the questions above can be readily answered, so the most important thing to do is start.  The first thing to do is determine if the organization can function without the impacted SolarWinds components running.  If the answer is yes, then CISA recommends that all federal agencies should disconnect their SolarWinds instances, take forensic images and memory dumps of them, and then power them down.  That is potentially good advice for any organization that can afford the business impact of taking that approach to start triaging and containing the incident.  The next two things that can be done at all organizations are to start gathering evidence (all available logs, system images, metadata, memory dumps, etc.), and to read up on the attack.  The first place to start reading is the page setup by FireEye, the group responsible for identifying the attack and notifying the rest of the community, but not before their offensive tools were stolen by this sophisticated attacker.  FireEye provides a good executive summary and detailed information on how to identify trojanized instances of SolarWinds (the actual SUNBURST backdoor malware), as well as information on how to identify beaconing behavior and memory-resident artifacts associated with this attack.

FireEye also published a GitHub repository of countermeasures for all 300 offensive tools that were stolen from them as a result of this attack.

The next resource to consider is the incident page set up by SolarWinds itself.  While some platforms are listed by them as “no action required,” it is advisable to use extreme caution and remain vigilant if any SolarWinds products have been used, or even tested for use in an organization’s environment.  In their writeup, SolarWinds discusses two separate attacks as if they are the same.  They reference SUNBURST (the source of the current, highly sophisticated, attack), and they also reference a separate campaign known as SUPERNOVA.  SUPERNOVA is separate and should be taken seriously and investigated thoroughly, but is likely to require a distinct response and investigation.

The Cybersecurity and Infrastructure Security Agency (CISA) is the agency under the Department of Homeland Security tasked with a broad mandate to provide for the cybersecurity and general protection of national critical infrastructure and government systems.  Many federal agencies have been impacted by SUNBURST as well as many different critical infrastructure components and their providers.  As such, CISA has taken an active role in reviewing, investigating, and providing guidance in response to SUNBURST.  CISA is providing information on their website in the form of guidance, intel, and executive summaries.

Information on the ongoing threat can be found here.

Information on general response guidance for leadership can be found here.

Information on supply chain compromises, like SUNBURST and others, can be found here.

Links to other information on the active exploitation of SolarWinds can be found here.

The CISA release providing details on the APT implicated in this attack can be found here and provides three categories of response for SolarWinds product owners.

Although the guidance and technical specifics for response continue to evolve and will vary from organization to organization (and may be complicated by the addition of SUPERNOVA as a response consideration), there are certain steps that any and every organization should take starting TODAY.

All organizations should continue to check the latest updates available through any credible source.  Since the initial disclosure, many more details have surfaced that indicate the extent of the compromise to be far greater than first suspected and therefore affecting many more potential victims.  Until the investigation is completed the totality of the impacts remain unknown.  Organizations need to keep up with this, and the threat landscape in general, to understand the extent of their risk against the evolving threat(s).  Operationally, organizations should evaluate and consider adding to their internal monitoring and response capabilities to ensure that the organization has the ability to act on additional intel and identify malicious activity as the threat actors traverse beyond the primary entry point (the SUNBURST backdoor).

If an organization is in need of additional resources, the damage done while hesitating to seek help could be immeasurable. The nationwide response to this threat has been overwhelmingly positive and Apollo Information Systems stands ready to help and provides a full spectrum of services and product partners to ensure that our clients weather this and future cyber storms.

For more information or to request assistance contact Andy Bennett, Apollo’s VP of Technology, CISO at andy@apollo-is.com, or call at 936-647-3151.