SolarWinds Orion Breach - What It Means for the Industry Writ Large

Stories about the recently uncovered breach of the SolarWinds Orion software have been dominating the news lately, and the situation is still continuing to evolve. In this episode, we speak with Jonathan Condra, senior manager for strategic and persistent threats with Recorded Future’s Insikt Group, to get his perspective of what this breach is all about, where we stand in terms of attribution, what it means for the security community writ large, and whether or not a breach like this rises to the level of a “Cyber Pearl Harbor” or “Cyber 9/11.”

This podcast was produced in partnership with the CyberWire.

For those of you who’d prefer to read, here’s the transcript:

This is Recorded Future, inside threat intelligence for cybersecurity.

Dave Bittner:

Hello everyone, and welcome to episode 191 of the Recorded Future podcast. I'm Dave Bittner from the CyberWire.

Stories about the recently uncovered breach of the SolarWinds Orion software have been dominating the news lately, and the situation is still continuing to evolve. In this episode, we speak with Jonathan Condra, senior manager for strategic and persistent threats with Recorded Future’s Insikt Group, to get his perspective of what this breach is all about, where we stand in terms of attribution, what it means for the security community writ large, and whether or not a breach like this rises to the level of a “Cyber Pearl Harbor” or “Cyber 9/11.” Stay with us.

Jonathan Condra:

As I recall, it was late on a Saturday night prior to the holidays, I forget the exact date of course, when our CEO actually flagged a story in the analyst channel for us asking, "Hey, we need a note on this." And at the time, it was the initial news story around the breach of the Commerce Department, and I forget the actual federal organization, but something to do with transportation, if I recall.

And at the time, we drafted something up very quickly on the weekend, which is rather odd, to some extent. So we knew it was probably significant, but we had no idea that it was going to explode into what it did. And of course, this had to have happened right before the holidays, as they tend to do.

Dave Bittner:

Yeah, yeah. Well, take us through, I mean, in terms of what we know now, where do we stand right now? Can you give us a little bit of the background of what happened and where we stand?

Jonathan Condra:

Yeah, sure. So for those who aren't aware, SolarWinds is a large U.S.-based software company that develops enterprise grade software to help its customers manage their networks and manage IT infrastructure, all their endpoints, et cetera. In particular, they create a product called Orion, so SolarWinds Orion, and that's the piece of software that was really affected. So SolarWinds in particular has about 300,000 customers, which is pretty significant, and 33,000 of which were notified in December by the company of the incident. But they've actually said that less than 18,000 were actually affected in some capacity by it.

That's still a huge figure. We can go into the scope of that later, what that breakdown looks like. But the high level is that attackers were able to gain access to SolarWinds, steal the code signing certificate, and then make malicious and unauthorized updates to a dynamic link library file, or otherwise known as a DLL, within the SolarWinds Orion product, starting back really actually in October of 2019, but the first malicious update is believed to have been pushed in March 2020, and that allowed them to backdoor. Basically, they opened a backdoor into organizations that were customers of Orion, if that makes sense. So this is a third-party compromise, a software supply chain compromise affecting these other organizations. And that initial backdoor has been called at least two different things by various research groups or companies, one being SUNBURST, and I think Microsoft calls it Solorigate.

Dave Bittner:

What is the intended functionality of SolarWinds Orion? If you were running this, what were you doing?

Jonathan Condra:

So, I mean, it helps you manage your IT infrastructure, be able to see what's installed on certain endpoints, be able to see who's talking to who, et cetera. It's an enterprise-level management suite that generally people in IT would use to make sure that everything is running smoothly and things can talk to one another, et cetera. There's a bunch of different modules and things, to my understanding, that you can purchase that do various things. But it's fundamentally not something that your average end user would use. It's something that's very specific and would probably be used by either the infrastructure team for a company IT or maybe even security to some extent.

Dave Bittner:

But by the nature of what it does it would be able to access many things?

Jonathan Condra:

Yeah, essentially. I mean, if they use SolarWinds on their network, it would have had visibility into essentially everything, and that's the concern here.

Dave Bittner:

So what do we know in terms of how they were able to get in and do the things they did to SolarWinds' software?

Jonathan Condra:

Good question. We don't really know, unfortunately. We don't have a lot of information on how they actually breached SolarWinds and got access to it. I mean, if you take a step back, the first big breach, aside from the aforementioned two government breaches that I mentioned, was FireEye dropping their notification that they had been compromised and that their Red Team tools had been stolen. But at the time, it hadn't even been linked to SolarWinds yet. And it took a little bit of time for them to even determine that the way that they were compromised was via the SolarWinds Orion package. But we still don't have any insight into actually how they initially got initial access into the SolarWinds, the company, to be able to do this to begin with.

Traditionally, APT groups have utilized common techniques like social engineering, via spear-phishing, for example, or email to deliver malware or harvest privileged credentials to move laterally or to elevate their own privileges, et cetera, within target networks. But they've also used the other things like watering holes, for example, tend to be a little bit less targeted. Malicious insiders is something that I think we need to consider here. And ironically enough here, third-party compromise is also a possibility. We just don't know.

There was something interesting that came in over the weekend in the New York Times. It was a little bullet point in an article that said that it was discovered that some of the SolarWinds software is developed in Eastern Europe. And so apparently there's an investigation into whether or not that has an access to it. I would hazard, I would caution, rather, that a lot of companies outsource the development, and just because they're based in a certain region of the world doesn't mean that's where it originated from. But that was an interesting tidbit that did come out.

Dave Bittner:

Yeah. Well, on January 5th, the Cyber Unified Coordination Group, which is a government taskforce, they came out and said that this was likely Russian in origin in terms of attribution. And from my point of view, I mean, that's been a really fascinating aspect of this whole thing, is that there've been lots of folks saying that this is likely the Russians. But am I right in my perception that there's been pretty thin evidence?

Jonathan Condra:

Yeah, you're absolutely right. So I just want to just first set the table for myself as well as Recorded Future, we're not making any claims of attribution at this time. My personal opinion on the subject is that this is obviously a very serious intrusion. If you quote CISA and the U.S. government here, "This is a grave threat to national security." We're not involved in the investigation. We don't have any particular insight into classified materials that might be going on. We haven't been briefed on it, et cetera. So my personal opinions here are not representative of potentially the reality, if that makes sense.

And so we're not making those claims just based on that. But what I would say is very quickly after this broached, after this happened, rather, there were reports both in the media and even statements from government officials, I think James Inhofe from Oklahoma, who's the Chairman of the Senate Armed Services Committee, and a few others came out and said that this was Russian in origin. And I think even some media reporting had explicitly tied it to APT29, which has been linked to the Russian Foreign Intelligence Services, otherwise known as the SVR. But not a lot of public information to back that up.

And if there's even some of the indications from conversations that I've seen in the community today, people are still asking the question, "What do we really have to tie this to Russia at the moment?" And I think the answer is not much, at least in the open source. I suspect that on the backend, on the high side, on classified materials, there's probably a fair bit of information that for whatever reason ties it to Russia, whether it's reused infrastructure or maybe even human intelligence sources within Russia or in region that know or are well-placed to know the types of operations that were being run. And so I think we have to be very careful with assumptions that we make, but also, I think we need to let the process play out, if that makes sense.

Dave Bittner:

No, it does. I mean, and to your point, I mean, what I've seen folks saying is that, and again, please correct me if I get any of this wrong, but that the tools that they used were pretty much all original. So it wasn't like you could look at a bit of code or something and say, "Oh, this is just like what the Russians used in this other campaign." But then at the same time, you've had some folks who I suppose you could say through their experiences might be in a position to know, have hinted that, "Yes, it probably was the Russians, and we think it was, but we can't quite tell you why yet."

Jonathan Condra:

Yeah. I mean, I think you're right. There's actually two strands to this. The first is the SUNBURST backdoor, as well as the first stage malware that was dropped, which if I recall correctly was called TEARDROP, but they also saw that was a memory-only dropper that would drop a Cobalt Strike beacon in at least one instance. And Cobalt Strike is an open source tool. That's not necessarily a Russian thing. So the evidence from a technical perspective is rather thin, at least to our knowledge at the moment, for Russia. That said, there's probably something else, is my guess, whether it's within the government or any of the other organizations that have better visibility globally.

But then there's the other aspect of this, and this came out maybe a week or two after the initial news, that there might even be a second threat actor group involved. And that's a very interesting, if not a little disconcerting, angle to this, where some of the other malware samples that have been identified, and I believe they're called SUPERNOVA and CosmicGale, don't seem to overlap at all with the previous samples that had been observed and don't seem to be related infrastructurally or anything like that to the initial threat group that has been at least tentatively linked to Russia. So that's a very interesting angle here. And whether or not that's a separate Russia group, or it's a criminal group, or whoever it may be, we may be actually looking at two separate things here.

Dave Bittner:

Well, I mean, obviously this grabbed all the headlines and I think appropriately so, but if I'm a security person and I'm in charge of protecting my organization, what should my stance be right now? Is this something that should have my attention?

Jonathan Condra:

I would think so. I mean, well, I would just say, SolarWinds on their website, prior to them pulling it down to protect the victims, said that they had 18,000 customers of ... Or, wait, it wasn't. It was the 33,000 customers of SolarWinds Orion, and 425 of the Forbes 500 as customers. And they listed a large number of very prestigious and powerful government organizations as customers as well. So I guess the implication there is just that there's probably a lot of high-level victims here and a lot of major organizations that were affected by this.

So all that is to say that, yes, I think most organizations have been, for the last few weeks, engaged in an incident response, an internal incident response effort. Whether they do that totally internally or they hire a third-party to come in and vet probably determines on their budget, et cetera. But that's the first step, I think, determining whether or not you were ever a customer of SolarWinds Orion and if you've ever had the specific affected versions, I believe it was between March 2020 and June 2020. And you can check the CISA guidance, therefore the specific versions that were affected, to see if that was ever on your network.

Actually, there's another interesting angle to this. Even if you weren't a direct customer, it's worth checking to see if you had any third-party contractors on your network that may have had it installed by their own organizations during that time frame or any IT personnel that may have installed the software for testing purposes. They did have a trial version that I believe was affected. And I know anecdotally, there have been a few organizations that were never customers of SolarWinds that have subsequently discovered that somebody in IT was playing around one day trying to see the utility of a tool and they may have been affected by that. So that's something to look at. Other than that, I would actually review your third-party vendors to begin with. Determine which other of your software or hardware vendors, et cetera, may themselves be customers of SolarWinds and vet their level of privileged access into your own network to determine if they could've moved laterally from that.

Dave Bittner:

Yeah. This strikes me as being a bit of a gut check for the whole industry. I mean, third-party breaches are something that had been talked about a lot, but when it actually happens and it happens at this scale and hits as many organizations as it did, it seems to me like there's, I don't know, maybe an unexpected emotional component.

Jonathan Condra:

Yeah. I mean, I think there is. I think part of that is the scale and probably the timing too, quite frankly, this happening right before the holidays after a year like this or like we've had just adds to everybody's stress levels. And during a presidential transition, of course,

I mean, I think this does, however, speak to the state of cybersecurity writ large in the U.S. and probably globally. There's a lot of different elements to securing an organization, whether it's cyber, physical, et cetera, and especially large multinationals and government agencies that run complex network architectures, run a diverse set of IT technologies and hardware, and even, depending on the type of organization, have differing requirements for data security based on data labels. So it's really easy, I think, for the general public to look at this, or even policy makers, to some extent, who don't have a background, to look at this incident and wonder, "How can this happen to such esteemed and well-resourced organizations, even intelligence organizations that should know better?"

But the reality is that it is a very difficult and complicated problem. And even if every organization had more than enough human and financial capital dedicated to cybersecurity, it's inevitable, to some extent. This could happen to anyone. There have always been breaches. The scale is definitely changing, we can get to that, but there've always been breaches of security. You can go back looking over the ... If you just want to limit it, look at all the compromises of the U.S. intelligence community over the Cold War. I mean, that's no different in some respects. The scale is a little bit scary, but I think that's an important thing to recognize here.

Dave Bittner:

No, it's a really good point. And I think it's also, I mean, related to that, it's interesting that there's been very little finger pointing, which I think speaks to your point that there's acknowledgement that this could happen to anyone.

Jonathan Condra:

I certainly hope so. I mean, I think some of the finger pointing right now is probably ... Maybe this is a little, I don't want to say dark, but a little bit pessimistic. I mean, I think maybe some of the finger pointing is a little dampened at the moment by the fact that everybody's hair is still on fire and we're still trying to figure out the scope of it. I do think there'll probably be an after action that's done after the full scope, after initial remediation is done and after the full scope is understood, I think there'll be a reckoning, hopefully at a national policy level to help us figure out exactly what went wrong and how we prevent it in the future.

But I just want to caution listeners that even the best policy frameworks aren't perfect and they can even have unintended consequences. So no matter what we do, this is going to happen again. But I do think this is probably a wake up call to the federal government that its current strategy is probably flawed. And I think it's time at a societal level to realize that this is a national problem that really requires the whole of government response and probably even changes to how we think about defending private networks from nation state adversaries.

I mean, thank God this was only a ... At least by the announcement from CISA and the NSA yesterday, thank God that this looks like it was just a cyberespionage campaign. I say just flippantly, but it could have been a lot worse. Like, if they had gone after critical infrastructure, for example, and turned off electricity and water and disrupted logistics systems, we would have a much different situation on our hands. And I think hopefully that this is a wake up call that, yes, we've invested a lot of money in detections for bad entering our networks or even looking at endpoints and removing anomalous files or isolating anomalous endpoints based on behavior. But we haven't really looked at the supply chain in the same way. And I think that this should be an alarm bell, if that makes sense.

Few organizations are well-equipped to alert on authorized access than they are on unauthorized access, if that makes sense.

Dave Bittner:

Yeah. Well, I mean to that point about espionage, I mean, one of the things that was noteworthy is in the immediate aftermath, you saw some folks, and in my mind, I remember particularly some politicians at the national level were saying things like this, "Was this a Cyber 9/11? Was this a Cyber Pearl Harbor?" I mean, do you have any insights on that in terms of how we should calibrate in our own minds for how we categorize something like this?

Jonathan Condra:

It's a good question and it's a very difficult one. I would shy away from viewing this as quote, unquote, an act of war or a Cyber 9/11 or Pearl Harbor. To my knowledge, nobody died in this incident. Yes, it may have grave implications for national security, and down the line that may have some of those effects, but there's generally a line drawn, at least in international relations and policy, between kinetic attacks on a state and associated infrastructure and cyber. I know that line is not very well defined and even in international law at the moment, which I think is something that should really take precedence at this point, but I don't see the comparison there. I do think maybe at a level of severity it's there, but I don't think it warrants the same response. I don't think we are going to war with Russia over this, presuming that it was Russia.

I think it's also really important to put this into proper context. This is espionage. This is one of the oldest professions on the planet. It has existed since time immemorial. It pretty much always will until there's ... I mean, actually, I think even if there's a unified world government, it'll still happen. And this isn't like this happens in a vacuum. I mean, the U.S. has the largest intelligence establishment in the world, the most resources, the most technologically advanced. It's not like we don't do these types of things ourselves, we just don't tend to hear about them for a variety of reasons. I can't speak to specific examples, but I do think it's important to have that in the back of your mind when you're considering, what would be proportional policy responses?

Dave Bittner:

Can you give us some insights? I mean, what goes on behind the scenes with you and your team there at Recorded Future, the Insikt Group? When something like this comes to your attention, what's the internal response like?

Jonathan Condra:

Yeah. So, I mean, I can say from an internal perspective, even independent of our duty to our customers and our clients, we adopted a two-prong approach. One was, of course, vetting whether or not we had ever been affected by this or we ever used SolarWinds. And I'm happy to say that the answer to that is no. That was the initial gut check. Then the second thing was starting to amalgamate what has been released in open sources or things that we're able to pull from data sets that we do have, and authoring reports for our clients in a condensed way, that's consumable, that they can brief up to their executives or their SOC, et cetera, including IOCs, et cetera, that they can use to defend themselves or notify whether or not they're seeing related activity.

That really started with the release of FireEye's Red Team tools on GitHub and the associated detections. Shortly after that, we started digging into some of the first stage callback domains, which were generated by the SUNBURST backdoor that used an algorithm, a DGA, a domain generation algorithm, to encode the domains of victims. And when I say victims here, I mean people, organizations that had installed the malicious package, but not necessarily victims of subsequent data exfiltration. We were using some decoders that had been published by other companies, as well as some internal work that we had done to develop our own, to decode those domains that we were able to see from sources like Pastebin, GitHub, Passive DNS, et cetera, and actually were able to provide a list to our clients of roughly 300 domains that we had observed with relatively high confidence communicating with the first stage callback domain.

That allowed the customers to not only determine whether or not they were initially impacted by this but also whether any of their third-party vendors might have been as well. And that's a small snapshot of the truth. That's a small snapshot of what we're able to see. I don't think we're fully out of the woods on this one yet, unfortunately. It's been really hard over the holidays and over the last few days to keep up with all the breaking news around it. It seems like every day you wake up and there's a new organization affected or there's a new little facet of it that's worthy of noting.

So I think that there's probably more to drop here. I don't think the final shoe has dropped, as it were. As the investigation plays out, I think it's going to be important to keep a cool head in all of this, especially at the national level. The people who are driving the train here, they need to understand that this was not at the level, at least in my opinion, of a 9/11, and that we respond in a proportional way that doesn't, at the end of the day, end up undermining stability internationally or putting more actual lives at risk, if that makes sense.

Dave Bittner:

Our thanks to Recorded Future's Jonathan Condra for joining us. Be sure to check out the blog section of Recorded Future's website for more on this research and many of the other things that the Insikt Group is up to.

Don't forget to sign up for the Recorded Future Cyber Daily email, where every day you'll receive the top results for trending technical indicators that are crossing the web, cyber news, targeted industries, threat actors, exploited vulnerabilities, malware, suspicious IP addresses, and much more. You can find that at recordedfuture.com/intel.

We hope you've enjoyed the show and that you'll subscribe and help spread the word among your colleagues and online. The Recorded Future podcast production team includes Coordinating Producer Caitlin Mattingly. The show is produced by the CyberWire, with Executive Editor Peter Kilpe, and I'm Dave Bittner.

Thanks for listening.