Securing Your Firmware
These days, most of us have a pretty good handle on protecting the software our computers run from viruses and other types of malware. We’re careful about downloading and installing software from unknown, insecure sources, and run antivirus applications to help keep everything safe. But what about the system-level code that runs deep within the devices we rely on every day? What about the firmware?
Our guest today is Terry Dunlap. He’s CEO and co-founder of ReFirm Labs, a tech startup that’s focused on firmware — analyzing the code and helping manufacturers, organizations, and governments ensure their devices haven’t been compromised. He’s got a colorful history that includes teenage hacking, time at the NSA, and the founding of several companies.
This podcast was produced in partnership with the CyberWire and Pratt Street Media, LLC.
For those of you who’d prefer to read, here’s the transcript:
Dave Bittner:
Hello everyone, I’m Dave Bittner from the CyberWire. Thanks for joining us for episode 66 of the Recorded Future podcast. These days, most of us have a pretty good handle on protecting the software our computers run from viruses and other types of malware. We’re careful about downloading and installing software from unknown, insecure sources, and we run antivirus applications to help keep everything safe. But what about the system-level code that runs deep within the devices we rely on every day? What about the firmware?
Our guest today is Terry Dunlap. He’s CEO and co-founder of ReFirm Labs, a tech startup that’s focused on firmware — analyzing the code and helping manufacturers, organizations, and governments ensure their devices haven’t been compromised. He’s got a colorful history that includes some teenage hacking, time at the NSA, and the founding of several companies. Stay with us.
Terry Dunlap:
In 1985, I was a junior in high school at the time. For a couple of years, I had a Commodore 64 with a 300 baud modem and a couple of floppy drives, and we were able to … I had a little team. We were actually hacking, or war dialing, the city of Sandusky, Ohio, looking for computers that we could interact with. We didn’t know what we would find. We didn’t know what they were, but we found it fascinating to play with these systems. Long story short, we did some nefarious things that eventually got us caught by the local authorities. They couldn’t arrest us on any hacking charges because the Computer Fraud and Abuse Act wasn’t in effect until the following year, in 1986.
Dave Bittner:
Timing is everything.
Terry Dunlap:
Exactly. So the only thing that they actually nailed us on, as juveniles, was credit card fraud. One of the things we were doing was just collecting credit card numbers out of dumpsters at the mall in the middle of the night and collecting all the carbons. And then, manipulating the phone system using some black-boxing and red-boxing techniques and tones to actually connect with The WELL — the bulletin board system out on the West Coast. We would upload our credit card numbers there, and we would download some of the credit card numbers we found out there and use those to order crazy stuff, like more computer components, firecrackers, and all kinds of stuff that teenagers would order.
But one guy ended up having packages delivered to an abandoned house next to his, and the people that own the property got suspicious and notified local authorities and UPS. They staked out the place and saw him actually pick up the package and go into his home. I guess his parents weren’t at home because he obviously let them in. Then, they discovered he had a handwritten diary with everybody’s name in it and what had been going on for the past year and a half. So I got pulled out of class in junior year of high school down to the principal’s office. Saw my parents there with the police.
Then, we end up going to the juvenile detention center for about five days before we’re hauled into court, and basically, given a suspended sentence, probation, and then told, “You are not to touch computers for the next three years. Pay some court fines. Once you become an adult, your record will be expunged.”
Dave Bittner:
So this moment happens. Did you straighten up and fly right from there on? Terry Dunlap:
For the first few months, yeah. But after, I didn’t do anything. Put it this way — back in those days, there was a television program called “Scared Straight.”
Dave Bittner:
Oh yeah, I remember it. Sure.
Terry Dunlap:
Let’s just say I was scared straight for a good while until I got into college.
Dave Bittner:
Well, I’m just trying to imagine, when computers are so much of your life … I remember that time. I, similarly, took part in a lot of those types of things back in the 8-bit computer days. The notion of someone telling me to not touch a computer for years, I mean … That would have been like telling me not to breathe.
Terry Dunlap:
Yeah, no. I mean, it’s impossible, especially when you end up going to college and you’re supposed to be doing papers.
Dave Bittner:
Right.
Terry Dunlap:
You’re not going to do it on a typewriter. I mean, you could. Some people still had typewriters in the 1986 to 1987 timeframe, but that wasn’t the most efficient way to do it. I ended up getting an Apple IIe anyway, but I wasn’t doing anything nefarious at that point. I pretty much learned my lesson not to muck around and get in trouble with the law.
Dave Bittner:
So you head off to college. What are you studying there?
Terry Dunlap:
Well, during high school, I was very good at science. One of the things that our program had was science fair competitions at the local and state level. We were always in the top five students in the science fair awards. And so, I thought I was going to be a chemical engineer until I went to college and realized that, you know, I don’t see myself walking around a BP plant with a hardhat and a clipboard building chemical systems. So, I looked for something more interesting, and honestly, I ended up majoring in poli sci and econ.
I never took any … Well, I did take one computing class my freshman year and it was assembly language. I dropped that class after probably about two weeks. I was like, “No way. This is not what I expected.” So I ended up bailing and focused on, primarily, poli sci and econ.
Dave Bittner:
So, you got out of school, and where do you head next?
Terry Dunlap:
I go back home to Northern Ohio and work with a fly-by-night brokerage firm because I really liked investing in economics. So, I worked with a fly-by-night brokerage firm. I left there and ended up working with Fidelity Investments down in Cincinnati, Ohio. Then, I guess I impressed enough people there. They suggested I go work with this new internal group that Fidelity was creating in Boston — where their corporate headquarters is — called the Premium Services Group, which still exists today, and it caters to high-net-worth clients. So I was there for a number of years, handling people’s trading accounts, trading stocks, options, bonds, things of that nature.
My goal was to actually become a chartered financial analyst and get the designation. I took two of the three tests. I wanted to become an analyst and work with the fund managers. But there was an internal hiring freeze at Fidelity during that time. I think it was probably the mid to early ’90s. And so, I didn’t want to stick around and wait for the hiring freeze to unfreeze. So I contacted some college buddies and said, “Hey, let’s start a publishing company,” because we had this idea in college to create a publishing company, because I actually had failed my freshman year. Among a significant number of the freshman class at Case Western Reserve University, we had failed first-year calculus.
So, these guys, two buddies of mine from Case Western, basically said, “The problem you’re having is that you’re being taught by these foreign TAs. Let us teach you and show you how easy it can be. So, they sat down with me and gave me a few lessons. I took some tests, and it was like, “Holy shit. I actually understand this stuff. This is cool. So, wouldn’t it be cool if you guys could … We could write a book, kind of like a … ” At the time — and I don’t know if it’s still a thing these days — but Schaum’s Outlines. Do you remember that?
Dave Bittner:
No, I’m not familiar with that.
Terry Dunlap:
Schaum’s Outlines was a series of books that taught you rapidly how to do calculus, chemistry, geometry — all this kind of stuff. So, we thought, you know, “Let’s compete with Schaum’s Outlines.” So, they would develop the chapters, I would read the chapters, and then I would have to actually execute and do the exercises at the end of the chapter. If I understood it, we basically said, “That chapter is done. Let’s move on to the next one.” If not, we would go back and revise it until we actually understood it. So, we actually created Prometheus Enterprises — our first business together, three guys.
I took my 401(k) savings and put it into the company. We actually got on a number of nationally syndicated shows, like Science Friday on NPR.
Dave Bittner:
Sure.
Terry Dunlap:
Once we got on there and talked about the state of calculus and education and teaching methodologies in our book, sales started taking off. We did that for a couple of years, but I had a disagreement with the partners on where the direction of the company was going to go. So, I asked them to cash me out. I left. I went back to Cincinnati and started looking for a job there, and ended up with Deloitte & Touche, working with their high-net-worth clients and kind of doing the investment thing again. I got really tight with the sysadmin guy there. He recognized my computer acumen. So anytime he took off on vacation, I’d step in to take over his sysadmin duties.
And so, one time when we were on the Ohio River, on his jet skis, he said, “Hey, you still into that hacking thing?” I said, “Well, I mean, I have stuff set up at home, but I don’t break into people’s systems.” He said, “Did you know that Deloitte actually has a team where they get paid to go in and break into clients’ systems?” This was the first time I have heard of people getting paid to do what was called “penetration testing.”
Dave Bittner:
Right.
Terry Dunlap:
So, he made some introductions, and then I actually went and interviewed, got on the team, and spent some time there at Deloitte & Touche, getting back into the hacking game and doing penetration testing against their clients, which were Fortune 500 companies.
Dave Bittner:
Now, how much did you have to get up to speed there? I mean, the skills that you had as a teenager — had that moved along?
Terry Dunlap:
Yeah, I was keeping up with stuff on my own outside.
Dave Bittner:
I see. Got you.
Terry Dunlap:
So, in my basement, I had Cisco routers, Linux systems, and I was playing with all different Linux distros.
Dave Bittner:
Got you.
Terry Dunlap:
So, yeah. I was pretty much keeping up with that stuff. One day, I was actually at a SANS conference in Baltimore and I was in line at one of the Starbucks kiosks in one of the concourses. We all have our first name and who we work for on our badge, and I noticed this woman behind me. I’d never seen her before. I turned to her and I said, “Man, it would be awesome to work at your place.” She worked for the DoD. I said, “You guys are probably under constant attack. That would be such an interesting environment to learn in and apply these skills.” And so, we started talking. I sent her my resume — she asked.
Then, a few months later, I get a phone call from a recruiter who’s with the National Security Agency. I had no idea that’s who she worked for at the time. But we did a phone interview, and I flew out there. I did some face-to-face interviews and ended up starting at the NSA in 2002.
Dave Bittner:
I have to ask — was there a conversation about your teenage exploits?
Terry Dunlap:
Yes. In fact, I brought it up first.
Dave Bittner:
Okay.
Terry Dunlap:
Because at the very end … When we were in person, they had flown me out to Fort Meade, and I did the rounds with all the different groups of people. At that time — it may be different — but at that time, the way that the agency worked was that they would fly a candidate out, interview with a number of different branches and organizations, and then by the time you got back to the recruiting center to speak with the recruiter, those people you interviewed with that day should have sent back an email to that recruiter to say whether or not they were interested in hiring you.
There were a handful of people that were interested in extending an offer. So he gave me the entire package for my background check and everything and said, “Get this back to us as soon as possible,” because the way that they operate is that for every position, they offer it to up to five different people because not everyone’s going to pass the background check.
Dave Bittner:
Right.
Terry Dunlap:
So when I heard that, that’s when I brought up my teenage exploits.
Dave Bittner:
Funny story.
Terry Dunlap:
Yeah, so I said, “Hey, look, this is what happened. I’m being transparent. I’m being honest. This is what happened. They say the record’s expunged, but I’m sure it just means that it’s moved to a different filing cabinet and only people like you have access to. So this is what happened. These were the outcomes.” He asked me, like most people do, “Are you still doing it today?” I was like, “No. I have systems set up at home that I can hack on. I’m not getting on the internet and breaking into places.” He said, “Then it shouldn’t be a problem.”
Now, it did take a year to vet me during that background check process. But in the end, I ended up getting my top-secret clearance and working for the agency.
Dave Bittner:
What types of work were you doing there, and then, how did that lead you to where you are today, at ReFirm?
Terry Dunlap:
So yeah, it was a great time there. I mean, it was probably the only job that I actually loved. I actually had a hard time deciding to leave because it was so great. Now, most of the work I did there was offensive cyber. So, some of the work I was tasked with was looking at certain types of embedded devices and looking for vulnerabilities that could be weaponized. So, we would weaponize those vulnerabilities for the intelligence and the military community. One day, a sister intel agency stopped by to see what we were working on. We demonstrated our capabilities and found out that they would like to use that capability in some of their missions. Management basically said, “No,” and I was stunned because, I mean, we’re in the middle of the global war on terror at this time, and the fact that there’s this capability that another intelligence agency could use, and they flat out refused to share it … I was like, “You’ve got to be kidding me.”
Dave Bittner:
What was your take on the justification there?
Terry Dunlap:
This is still the stigma that is out there — that these other agencies have a tendency not to take care of these tools, tactics, and techniques and have a propensity to burn them.
Dave Bittner:
I see.
Terry Dunlap:
Meaning, they usually end up getting caught. Sometimes they might end up in court, where they have to reveal how these platforms work. So, the NSA was, and probably still is, very protective with these capabilities and making sure not to put them in other agencies’ hands that could expose how these capabilities work. So, after seeing that there was a potential market opportunity for other agencies, I decided to leave and create my first company, which was a government contracting company called Tactical Network Solutions. That was started in 2007, literally in my basement, and I spent about nine months there developing a universal, plug-and-play remote exploit for Linksys routers that I knew that certain elements of the military needed.
I still maintain my contacts with some of the special mission units that were forward deployed overseas. And so, I approached the unit commander with the capability I had developed. I said, “Is this something that you guys would be interested in?” He said, “Absolutely,” and then ended up buying multiple licenses to that capability, and that was my foray into government contracting. So now, I had to create all the stuff that goes with creating a government contracting company — applying for D-U-N-S numbers and CAGE codes and getting a top-secret facility clearance and all that kind of stuff.
So, I built Tactical Network Solutions. In fact, the company’s still around. The focus has always been primarily offensive cyber capabilities — going after IoT and embedded, connected devices, and offering training classes because people don’t have this skill set and no schools are teaching this skill set. Now, going back a little bit — part of our work at Tactical Network Solutions, in the 2010 to 2011 timeframe, was requests from our customers to rapidly identify zero-day vulnerabilities that could be weaponized.
We did a lot of this analysis and discovery process by hand, and then realized that in order to keep up with demand, we needed to create an automated system to help us quickly and rapidly identify where our time would see a significant return on investment instead of just trying to hunt for the needle in the haystack. So, we developed this internal platform, and it was very good. We could take any compiled firmware image, put it in the platform, and in 30 minutes, it could identify potential zero days that we can now spend our R&D time on, seeing if we can weaponize those.
So, it was only a year ago, in 2017 — June to July timeframe last year — that we were presenting this platform at a Maryland TechBreakfast here in Columbia, and some investors happened to see what we had developed and approached us after the demonstration. They said, “Have you ever thought about using this platform as a defensive tool instead of an offensive tool?” And of course, coming from where we came from, we didn’t, and we didn’t think about protecting anything. It’s more fun to hack stuff. Come on, who are we kidding here?
Dave Bittner:
Right, right.
Terry Dunlap:
But they presented the market opportunity of, you know, here are the manufacturers and the enterprises that could benefit from this, and how much something like this could be sold for in the marketplace. So, after a few weeks and months of discussing this, we finally agreed that, okay, we will spin this platform off into a brand new company. It will be completely separate from Tactical Network Solutions. The platform, the intellectual property, everything, will come over to this new company called ReFirm Labs, which was launched last July 7.
We took a seed round of 1.5 million, and over the past year now, we’ve been creating this platform that was once used to just scratch our own itch to become an enterprise-grade, SaaS, cloud-based platform that’s completely scalable for enterprise and developers to use during the development process of IoT-embedded firmware.
Dave Bittner:
Now, let’s back up some, just with some definitions. What is firmware?
Terry Dunlap:
Firmware. I know we’re all familiar with hardware, we’re all familiar with software. But firmware is probably a relatively new term for most people. Firmware is about as close as you can get in terms of accessing the hardware on a particular platform. So, for example, the following have firmware: your phone has firmware, your car has firmware, the Polycom phone system in your boardroom has firmware, and your TV has firmware. Basically, it’s the operating system of these connected devices that we don’t interact with via a monitor or a keyboard and a mouse.
These are the brains of these devices. Your wireless router at home, your security cameras — they all have firmware. That’s what makes them run and do what it is that they do. So, firmware has become extremely sophisticated. I mean, back in the days of the Commodore 64, the firmware that’s in there usually just would help either boot the system or help operate the 300 baud modem. But now, we’re at the point where firmware is in your car and basically steers the car for you with lane assist or adaptive cruise control. This is all firmware that’s in the car that can be easily manipulated if an attacker is persistent enough.
Dave Bittner:
So, help me understand the difference between firmware and software.
Terry Dunlap:
Firmware is designed at a significantly lower level to run the very specific hardware components. Software is higher up the stack, if you will, where you can actually interact with programs and install stuff and uninstall stuff, and you have more control over software versus firmware. Firmware is developed by the manufacturer. It’s flashed onto a device. Software, you can install and uninstall. At will, you have complete control over it. So, the firmware is usually designed to run a very specific piece or collection of hardware components on a printed circuit board or an electronic control unit or something very low level that you and I normally would not interact with on a regular basis.
Dave Bittner:
Now, in the old days, that firmware would have been baked into, you know, like a ROM chip or something like that.
Terry Dunlap:
Yeah, it needs to be burned in.
Dave Bittner:
So today, firmware can be updated and can be changed over time?
Terry Dunlap:
Yep. Nowadays, firmware is to the point where, if a manufacturer realizes that there’s a security vulnerability in your wireless router, they can create an updated firmware image, which then, the responsibility is pushed to you to actually be aware that the vulnerability exists. Go to the manufacturer’s website, and under their support page, there should be a firmware image that you can click and download. Then, you’d have to log into your wireless router to actually apply that upgrade and then change that. It’s still flashed into the system, into the hardware, but it’s not burned in. Once it’s burned in, it’s very difficult, if not impossible, to update.
But firmware today is very updatable. Your phone’s a perfect example. Especially if you’re an iPhone user, you constantly see iOS updates coming out. When you take your car into the dealership, chances are, they’re checking to see if there are any firmware updates to your infotainment system and applying those all without you having to do anything.
Dave Bittner:
So, what are the specific vulnerabilities that come with firmware versus the higher-level things that are running on a system?
Terry Dunlap:
What we found over the many years — and it continues today — that we’re discovering in firmware is … Do you remember the days, back in the ’90s, when Windows 95 first hit the scene, and then all the problems and the attacks that Windows users were hit with … not ransomware, but strictly hack attacks and viruses?
Dave Bittner:
Mm-hmm (affirmative).
Terry Dunlap:
I mean, all that stuff, if you recall those days … We don’t see that happening at the desktop and the server level and on laptops anymore. That stuff has been pretty much eradicated. But we’re seeing those same exact issues resurrect themselves in IoT and connected devices today, and basically, it comes down to one fundamental flaw in secure coding practices. Microsoft and the open source community have been very, very good over time at implementing more and more secure coding practices to prevent the types of attacks we saw back in the ’90s. However, for whatever reason, when we look at firmware, we’re finding the same sloppy coding practices that led to the issues in the ‘90s leading to the issues today in IoT and other embedded devices.
So, without getting too technical, we see a lot of buffer overflows. Basically, a buffer overflow is when you’re trying to copy or move too much data into too small of a space. When that happens, that causes adverse effects to the system. For a sophisticated person, when they find something like that — a buffer overflow — they can find ways to then take control of that system and they basically become the root or the administrator of that particular device. That’s how a lot of these botnets end up being created, is that they find these low-hanging-fruit problems, take advantage of them, and then put them all under the control of one particular master, and then create botnets that can take out the likes of Netflix and eBay and Yahoo and others.
Now, there are a couple of other things that we’re finding in firmware images today that are problems — hard-coded usernames and passwords. Now, in most cases, these are engineering debugging accounts during the testing phase that, for whatever reason, accidentally are left in during production. But for a sophisticated attacker, it doesn’t take them very long to identify hard-coded usernames and passwords, and again, being able to use these to create some type of botnet under their control. The third thing that we see, which we should never see, is private signing keys being left behind in firmware images. Let me explain what that is.
Dave Bittner:
Yeah.
Terry Dunlap:
When a manufacturer or a developer authors a piece of software or firmware, in order for the system where this is going to be run to trust that, they will check to verify that the manufacturer signed it with a key that they recognize — their signing key. So, if I see that Dave’s software development shop was issued a signing key, and I check that key against a database, then it’s like, “Okay, I’m going to let that software get installed on my hardware or on my desktop system or on my server.” However, if you leave that signing key in the firmware, it can be extracted, which can easily be done and we’ve done this before. We can then modify that firmware image, re-sign it as you, and then flash it onto the device as if it trusted the changes.
Dave Bittner:
Right. It’ll seem legitimate.
Terry Dunlap:
Yeah, so, I’ll give you a real example. We were working with an automotive manufacturer. They were concerned about the security of one of their tier-one suppliers, so they provided us the piece of hardware and the firmware image, and we did find the tier-one supplier’s signing key in there. However, the people we were working with at the automotive manufacturer didn’t understand the risk that that presented. So, we had to take it a step further and actually modify the firmware, and then sign it as that tier-one supplier and flash it back onto the vehicle.
Now, once it was on the vehicle, we demonstrated that when the person actually turns on the turn signal to turn right, the left blinker illuminates. When you turn it left, the right blinker illuminates. When you turn on the AC, the heat is activated. When you turn on the heat, the AC is activated. Now, we could have been significantly more malicious, but I think that proved the point.
Dave Bittner:
It’s an unambiguous demonstration.
Terry Dunlap:
Exactly. So, they clearly saw what could possibly happen with somebody’s signing key accidentally left into the firmware image, and that, shockingly, is a big problem as well. Dave Bittner:
Now, we have these stories coming through about these international intrigue stories, with folks out of China being able to gather data, and so forth. You all have some experience with this. You shared a story with me about some work that you did. I think it was with a cable box, a while back?
Terry Dunlap:
Yes. Actually, it was a British Telecom router that they … This was years ago.
Dave Bittner:
Okay.
Terry Dunlap:
This was probably 2009, maybe 2010. I’m not sure of the exact timeframe, but it was a while ago. What had happened was … There’s actually two stories. The Huawei story was, British Telecom outsourced development of the BT Home Hub to, apparently, Huawei. There was some checks and balances to make sure the firmware that they were receiving for this platform was secure. There was one of those hard-coded usernames and accounts that was discovered in the firmware. Assuming it was an engineering mistake, the vendor was notified. “Hey, this is what we found during our security screening process. Can you remove it?”
And so, after a number of days, or a week or so, they get the new firmware version for this device that is going to be deployed into millions of BT customers’ homes. They did remove it in the firmware. However, it appeared in a different location in the firmware, which led to the conclusion that this was probably done intentionally. I don’t know the exact outcome of what British Telecom said to Huawei after that fact, but that was the first identifier for us as a company that, “Okay, this looks like this could be malicious.”
Now, fast forward to just November of last year when we did some work for a Fortune 500 company who had some concerns about some Chinese-made security cameras that were deployed on some of their properties. The manufacturer was Dahua. We looked at the firmware. We actually identified a hard-coded username and password in there that by all accounts, based on our decades of experience looking at firmware, was not accidental. This was intentional. So, we notified the Fortune 500 customer to be on the lookout for suspicious activity on these cameras.
Within about 36 hours, the chief of digital security contacted me via text with a firewall log and said, “You guys were spot on. Look at what we discovered.” When you look at the log, you can see that there was actual traffic from the client’s network being sent in and out through these cameras to Chinese IP addresses. They were able to block the traffic, and eventually, over time, they replaced all of these security cameras on their network. So, we actually took those results without naming the Fortune 500 company and published them in a vulnerability research report that we issued back in November.
Obviously, Washington Post picked up on it, Fortune picked up on it, and of course, Dahua picked up on it. They emailed us saying, “Hey, you looked at the wrong firmware. We knew that was a mistake. Here’s a link to the latest firmware.” So I said, “Okay. We’ll take a look at it.” It was still there. I pushed back to this Dahua representative. I said, “Here is all the evidence of the backdoor still existing in this, quote-unquote, ‘new-and-improved firmware image.'” The response from Dahua was, “Well our engineers told us that the backdoor has been removed.”
My final reply to them was, “You have all the proof that the backdoor is actually still there. Are you sure that your engineers aren’t working for your Chinese government?” I never heard a reply back after that.
Dave Bittner:
Right. Wow. It’s an interesting story and certainly sheds light on some of the things we’re seeing in the news regularly now. One of the focuses of this show is threat intelligence, and I want to get your take on that because I think you all come at this from an interesting perspective. As people are looking to protect themselves and get information to do that, what is your take on threat intelligence and how folks can best use it to help protect their organizations?
Terry Dunlap:
I think there are a lot of threat feeds that are out there. I’m not sure, since I don’t manage the security of a Fortune 500 enterprise, how these companies are ingesting all of these different threat feeds. I’m sure there are platforms out there that do this and feed them into a SOC to give them a single pane of glass to let them know where they reside. I think they serve their purpose. But my feeling is that a lot of the threat feeds and threat intel that’s out there focuses on the “here and now,” and not what has already been deployed and could be happening.
So, based on the threat feeds that I’ve seen, I think it’s usually some research people have done and identified a bad actor either pre-positioning or taking advantage of a vulnerability. They’re seeing telltale signs of activity, and then they issue threat reports based on what they’re seeing in the wild. Now, what we wanted to be able to do from a quote-unquote “threat feed perspective,” one of the things that we used to do and are still doing — it started at Tactical Network Solutions — is to meet that zero-day weaponization demand that the government had. We scraped the internet and continued to scrape the internet of all public-facing firmware images on support pages from the likes of Sony, LG, Samsung, Linksys, and D-Link.
Anybody who has firmware available for public download so customers can upgrade their stuff … We’ve been massing all that firmware, and we continue to do so under ReFirm Labs. What we would like to be able to do — and we haven’t completely fleshed this out yet, but as part of a threat feed option — is to provide, say, procurement offices in Fortune 500 companies that might be considering, oh, I don’t know, the latest HP multifunction printer. We’d like for them to be able to go to an interface and punch in the make and model of what internet-connected device they’re looking to buy and get consumer reports like analysis with a star rating to see how secure that firmware is for that particular device, and how that manufacturer has been treating firmware security issues over time. Dave Bittner:
Rating the manufacturer, that they are or may not be someone who takes this with … An appropriate level of seriousness?
Terry Dunlap:
Exactly. I’ve been in talks with a lot of people at companies like D-Link, TRENDnet, Netgear, and Belkin. Adding any more costs, especially security cost, to the development lifecycle of these types of platforms is really a no-go for a lot of them because the margins are so thin, and the product lifecycles are so short that one company told me that if I add another dollar of cost, that equates to a five dollar increase at the retail level. However, when you look at other companies like Honeywell — which we’ve been talking to — who’s a big player in critical infrastructure, they’re all about being as proactive as possible and being willing to adopt and spend money on security solutions now to prevent some bad stuff happening down the road.
So, it depends on where in that chain of products you’re targeting these solutions to. There are some manufacturers who could care less because their product lifecycles are so short and the margins are so thin. But then you have people out there like your HPs of the world, all the way up to industrial control systems like Honeywell who are very serious about security.
Dave Bittner:
I want to wrap up with you, but I want to finish by getting your take on, you know, for that person who is out there and is charged with securing their company, how much should firmware be on their mind? I realize that this is like asking a barber if I need a haircut. But how much should firmware be on their minds? And what are the ways … We talk about basic hygiene. What are the basic hygiene ways that folks can keep up with making sure that they are where they should be when it comes to firmware?
Terry Dunlap:
Well, one thing I would recommend is to start at least considering it and taking it seriously. See if you even have an inventory of all of your IoT connected devices, and I mean, look at that Polycom that’s sitting in the boardroom. When was the last time that you’ve gone out to Polycom’s website to see if there’s a firmware update for that Polycom system sitting in the boardroom? Or the TV? Usually, the TVs will tell you if there’s an update available. Security cameras are a big thing. Now, a lot of Fortune 500 companies probably do deal directly with the vendor when they buy security cameras.
They probably deal with some value-added resellers, some VARs. But I would be in contact with that rep to find out what the most recent security issues with these cameras have been. Have you or your team come in and actually updated or applied any of the patches? Because a lot of companies actually, unfortunately, have Dahua and Hikvision security cameras deployed throughout their network. It wasn’t too long ago that Wall Street Journal reported that Hikvision is 42-percent owned by the Chinese government.
So, I would at least put it on my radar. The easiest things to do … Try to get an inventory of these devices, and if they were handled and installed by a VAR, find out when the last security updates were issued. If they tell you that there have been no security update issues since they’ve been deployed, I’d be very, very curious as to why.
Dave Bittner:
Should you routinely be looking at the traffic to and from these devices as you’re checking in on them? Or is that … Where did we stand with that?
Terry Dunlap:
You know, if you know the type of traffic that should be going back and forth to these devices … For example, a Polycom phone.
Dave Bittner:
Right.
Terry Dunlap:
I mean, you expect some SIP traffic and some RTP, real-time protocol, traffic. If you see HTTP traffic coming out of a Polycom phone going to an IP address outside your network that you don’t recognize, I would be very suspicious that something’s wrong with that phone. And to be quite honest with you, we were engaged in a project where we were doing some pen testing, which is something we don’t normally do. But we decided to do it in this one case. We were able to actually hot mic a Polycom system in a boardroom, sitting here in Columbia, Maryland, and it was up in Michigan.
Now, it required us to actually — believe it or not — downgrade the firmware on that Polycom phone. So, we had to go to Polycom’s website, get an older version of the firmware, and then flash it onto the existing device. But then we were basically able to do … We had complete control of that particular platform.
Dave Bittner:
Wow.
Terry Dunlap:
So, I would be looking … Yeah, you could look at traffic. You have to be certain of the type of traffic expectations. Get a baseline for that, and then look for any anomalous behavior outside of that.
Dave Bittner:
Our thanks to Terry Dunlap from ReFirm Labs for joining us.
Don’t forget to sign up for the Recorded Future Cyber Daily email, where every day you’ll receive the top results for trending technical indicators that are crossing the web, cyber news, targeted industries, threat actors, exploited vulnerabilities, malware, suspicious IP addresses, and much more. You can find that at recordedfuture.com/intel.
We hope you’ve enjoyed the show and that you’ll subscribe and help spread the word among your colleagues and online. The Recorded Future podcast team includes Coordinating Producer Amanda McKeon, Executive Producer Greg Barrette. The show is produced by Pratt Street Media, with Editor John Petrik, Executive Producer Peter Kilpe, and I’m Dave Bittner. Thanks for listening.
Related