Threat Hunting, Mentoring, and Having a Presence
December 2, 2019 • Monica Todros
Our guest today is O’Shea Bowens. He’s CEO of Null Hat Security and a SOC manager for Toast, a Boston-area firm, where he focuses on threat hunting, incident response, SOC operations, and cloud computing.
O’Shea shares his early beginnings as a teenage hacker learning the ropes, his career path, and why he believes it’s important to be a role model, a mentor, and to have a presence in the security community.
This podcast was produced in partnership with the CyberWire.
For those of you who’d prefer to read, here’s the transcript:
This is Recorded Future, inside threat intelligence for cybersecurity.
Hello everyone, and welcome to episode 136 of the Recorded Future podcast. I’m Dave Bittner from the CyberWire.
Our guest today is O’Shea Bowens. He’s CEO of Null Hat Security and a SOC manager for Toast, a Boston-area firm where he focuses on threat hunting, incident response, SOC operations, and cloud computing.
He shares his early beginnings as a teenage hacker learning the ropes, his career path, and why he believes it’s important to be a role model, a mentor, and to have a presence in the security community. Stay with us.
I essentially was introduced to … In reality, I guess that you could say it was hacking, but no one really called it security. It was like maybe ’97 or ’98. I was around 13. I was taking a computer class in, actually, I was, yeah, I was 12 or 13. I think it was the summer I was turning 13. But either way, I was taking a computer class at the local school. And as I was transitioning into the summer, a friend of mine, actually my best friend, his mom had bought the movie Hackers for him and we went to his house and watched it that night. And then I was obsessed with that movie, ridiculously obsessed. So I literally went home the next night and I was in AOL chat rooms typing in, “What’s a hacker? What is hacking?” And then you get pushed to all these other chat rooms and then you go down this rabbit hole.
But of course at that age you don’t really totally understand the technical concepts. At least I didn’t. I could do a light bit of programming, but the thing that had me … The area of security that grabbed my interest was really networking. I wanted to understand how the actual internet worked. It was mind boggling to me that I could communicate with someone in Mexico or wherever else in the world, all these other weird places inside of these chat rooms. And then you start just asking questions and that’s what led me to where I am today. I still just ask questions.
Well, can you give us some perspective on what was the state of things when that interest in you was sparked? I mean, what types of computers were you using and how did you go about accessing the internet?
It was still a modem dial up. So this was before… I think this was like … I want to say it was a Windows 95. Yeah, I think it was Windows 95. It was still a dial up. The landscape, from my understanding of it, it seemed … I mean, I guess in reality, if I take a reflective approach, it seemed like things were just wide open at that time.
I remember I got introduced to Sub7 around that time, around middle school, I guess. It was Sub7, I think, going into high school. That was the first rat that I’d ever touched. It seemed like you could populate … You could create these weird binaries and just throw them into chat rooms and people would download it and click on it. There was no filtering, there was no limiting or content filtering. It was just like whatever goes. It was a weird … When I look back now, I was like, “You totally shouldn’t have been allowed to do that.” I guess security was an afterthought in a lot of products, which isn’t totally different from today. But today’s a bit more stimulated with repercussions that are financially motivated.
What led you to stay on that path to being on the good side of things rather than venturing off into places where perhaps you shouldn’t have?
I think it was more like … It was probably a fear motivation thing I had … My dad was incarcerated at one point in time. I remember going to visit him when he was incarcerated. It was like, “All right, I don’t want to do that. I never want to be in that position.” But also, you’re a teenager, so your compass is morally shifted towards how you feel that day.
I do remember times I’d wake up and I wasn’t necessarily receiving all of my news from the television. A lot of my news came from other websites that actually published news across the country and then across the world. And there are times I used to be really obsessed with the rainforests and I had this fear when I was younger that if something happens to the rainforest, we’re all going to magically suffocate, which in some ways isn’t too distant from reality. We don’t have the rainforest, we don’t have enough clean oxygen and photosynthesis and all that stuff.
But when I’d read about companies that were destroying pieces of the rainforest, I’d get crazy upset and then there’d be … You’re in these chat rooms online and someone’s like, “Hey, I’m going to try to hack this construction company.” And you’re like, “Yeah, do it, do it.” You’re not thinking clearly because you’re 14 and 15 years old.
Right, yeah. You want to see what happens.
Yeah. And then now you’re like, “Well, that’s probably not the smartest thing in the world.” They’re just guys that need a job. They were just following instructions. They’re just like me and you except for their job is just partially destroying part of the planet. But you get this context, you begin to gain more of a context around, “Okay, is it really you being upset with an organization or their goals? Or is it the people that are carrying out the orders?” And it can’t be the people because those are just people that are working a job. So I think I had that in the back of my head, especially with the dad being in jail thing. I always thought, “Okay, if you get caught, they’re definitely going to throw you in jail.” Because this was the mid ’90s going into 2000. And then people were definitely going to jail at that time and period. So that always stuck with me.
Now in terms of you entering the professional workplace to do this, this becoming something that you could do to earn a living, I mean, how did that transition happen for you? When did you see that being a real opportunity and something you wanted to pursue?
I left school… I went to school originally for fashion design. I thought I was actually done with, in general, technology for some reason, I don’t know why. Not I thought, I definitely believed that I wanted to do something else. And I had this interest in fashion, just being creative and someone wearing your creativity. That’s still a fascination to me that you can come up with this idea of clothing and other people agree with it by totally just wearing it. So it’s like they’re walking around wearing your ideas. That was just really cool to me.
The introduction to my first professional role, I believe I was working at a telecommunications company in Dallas, Texas. That’s where I’m from. I was working in networking, actually. I began to realize that more people were finding jobs in security. So this was maybe like 2007, 2008-ish, I guess. Because Devcon had become to get a bit more popular. I was meeting people at the 2600 meetup that was happening in Dallas. I haven’t lived in Dallas in a while, but I know the security scene has blown up there. But at the time, there wasn’t a lot of security stuff happening on the scene.
There were a small group of individuals like 2600 that were there, but some of those people actually had jobs. I just thought that was cool. I was like, “Oh, are you actually getting paid to perform intrusive analysis?” Or some people were getting paid to … Different areas of security. So as I was working in networking, I lasted maybe a year. I shouldn’t say I lasted as if I couldn’t take it, but I knew I wanted to move into security. I worked in networking for about a year or so, or a year and a half maybe, until I could find the first security job, which was actually at a rival telecommunications company.
Well, and so where are you these days? What is your day-to-day like? What are the things that you’re focusing on?
It’s madness, honestly. I’d say during the day … I’m at Toast up here in Boston. We’re a point of sale terminal company for restaurants and we’re kicking butt. So my primary job there is I’m the first person to take a stab at building out the SOC, so security operations center, at Toast. The challenge, it’s a good challenge, but the challenge here is really thinking about how do you operationalize application security so you’re not digging into the weeds of explicit code reviews and bug fixes and bug bodies. We have one or two individuals that are great at that. But it’s really understanding how you apply those traditional models of detect response and monitor into your application.
So if you can identify that, “Okay, this is definitely an SQL attempt because we were alerted upon it.” How can we create more counter measures, not necessarily on the code side, but also from a layer three and layer seven perspective? So we receive responses in real time to help us foster up our defenses. That’s the challenging area right now. It’s really the operationalizing of application security, of a bit of cloud security and container security, which is a whole new area to me. That’s really applying those other three tiers of defense into your microservices or your micro-containers within AWS, specifically for us with AWS.
What are some of the specific challenges that you face in the hospitality industry?
I mean, there’s one side of it. You have to protect the customer. You want to ensure that there is no interruption of traffic from an encrypted point of sales terminal back into our environment. But I’d say one of the biggest things is at times you don’t control their network. We can provide the tablets and of course you can opt in and we can actually set up a network for you that we can help monitor and we can help alert to weird activity, not as an MSP but really more of we have a certain product for routing and switching and we can place that inside of your environment.
But at times, if you can’t really understand what’s happening in their environment from a networking perspective, that’s a massive challenge because it’s not too far fetched to think that if an attacker wants to learn more about the terminals but they can’t gain direct access to them, well maybe they can hop on the network. And if they can hop on the network, they can begin to monitor or sniff traffic that’s occurring, then maybe they can start to determine, working backwards, well, what’s the best way to actually gain access to the physical device, whether it be the terminal. And then what can I go do from there? So it’s always limiting what they can learn, which is impossible. But when you don’t have an eyeball into what an attacker possibly has an eyeball in, that’s a problem.
Yeah. It’s interesting. I mean, it reminds me of that old saying about how no battle plan survives contact with the enemy. That you can do all the planning you want, but when you place those terminals out there in restaurants, that’s an interesting environment all in its own.
Yeah. It’s a fun challenge. That was one of the big reasons I took on the task. I thought, well, I still think this … But what I envisioned for … When I think about the future of security over the next four to seven years or maybe five to ten years, I think a lot of it’s … There’s a couple different areas. One, the cloud side. And I think everyone’s starting to catch onto that now. But also, with cloud, more individuals are building out these micro services and these containers. So really, how do you detect and defend with that in place? And most likely, that’s critical to your application, which is critical to your business.
And then the other side of it really is how do you begin to create slower times of slower null periods between threat hunting engagements and monitoring and detection across those two mediums I just mentioned beforehand? It’s a weird area because there’s not a lot of companies that are totally focused on this.
So that’s what I envision. I mean, that’s what I … When I state operationalizing application security, that’s what all of that encompasses. It’s super challenging because, like I said, there’s not too many people that have this mapped out. There is no O’Reilly book on how to tackle this.
I know something that’s important to you is being a mentor and helping other people find their place in the industry. Can you share with us, I mean, why is that important for you? Why do you want to spend your time doing that?
Just from the human perspective. If you think about growth in different areas of what have we become from, I don’t know, from inventions and industrial era to all these other different areas that … All these things that we’ve created is typically built upon learning from the next person. So the next revolutionary change is built upon a prior brick and mortar, not necessarily brick and mortar, but a prior brick and mortar type of idea. I think it’s the same thing in security. I think it’s easy to go down this rabbit hole of, well, do I need to know pen testing? Do I need to know reverse engineering? Do I need to be a solid developer, have a deep programming background? Or do I need to understand network security?
There’s so many different areas of security that when you’re starting out, I feel like you can be pulled in just a crazy direction where you’re not as focused, number one. But also, having someone that you can look to or really that can provide guidance is key in helping someone else establish their future and build it upon their foundation. I feel like if you could help someone do that, it’s your duty in a sense. If you look at the whole going back to old school hacker manifesto, it doesn’t matter who they are or what they look like, we’re in this together. So why would you not help out the next person?
I want to get your perspective on threat intelligence some. How do you think threat intelligence should be integrated into an organization? What’s your take on proper use of it?
I’d say the first step is ensure that it’s applicable to your environment. I’ve been in environments before where there was a decent amount of budgeting set aside for threat intel and there’s more than one vendor, actually, inside the environment and they’re just pumping all of this information into the security team, but no one was properly vetting this. So if it’s a bunch of information on Linux exploits or a bunch of information for attack vectors that aren’t necessarily relevant to our line of business, you can toss that to the side. Not toss it to the side, but you can start limiting that type of information out. I think learning how to vet is one of the first steps of … And this goes into building blocks of different areas of your security program also.
So if you can properly vet the intelligence that you’re receiving, and ensuring that it’s relevant to your tech stack, your vector of business, and you actually can notify and work with, whether you have a security engineering team or whether that’s totally internal, you can leverage that type of data to help bolster your security posture. That’s a great use of having intel in place.
I simplify it as can I actually use it to become better at defense? If it doesn’t help me attribute to our defensive posture from either monitoring or creating counter measures or signatures, it’s not as valuable. And you can take that line of thinking and you apply it to what’s hot in security right now with threat hunting.
I travel and I speak at different conferences mostly around incident response and threat hunting. And as I’m learning more on the cloud and container side, I’m sure that’ll come out the wash too. But one of the things I always emphasize after I speak or when I receive questions around building out a threat hunting program, I always say, “Well if you don’t have intel, it’s hard to build out threat hunting because it’s hard to determine what you’re actually searching for. Of course you can leverage attack mitre and look at those techniques, but you need relevant data that’s pertinent to your environment. And that’s where having vetted intelligence is key.
Do you have any interesting stories to share when it comes to threat hunting?
So there’s two different buckets that … Actually, I’d say two different buckets that I place threat hunting or, I’m using air quotes here, the idea of threat hunting into. And that’s successful or unsuccessful. I’ve done a lot more unsuccessful than I have successful. That’s just blunt honesty.
I think a lot of the unsuccessful are emotional reactions. You are in this position of, okay, maybe you have a SIM or you have whatever your IDS is. And this particular alert of interest triggers. It states maybe it looks like it’s ransomware, maybe it looks like some new variant of malware. And everyone freaks out. So you move yourself into this, okay, we need to hunt for this, right? In reality, it’s more analysis. But you’re like, “We need to hunt.” And so you’re trying to carve out these IoCs or TTPs from scratch or really at an ad hoc perspective without any planning. That’s one side of it. So there’s an emotional side that drives this trigger reaction to start just combing through your environment based upon an alert from whatever solution you may have in place. I can say that because I’ve done it. So these are all things that I’ve actually done. And so I was like, “I’m not too proud to admit the mistakes.”
And the other side is totally reliant on that solution for hunting. Ideally, when I think about a solid threat hunt you should have multiple sources of data that aren’t totally related to security only. So when I say security only, it’s not just your firewall logs, your EDR solution, or your AV logs. It’s really understanding, do you have your proxy logs in place? Do you have VPN logs in place? Are you pulling down inadequate amount of CIS logs or do you have the correct level of windows events that you’re ingesting? Those are the areas where you begin to find benign activity. If you think about the whole concept of threat hunting, you’re looking for something that’s gone undetected. If it’s gone undetected, likely it’s not in your AB and SIM solutions logs, you know what I mean? Because I’ve done that. So it’s like you start to learn from your mistakes.
And when I think about something that’s successful, it’s really taking the time to sit down and really plan it out. So you’re planning about and you’re constantly thinking about, one, how can our intel vendor help us? Is there something there that we can leverage that’s been ingested or can we put a request in to whoever that vendor may be for just a bit of guidance?
The other side is actually sitting down and when you formulate this hypothesis, you’re mapping out different data sources that you believe are pertinent to this. So if you have Windows Events Logs, do you have those 542s? Do you have logs that could help you support the notion of lateral movement that’s occurring across the environment? And then, are you classifying that data? How relevant is that data to your particular hunt? So you’re taking this 4,000 foot view or 3,000 foot view of what data is available in the environment, how you can leverage it, how you can leverage your intel, and then you plan out the hunt.
That’s not something that happens in … In my experience, I can only speak for myself. But that’s not something that happens in a two hour time span. That’s usually a couple of hours or a day or two of just mapping that out, planning it, creating stories for this. I’ve started to learn over the last year, when I engage in threat hunting inside of an environment, I want to have some type of epic or story behind it so I can detail the steps I took, the amount of time it took me to reach those goals, and build upon that with each engagement.
It’s interesting to me how it sounds like you have gained as much wisdom and good experience from the failures as the successes.
Yeah, that’s a fair assessment. Honestly, you fail, not in life in general, I don’t know, maybe it is. But at least in certain areas of security and specifically threat hunting, I felt like I … Most people, me included, but I think a lot of people fail more than they succeed. Well, whether they’re willing to admit that is another story. But if you have 80% success rate with your threat hunting, something tells me that’s not actually threat hunting.
I know something that’s important to you is supporting diversity in cybersecurity and efforts there. What sort of things do you do for that aspect of the industry?
One of the things I identified first is really having a presence of … And I say a presence, but it’s really a presence amongst the security community. I think we’ve all been on different IRC channels for Porsches. I’m 36, so there’s a couple different IRC channels I’ve been on for over a decade. And half these people I’ve never met in my life, probably more than half actually. But one of the things I noticed when I would attend conferences and different conferences or meetups was, even going back to 2600 when I was a teenager, it’s like there weren’t a lot of people that looked like me. I didn’t place much thought upon that as a teenager. It wasn’t until I started to move into the professional realm where it became a bit more evident.
There weren’t a lot of African Americans in cybersecurity. And as I started to begin to go to more conferences, it became way more evident. One of the things I thought of was just from a representation side, maybe that’s a place to start. Just start speaking and putting yourself out there and maybe when more people see that, that can help encourage others to maybe submit to CFPs and attempt to share their knowledge.
But one of the things I stated earlier was I started a podcast with a buddy of mine, Doug Bryant Jr who was over in Jacksonville, Florida. We wanted to start a security podcast. I just thought it was this really awesome idea and powerful idea to have a podcast where there’s two African Americans in cybersecurity dissecting these deep issues.
That highlights another side of the representation aspect of knowing that there’s other people like you that exist and there are different mediums where you can connect and share ideas. And that was actually another reason I wanted to mentor, too. I’ve never necessarily had a mentor mentor when I was fairly younger. So helping someone else navigate the field and helping encourage other people of color in cyber, I feel like that’s just an important mission. I don’t know what the success meter is for this, but I feel like the more people I can reach, maybe the better I’m doing or the more of an impact I can actually have.
When you go to conferences, when you speak at conferences, is your sense that over time things are getting better?
Most definitely. Yeah. I would say even between the last two years. I’ve seen more minorities, more people of color, specifically more women that are speaking. I just think that’s awesome because, again, going back to the hacker manifesto, it’s all inclusive. It doesn’t matter what you look like. So it’s really okay, how do we reflect that in reality? And that’s a challenge. I don’t have the answer for it. I just feel like the more I’m out there, hopefully the more I can encourage other individuals to step up and step out and let their voice be heard also.
Our thanks to O’Shea Bowens for joining us.
Don’t forget to sign up for the Recorded Future Cyber Daily email, where every day you’ll receive the top results for trending technical indicators that are crossing the web, cyber news, targeted industries, threat actors, exploited vulnerabilities, malware, suspicious IP addresses, and much more. You can find that at recordedfuture.com/intel.
We hope you’ve enjoyed the show and that you’ll subscribe and help spread the word among your colleagues and online. The Recorded Future podcast production team includes Coordinating Producer Monica Todros, Executive Producer Greg Barrette. The show is produced by the CyberWire, with Editor John Petrik, Executive Producer Peter Kilpe, and I’m Dave Bittner.
Thanks for listening.