Intelligence for the OSINT Curious

July 8, 2019 • Zane Pokorny

Our guest today is Micah Hoffman. He’s principal consultant at Spotlight Infosec, and one of the founders of OSINTCurio.us, an online destination for enthusiasts and students of open source intelligence gathering and analysis techniques.

He shares his professional journey from psychology to information security, his insights on the growing availability and importance of open source intelligence, his emphasis on ethics, and how organizations can best integrate open source tools into their security strategies.

This podcast was produced in partnership with the CyberWire.

For those of you who’d prefer to read, here’s the transcript:

This is Recorded Future, inside threat intelligence for cybersecurity.

Dave Bittner:

Hello everyone, and welcome to episode 115 of the Recorded Future podcast. I’m Dave Bittner from the CyberWire.

Our guest today is Micah Hoffman. He’s principal consultant at Spotlight Infosec, and one of the founders of the OSINTCurio.us website, an online destination for enthusiasts and students of open source intelligence gathering and analysis techniques.

He shares his professional journey from psychology to information security, his insights on the growing availability and importance of open source intelligence, his emphasis on ethics and how organizations can best integrate open source tools into their security strategies. Stay with us.

Micah Hoffman:

I’ve had an interesting one over the years, my undergraduate degree is in psychology and after doing some things in the field I figured out that I really needed something more concrete and switched to computers and fixing them, breaking them and doing penetration testing and a lot of cyber-y things. And then throughout the years as I did more penetration tests and risk assessments, I always found myself gravitating more towards open source intelligence.

Although back then I had no idea it was even called OSINT, it was just doing reconnaissance in support of our gigs. About five years ago I started up with the SANS Institute as an instructor, and over the years I decided that it would be really neat to migrate fully into OSINT world and create a course on open source intelligence for the SANS Institute. And that’s where I’m at right now.

Dave Bittner:

Well let’s back up a little bit in and share some definitions. I mean, how do you define open source intelligence?

Micah Hoffman:

Well, the classic instance is anything that is retrievable by the public, it’s in the public domain. So it could be television broadcasts, radio broadcasts, things that you go down to the courthouse and have to apply for in person. But for most of us in the cyber OSINT world now we think about things that are online, things that are accessible from our computer systems, whether it’s social media or domain history or things on the dark web.

Dave Bittner:
And take us through a little bit, what are some of the practical applications of OSINT these days?

Micah Hoffman:

Well, it depends. It depends on what your goals are. What I find over the years is that people use OSINT and they don’t even know it’s OSINT, kind of like my beginnings. I’ve seen businesses use OSINT for understanding what a company might be like if they were to buy it or merge with it. I’ve seen people in law enforcement use OSINT to find out what the upcoming threats might be for some type of activity, or gathering, reconstructing crimes, gathering photos and videos of a natural disaster or some type of a terrorist event. And then criminals use OSINT as well for finding targets, attacking them, for spewing propaganda, for phishing attacks. A lot of people use OSINT.

Dave Bittner:

And so in terms of a company trying to manage what’s out there about them, what are your thoughts there?

Micah Hoffman:

It’s a challenge. I mean, you’re essentially trying to look at all the different places where people may be discussing topics, projects, people that are associated with you. So it’s a challenge, but for many of us, it’s a really fun challenge. It’s those detective or cops and robbers games that we used to play, that I used to play as a kid. You’re always looking for something and figuring out what it means and then looking for something else. So there are lots of places to go too. We see a lot of tools and techniques offering searches in the dark web and even searches within or searches or gathering data from social media and other places that are on the surface.

Dave Bittner:

Yeah. And one of the projects that you were co-founder of is the OSINTCurio.us online site. Describe to us, so what prompted you to create that?

Micah Hoffman:

That was a fun project. It was myself and some of the other people that are in the project were sitting around and talking about how nowadays within OSINT we’re having this blossoming of sharing information and it’s absolutely wonderful compared to what it was maybe five years ago where you had to go to a specific law enforcement training or into some secret room to learn about these techniques that only your company or your organization can know. And we thought about how there’s a lot of people that we are sharing information with that may have flawed analysis, may not be showing the tools in the best light. So we thought it’d be a great place to go ahead and create a community spot where we could share our experiences and do our research and share it all for free with whoever wants it on the Internet.

Dave Bittner:

Can you give us some examples? What are the kinds of things that you’re sharing there?

Micah Hoffman:

We have many blog posts that range from a more legal type of things. Like what happened to OSINT after the GDPR privacy regulations went into effect in Europe, to darknet and how we can discover the systems that are being used to host different darknet websites, to one of our members just posted one on using the command line tool curl for interacting with application programming interfaces. And that’s just the blog stuff.

We also have, we’ve thought about how people like to consume information and realize that everybody’s busy. So we started creating these ten minute YouTube videos. We call them Ten Minute Tips, and they are usually a discrete skill or a set of skills that are OSINT-focused that somebody can watch the video, learn a new tool in just ten minutes.

Dave Bittner:

Yeah, I mean, it’s really interesting to me because I think about, as we’ve come into the cyber age, over the past couple of decades, I think many of us in our minds, probably have the notion of that old private eye, sitting in a small office somewhere, a small smoky office, gathering information and the availability of that effort has really opened up to a much wider group of people.

Micah Hoffman:

Absolutely. And I love your example there, because that’s the same thing I have in my mind, something like Colombo or something where the guy’s wearing a fedora and a trench coat and that’s what we like to think about. But I’m watching social media, whether it’s Reddit, the OSINT subreddit, or in the OSINT team group, or even just on Twitter, looking at the hashtag OSINT. And I’m seeing a huge number of enthusiasts, just people that are interested in learning and growing skills, saying, “How can I contribute and how can I just get better at this?” And it’s wonderful to see.

Dave Bittner:

Now what about the ethical side of OSINT collection? How do you address that and keep people from crossing over into things like doxing?

Micah Hoffman:

Yeah. The ethical side is something we definitely talk about in OSINTCurio.us behind the scenes, and also in a lot of the other training classes that I teach, it’s an interesting line to talk about. You mentioned doxing is, that’s one thing, but even just the collection of information of people from a certain country without their permission, that could be against the law or many times we create fake personas or research identities so that we can get onto social media platforms. And when we do that, for most people, we now understand it’s a violation of terms of service, but for some law enforcement that actually can get them into hot water ethically.

And then we have the usage of breach data, that data that is stolen from some company and with usernames and passwords and then pushed online that we can access and find out the passwords of our targets. There are a lot of different reasons, things that we talk about and I would love to say there’s a line in the sand that we don’t cross, but the reality is that depending upon the country that you’re from, the countries your targets are from and what your organization and you are allowed to do, that line shifts. I have people that are from the EU and they have fewer things that they can do and people that just are private citizens and can do a lot more.

Dave Bittner:

Now within that sense of community, is there a certain amount of self-policing?

Micah Hoffman:

I think so. I like to think of it instead of policing more helping to learn and grow. We’ve seen some blog posts that are out there that maybe the person has created some logical fallacies in their analysis and they say, well, this happened, this happened, so that must mean this. And there is sometimes a reluctance to share publicly that information that, “Hey, you know what, you might’ve jumped to a conclusion there or created a false dichotomy or whatever.” But in private, usually, and this is the great thing about the community that we’re building is that we have a lot of different ways to reach out to people and say privately, “Hey, did you know that, you know, this could also be the case or that,” to advise in private.

Dave Bittner:

Yeah. I mean that’s a really interesting point that, you have these two phases. You have the gathering and then you have the analysis. Could you give me some insight as to what’s the difference between the two of those and do certain people gravitate towards one or the other?

Micah Hoffman:

Absolutely. When I was doing penetration testing and security assessments, I loved running tools. I loved running this cool python tool that would really quickly grab all the information about whatever target IP domain or person was out there. And then I’d run another tool that would do all of the thousands of Google dorks against the domain I’m interested in. And I would get all of this data and then I would have to sort through that, remove false positives and that was less fun. And then doing the analysis of what’s important, what’s an actual target and what should I go for next? That was intriguing as well.

So we do have people that are more focused on OSINT tools and hey, this is a cool tool and this was a tool to do that, building that OSINT toolbox of capability. But we also have a lot of people that are maybe coming from more intelligence or analytical backgrounds that are just killing it when it comes to, or just doing really well, when it comes to the analysis of what the data means. So I think we do have several different camps there and some people, many people, are mastering both.

Dave Bittner:

And it seems to me like an interesting mix of both art and science.

Micah Hoffman:

Yeah, I’d say that that’s very accurate. And the science would be the running of the tools and collecting everything, but the art is really understanding what the heck does this mean and is it relevant, is it truthful? That kind of stuff.

Dave Bittner:

I think a lot of folks today when they see how much information is out there about all of us and the ease with which it can be accessed, I think there’s a sense with some folks that there’s a certain disproportionality at play here that, for the folks who are out there gathering stuff, like they have the advantage these days?

Micah Hoffman:

That is actually something that is quite possibly true depending on who we’re talking about. There are some countries, like the United States, that shares a huge amount of information about the people inside of it. So even if you and I are not on any social media, the government and the organizations that we might visit in our neighborhoods and in our communities, they might share a bunch of information about us without our permission. And then you have corporations that are collecting and analyzing and sharing our information behind the scenes. So you don’t necessarily have that in other cultures and other places in the world because there are laws and regulations that prevent that.

Dave Bittner:

Yeah, I mean it’s interesting. I think even I remember when certain public record databases were taken online so you didn’t have to make that trip down to the courthouse or down to city hall to pull up information and that really allowed a different velocity of data gathering then you had in the past.

Micah Hoffman:

Yeah. I must admit that my OSINT experience, it doesn’t go back quite that far, but I’ve heard similar stories that we used to have to get a runner, you would contact a company that’s in whatever city, state, or location of the world you wanted them to gather information from, and then they would run down to the court and get whatever records you needed. And then send them to you by fax or by some other method. But yeah, I think the proliferation of information that’s being pushed to the Internet now is really an issue in some cases. As an OSINT professional, I love all the data that people are pushing, but as a private person, it really is scary.

Dave Bittner:

How do you reconcile those two sides of it?

Micah Hoffman:

I draw a very strong line in between them. It is a hard thing to do because, I educate my family and my friends, well with OSINTCurio.us, we educate anybody that we can and show them what we can as far as OSINT. And then in many of those situations we also tell them how to protect themselves so they don’t do the things that allow us to collect their data. And some people listen, some people don’t. But you’ve been around long enough, you know that it’s all an evolution, right? We’ve seen some things in the OSINT world in the last month or two that have been a huge shift from how we used to do things for years and we’ll adjust, we’ll adapt, and we’ll keep moving on. So even if we do protect ourselves in one way, then there’s usually other ways to get at that data or similar data.

Dave Bittner:

What kinds of shifts are you talking about? What’s happened recently?

Micah Hoffman:

Well we’ve had some major things happen. One of the biggest and most international is Facebook changing its graph search. So techniques that we used for many years to relate information and data about a certain person or a group of people has now changed dramatically. And to some degree decreased our ability to get at that data easily and quickly. There’ve been other shifts too, Michael Bazzell’s IntelTechniques.com site moving its tools and things behind a paywall and registration wall. That took a lot of people that were using those for law enforcement and other reasons and really truncated their capability, cut them off at the knees and they were looking for other places to do certain things. And then other things too, I mean people.com moved removed their public search so that now it’s only a paid service and there are some other ones out there too.

Dave Bittner:

Yeah. Now you are also the author of the SANS Institute’s Open-Source Intelligence Gathering and Analysis class. What prompted that, what’s the demand there and how’s that going for you?

Micah Hoffman:

It’s been a neat journey. I started that in 2016 and the impetus for that was I’d been in cyber for many years. And what I realized was that for somebody like me that really enjoyed the OSINT portion, that portion of the data gathering for a cybersecurity assessment or for whatever, it was a very small portion of many of the different classes. And I pitched it to SANS and said, “Hey, I think we could centralize this and go a lot deeper and provide more information to all different types of cyber-y people, whether they were penetration testers or digital forensics people or classic OSINT people as well.” And they liked the idea. So the class went public in September of last year and we’ve trained hundreds and hundreds of people all around the world. This year, it’s being taught 27 times live around the world and it’s an on demand as well.

Dave Bittner:

Now going through that process of putting together the class and then getting feedback from your students, were there any surprises, any assumptions that you had made that turned out to not be so?

Micah Hoffman:

Absolutely, yeah. And I tried to, with that psychology background that I have from way back when in my college days, I always tried to ask for feedback along the way so that I make the best product for my customer no matter what it is. And in this case the product was the class and the customer was, well that was the main problem. The customer was everybody and anybody. And so, we meant for this class to be an entry level, in the door type of class for people that had no prior experience or maybe had some prior experience with OSINT, and then we would get you up to an intermediate level. And then the plan was always that in the future there would be an intermediate and advanced class that would go deeper into some of the more technical things.

And one of the things I absolutely love is getting feedback from students saying, “Hey, this section, have you thought about looking at this resource? We use it a lot.” Or, “That tool isn’t as good as this one,” or, “Why don’t you cover X, Y, Z?” I love getting that feedback because it makes the course more applicable and better. So in the last year we’ve evolved the course, I’ve evolved the course, and my students have helped me a great deal to make it more relevant and relatable to everybody.

Dave Bittner:

I want to touch on threat intelligence and your take on it, the part that you think that it plays in cybersecurity, what are your thoughts there?

Micah Hoffman:

I’ll be very honest, threat intelligence isn’t a big part of what I do, so I’m not as well informed about it. But cyber threat intelligence is something that can help understand the risks that are coming to an organization or that are out there discussing an organization or, for the malware that’s out there and understanding who’s making it. And I think it’s something that we need to stay aware of, but I don’t have much more expertise in that.

Dave Bittner:

Yeah. So in terms of organizations who are looking to use open source intelligence to better protect themselves, what kind of tips do you have? How do they dial that in? How do they know how to start down that path?

Micah Hoffman:

Yeah, the first thing is a requirements gathering. When we talk about doing OSINT, I use an example in class of a person that’s told to do OSINT to about an event. And the question is what do you research? In the OSINT world, we can look at people, we can look at computers and domains and IPs, we can look at locations, we can look across social media platforms for sentiment analysis or hashtags that are being used, we can look at competitors.

So the first thing that I suggest is understanding what you’re looking for, what is the question or questions that you have, that you want to find out. Are email addresses found in any data breaches? Or is our competitor going to launch a new product? Or whatever it is. Figuring out what you care about is important so that you spend your resources appropriately. That’s really what drives it is that requirements gathering. Once you gather requirements, then you choose those tactics, techniques, and procedures that will help you achieve your goal.

Dave Bittner:

Yeah. It’s interesting to me that I think more and more folks in cybersecurity these days, you came to it in kind of a roundabout way. I mean it wasn’t, you didn’t set off right through high school and college that you were going to be a cyber person. You came in the side door but really found the thing that you love?

Micah Hoffman:

Yeah. And I’m seeing that more and more people are reaching out on Twitter all the time saying, “I am a bartender or I am doing this, but I’m drawn to that, the OSINT, and searching for people or searching for things online.” And what I love about it is that there are, in the cyber world, we’ve had capture the flags and online training and tutorial things that are free for a decade or more. But in OSINT, we’ve just started seeing in the last couple of years an evolution of the same type of thing where people can practice their OSINT skills by watching the quiz time Medium site and seeing what challenges they’re doing with geolocation. Or they can participate in a trace labs OSINT CTF and actually try to find missing people in the real world. There are some great OSINT for good and just OSINT challenges out there that anybody can use to build their skills.

Dave Bittner:

It was funny, I was going to say for the OSINT curious, what is the best place for them to get started. But of course you’re co-founder of a website called OSINTCurio.us. So I guess that’s step one, right?

Micah Hoffman:

Yeah, I would, I would recommend going to OSINTCurio.us.

Dave Bittner:

That’s your unbiased recommendation, right?

Micah Hoffman:

It is. Yeah. Well, and one of the things that we always try to do is provide attribution. So when Dutch OSINT guy sees that somebody in the OSINT team group puts in OSINT sent team chat a certain resource, he’ll go to Twitter and say, “Hey, this person in that place suggested this resource.” Always trying to link back to the actual person. And we try to do that all the time. Give credit where credit is due, on the website as well. So our site isn’t just about resources that we like and enjoy. It’s about providing that input from other people.

Dave Bittner:

Our thanks to Micah Hoffman from Spotlight Infosec for joining us. If you are curious about open source intelligence, be sure to check out the website OSINTCurio.us.

Don’t forget to sign up for the Recorded Future Cyber Daily email, where every day you’ll receive the top results for trending technical indicators that are crossing the web, cyber news, targeted industries, threat actors, exploited vulnerabilities, malware, suspicious IP addresses, and much more. You can find that at recordedfuture.com/intel.

We hope you’ve enjoyed the show and that you’ll subscribe and help spread the word among your colleagues and online. The Recorded Future podcast team includes Coordinating Producer Zane Pokorny, Executive Producer Greg Barrette. The show is produced by the CyberWire, with Editor John Petrik, Executive Producer Peter Kilpe, and I’m Dave Bittner.

Thanks for listening.