Beyond Feeds: A Deep Dive Into Threat Intelligence Sources

Editor's Note: The following blog post is a partial summary of a SANS webinar we co-hosted with Dave Shackleford.

Annoying, isn’t it?

No matter how hard you try, you just don’t seem to get the results you’ve been hoping for.

You invest resources, time, and energy. And yes, you see some results.

But deep down, you know you could be achieving more. Maybe with a bit more expenditure, and a few more skilled analysts, you’ll finally get to a point where threat intelligence is making a measurable difference to your organization’s security profile.

Unfortunately, many organizations never quite reach this point.

Recently, Recorded Future co-hosted a webinar with SANS Institute, with the goal of helping security-conscious organizations identify and avoid some of the biggest pitfalls in threat intelligence. In particular, the webinar covered how organizations should think about selecting threat intelligence sources, and the dangers of relying too heavily on a single source.

The speakers were Chris Pace, technology advocate at Recorded Future, and Dave Shackleford, SANS analyst, instructor, and course author.

Your Sources Determine How You 'See' Threat Intelligence

To kick things off, Dave put up a graph depicting some of the results from a SANS survey conducted last year on cyber threat intelligence. In this case, respondents were asked for their top three use cases for threat intelligence within their own organization.

Straight away, we saw something interesting. A very high proportion of organizations were already using threat intelligence to block malicious domains and IP addresses, with many also using it to add context to investigations or compromise assessments.

Very few organizations, though, went further than this. The third most common usage of threat intelligence was employed by less than a third of all respondents, and things only descended from there.

Recorded Future’s own Chris Pace explained why: “If you have a stack of feeds that give you malicious domains and risky IP addresses in a very binary way, that's how you’re going to perceive the usefulness of threat intelligence, because that's what's accessible to you.”

“I think the reason that number on the left of the graph is so high, and the other more interesting and potentially more useful things are lower down, is the perceived barrier to entry for accessing that intelligence, ingesting it, and converting it into a usable format.”

Common practice dictates that a threat intelligence initiative should start with a basic platform and a few open source feeds. And in a sense that’s understandable, as it’s certainly the cheapest way to get started.

But in reality, single source threat intelligence just doesn’t work. Instead of solving an existing operational problem or addressing a known risk, using a single source can force you into a very reactive position, whereby analysts are forced to constantly triage new alerts that contain little (if anything) in the way of context.

And that’s just it. If all you currently have access to are stream-of-consciousness threat feeds, a lot of the really worthwhile aspects of threat intelligence are going to seem unattainable. Sadly, many organizations assume this is a failing on their part, but in reality it’s nothing more than a basic misunderstanding of how powerful threat intelligence facilities are built.

'Threat Intelligence Is Hard'

Of course, sources aren’t the only barrier to effective threat intelligence gathering. To further explain why many organizations are finding threat intelligence such a struggle, Dave put up a second SANS survey graph.

Take a look at the top two barriers to effective threat intelligence: Lack of trained staff, and lack of technical capability.

Now, the skills gap isn’t exactly a new story, but consider the third most common barrier: Lack of management buy-in. If you had to guess, it seems highly likely that this absence of buy-in from executives and budget holders is at least partly responsible for the lack of skilled personnel needed to implement and maintain a powerful threat intelligence facility.

Now, look at the rest of the barriers cited by survey respondents. For the most part, they relate to the high volume, lack of perceived relevance, and lack of prioritization. Put simply, many organizations are overwhelmed with alerts, too many of which are false positives.

Once again, this is an issue of starting from the wrong position. Since management buy-in can be difficult to come by, organizations seek to launch their threat intelligence initiative in what appears to be the easiest and cheapest way: threat intelligence feeds.

But in practice, these feeds are anything but easy to use. They absorb a huge amount of analyst time, deliver very little, and worst of all, they give organizations the toxic view that threat intelligence is simply about playing “whack-a-mole” with the latest vulnerabilities and malicious domains.

“It’s a self-fulfilling prophecy,” explained Dave. “You get this easily understandable data that’s readily accessible, and you feel like you can immediately put it to use ... but it stops there.”

So if not with threat intelligence feeds, where should you start? Simple: Start by solving an existing problem.

“It doesn’t matter whether it's brand monitoring, leaked credential monitoring, uncovering emerging threats, or vulnerability identification and prioritization,” explained Chris. “You need to have a goal first before you can get into the weeds of how you are going to use threat intelligence.”

A Rundown on Intelligence Sources

Of course, to have an understanding of which problems can be solved using threat intelligence, you do first need to understand what intelligence is potentially available. During the webinar, both Dave and Chris spent some time covering the most common sources.

1. Open Source Feeds

There are hundreds of these available, covering every aspect of security you can possibly imagine. Implement a basic threat intelligence platform (TIP), and you have everything you need to start digesting truly unmanageable numbers of alerts.

2. In-House Threat Intelligence

Often referred to as “security analytics,” in-house intelligence is produced by incident response teams, security operations center (SOC) personnel, and security analysts. Past attack forensics are a common example.

3. Vertical Communities

Certain industries and verticals have access to intelligence-sharing communities, such as the Financial Services Information Sharing and Analysis Center (FS-ISAC). These communities can be extremely valuable, but are often gated, meaning even organizations processing a high volume of transactions may be disallowed access if they don’t possess the “right” brand name.

4. Commercial Services

Offered by dozens of security vendors, often referred to as providers, commercial threat intelligence services vary wildly in quality and scope. At their best, they offer vital insights into one or more areas of intelligence with far fewer false positives than their open source alternatives. At their worst, they’re expensive, and fail to deliver truly actionable intelligence.

5. Dark Web Intelligence

Try to forget about your preconceptions. Functionally, the dark web definitely isn’t “everything not indexed by Google.” From the perspective of cybersecurity, the dark web is around 500–600 underground forums, which can only be accessed using specialized browsers.

The most recent darling of the threat intelligence world, the dark web offers organizations the chance to identify stolen assets, explore threat actor targeting, analyze exploit kits, and much more. Unfortunately, given the need to understand multiple languages (including relevant slang) and potential for drawing the attentions of dangerous parties, gathering these gems in-house can be more trouble than it’s worth.

Many of the more secretive (and thus valuable) communities are extremely difficult to join, requiring existing members to vouch for new applicants, and even demanding joining fees running to thousands of dollars.

Challenges in Threat Intelligence

As you can see, open source intelligence is just the tip of the iceberg. Unfortunately, though, since many organizations quickly become overwhelmed by the sheer quantity of low-yield alerts these sources provide, they never have the capacity to take on additional sources.

The simple truth of threat intelligence is that more isn’t always better. In fact, when it comes to alerts, more is nearly always worse.

“It’s not surprising that somebody would be overwhelmed by the magnitude of feeds and data that are available,” Chris noted during the webinar. “As vendors, as providers of threat intelligence, it's down to us to deal with that problem. It shouldn't be down to an organization to try to drink from the fire hydrant of data.”

Note the use of the term “fire hydrant.” In reality, the reason why many organizations feel overwhelmed by threat feed alerts is that what they’re receiving isn’t intelligence at all, it’s raw data. To be considered intelligence, that data would need to be filtered, processed, and formatted to ensure only relevant and valuable alerts are fed through the human analysts.

Most of the time, this just doesn’t happen.

So if more alerts isn’t the answer, and most feeds provide little more than raw data, how can organizations possibly be expected to develop a powerful threat intelligence facility? This is where Dave and Chris introduced us to to the “two Cs” of threat intelligence: context and corroboration.

The Importance of 'All-Source' Corroboration

There’s no getting around the fact that having more data available to you will necessarily improve the likelihood of receiving the alerts your organization needs to remain secure.

At the same time, as we’ve already seen, having access to massive quantities of threat data is totally unmanageable for even the largest and most highly skilled teams of analysts.

So in a world where skilled analysts are at a premium, balancing the need for more data with the imperative to minimize alerts is the name of the game. The answer is simple: technology must play a central role in corroborating and contextualizing threat data before handing it over to human analysts for action.

By combining and contrasting threat data and information from a broad range of external and internal sources, powerful threat intelligence solutions such as Recorded Future utilize machine learning and artificial intelligence capabilities to produce genuine threat intelligence. As a result, the outputs from these solutions are far less numerous than the single source alternatives we discussed earlier, but dramatically more actionable.

For a very simple example, consider firewall logs: A high-volume, low-context source of threat data that’s extremely time consuming to process manually. When enriched by Recorded Future, which draws on a broad range of external intelligence sources to provide context and discount false positives, an analyst was able to process these logs 10 times faster than the manual alternative.

“For me, these two things are the most important areas to consider when working with threat intelligence. We know timing is critical. We know dwell time is a big concern. We know the longer something goes undetected, the greater the risk that it presents. Vitally, we know we need context in order to be able to make decisions,” Chris summarized.

“As threat intelligence begins to mature, and organizations become more familiar with it, I think striking a balance between these two things will become the ultimate goal for anyone looking to incrementally improve their threat intelligence capability.”

To see all-source threat intelligence in action, sign up for our free Cyber Daily email to see the top daily results for technical indicators such as top targeted industries, threat actors, and exploited vulnerabilities.