Threat Intelligence Feeds: Overview, Best Practices, and Examples
By RFSID on March 23, 2017
- Threat intelligence feeds are constantly updating streams of indicators or artifacts derived from a source outside the organization.
- By comparing threat feeds with internal telemetry you can automate the production of highly valuable operational intelligence.
- Selecting the right feeds isn’t enough. You should be constantly monitoring the ROI of free and paid feeds to determine their value to your organization.
- Advanced threat intelligence products hugely simplify the process of combining and prioritizing alerts from multiple sources, as well as providing a wide variety of additional benefits.
There are many ways to gather threat intelligence.
You can, for instance, purchase intelligence directly from a security vendor in report form. You can join an intelligence sharing group, such as the financial service industry’s FS-ISAC. You can even produce your own intelligence by analyzing incoming connections to your network.
But over the past couple of years, threat intelligence feeds have become the go-to intelligence gathering process for many organizations beginning to develop their threat intelligence capability.
And, in many ways, that’s great. After all, feeds have a lot of benefits: They combine intelligence into a single source, they’re easy to digest, and they have the potential to play an important role in security operations.
But if you believe threat intelligence feeds will answer all your prayers, you may be setting yourself up for disappointment.
There are several challenges inherent in using feeds, most notably that they lack context. In essence, a feed is a source of non-prioritized black and white information, so while it may provide a great deal of value it also requires a human analyst to check each alert individually.
Unsurprisingly, this can take up a lot of analyst time, and in reality it often results in many alerts that will never be triaged.
But we’re getting ahead of ourselves. Before we start to think about the benefits and challenges of threat intelligence feeds, there are a number of other things to consider.
What Are Threat Intelligence Feeds?
Let’s start off with a definition. Threat intelligence feeds are third-party streams of indicators or artifacts, with the singular goal of learning from other organizations’ access and visibility to improve your own threat awareness and response.
Functionally, threat intelligence feeds are almost inevitably delivered online, and usually focus on a single area of interest. For example, a feed might focus exclusively on domains, hashes, or IPs known to be associated with malicious botnet activity. An organization subscribing to such a feed could use the information provided to blacklist communications and connection requests originating from malicious sources.
Both free and paid feeds are readily available, and are created and distributed by non-profits (e.g., Shadowserver Foundation), industry groups (e.g., FS-ISAC), and vendors. But while the content and motives of each feed varies, the form taken is very similar: a growing list of security alerts that automatically updates when a new threat is identified.
The real-time nature of threat intelligence feeds is important, because when integrated with threat intelligence or SIEM (security information and event management) platforms it enables the automatic comparison of feed entries with internal telemetry such as firewall and DNS logs to identify potential attacks. Although very basic, this highly valuable form of operational intelligence can be tremendously beneficial to an organization’s security program.
Each alert will still need to be manually triaged, but so long as you’re careful about which feeds you subscribe to and your process for eliminating false positives a huge amount of analyst time can be freed up to focus on producing more complex threat intelligence that can inform improvements to defensive architecture.
Of course, in many cases, it isn’t possible for the entire process to be automated. In these instances, once a potential threat is compared with internal telemetry and identified as a concern, an alert will be created. If after investigation it is determined that a new security control is needed (e.g., a new rule for the firewall), it can be completed as with any other security update, and the alert marked as completed.
Hopefully, at this stage, everything seems reasonably straightforward. But it’s about to get a little more complicated …
Evaluating Threat Feeds
If you’re planning to develop a world-class threat intelligence capability, selecting the right threat feeds is essential. Sadly, there’s no way to objectively measure or compare the quality (or lack thereof) of available feeds.
There are some feeds that nearly every organization will find useful, and there are others that (at best) are pretty sketchy. But there’s also a huge array of threat intelligence feeds out there that are potentially useful, but which can only be evaluated with time and first-hand experience.
Equally, while there is good industry information available by experts on which feeds to use, in-depth understanding of your organization’s infrastructure is essential to provide truly meaningful advice. Choosing to work closely with a specific vendor is a good way to ensure information is specifically tailored to meet the needs of your organization.
But in absence of a paid security mentor, it’s important to understand how you can go about selecting the best threat intelligence feeds for your organization.
Naturally there is far less scope for regret with free feeds than there is with paid feeds. If it turned out little or no value was being gained from a free feed, you could easily drop it without a second thought. With that in mind, any feed that seems potentially useful could be tested for value.
With paid threat intelligence feeds, however, it’s a different story. As pointed out by Levi Gundert in his white paper on building a threat intelligence capability, at least one talented and experienced data architect is vital to the intelligence process. Assuming you have this person in place, it will be his or her job to evaluate paid feeds based on a variety of factors:
There are six main data source types (open source, customer telemetry, honeypots/darknets [deception], scanning and crawling, malware processing, and human intelligence), and ideally you’d want to cover as many as possible with your threat intelligence feeds. If certain data sources are more important to your organization than others, they should be prioritized.
Percentage of Data Unique to the Vendor
This is pretty obvious when you think about it. When selecting paid feeds, you’ll want to make sure you aren’t simply paying for an amalgamation of threat data that could be freely obtained from a mixture of free feeds.
Periodicity of Data
How long is the data relevant for? Is it relating to specific, immediate activity, or more strategic intelligence on long-term trends?
Understanding why certain items (e.g., IP addresses) have made it onto a feed is an important consideration, as it may make the feed more or less useful to your organization.
And as with any business initiative, simply choosing feeds isn’t enough. You need a way to tell whether they’re working for you.
Return on Investment
Naturally, now that you’ve invested time (and potentially money) in selecting the right threat intelligence feeds, you’ll need to make sure they’re performing. After all, business is business, and nobody wants to pay for something that doesn’t deliver.
And in order to do that, you’ll need to track your feeds.
Calculating the ROI of a particular feed will usually involve tracking the correlation rate, which is simply the percentage of alerts that correspond with your internal telemetry in a given week, month, or quarter.
Beyond this, you could go a step further and track effectiveness of any new security controls created as a result of each feed. For instance, if a new security control resulted in significantly more malicious connection attempts being blocked, it would be considered highly effective and reflect positively on the feed that informed it. On the other hand, if a particular feed led you to implement several new controls, but none of them were effective, you’d probably consider that feed something of a disappointment.
All of this assumes that you have a tracking process in place. Most threat intelligence and SIEM platforms include these types of monitoring functions, particularly if they have access to your network telemetry, so if you have the option this is certainly the easiest way to go.
Manual tracking is possible, but extremely cumbersome, so in absence of a vendor-bought platform it would be sensible at the very least to develop your own automated ROI tracking functionality.
However you decide to track ROI, it’s vital that you have a clear audit trail. Being able to link each external feed to the specific controls it informed (and their effectiveness) is essential.
Threat Intelligence Platforms
As already alluded, threat intelligence platforms make the task of ingesting, storing, organizing, and comparing threat intelligence feeds much more manageable.
Instead of viewing dozens of feeds separately, a good threat intelligence platform not only combines them all into a single feed, it compares them with internal telemetry and generates prioritized alerts for your incident response or threat intelligence team.
For example, if the IP address 220.127.116.11 was listed in a threat feed and also found in a web proxy log, that information would quickly find its way into the hands of someone with the necessary skills to blacklist the IP.
Perhaps the best way to consider threat intelligence platforms is through comparison. A financial trader can’t work without a Bloomberg terminal, because if they tried to do so they would be at a huge information disadvantage. Similarly, IT professionals in incident response, threat intelligence, and security operations centers (SOCs) are severely limited if they don’t have access to high-quality threat intelligence.
But selecting a threat intelligence product can be tricky, and there’s a world of difference between each vendor’s offering. On one end of the spectrum lie simple threat intelligence platforms, which simply ingest, store, and organize threat feeds. At the other end of the spectrum, you’ll find products that can perform a wide range of advanced functionality.
In our case, Recorded Future offers real-time threat intelligence (including context), with a huge breadth of source coverage (Tor and criminal forums, paste sites, code repositories, IRC, security blogs, and much more) and advanced search and alerting functions based on foreign natural language processing (NLP). Since many of the most prominent organized cyber crime syndicates have been based out of Russia and Eastern Europe, this last part is particularly important both to our customers and to our unique product offering.
Selecting the best threat intelligence product for your organization is a tremendously important decision, and one that should be entered into with as much knowledge as possible. Take the time to really consider your options, and don’t make the mistake of assuming that any two products are the same (or even particularly similar).
Designing Custom Feeds
One of the great things about the most powerful threat intelligence products is that organizations can make use of API to create and deliver their own customized threat intelligence feeds.
For instance, with Recorded Future you could create and send a daily file of all new IP addresses or malicious hashes with a risk score of 25 or more and correlate each with internal telemetry.
Alternatively, you could enrich all operational alerts by automating requests to Recorded Future for additional context on each IP, domain, or hash listed.
And these are just examples. Depending on your needs, you could use this functionality to run almost any computational analysis on a scale far higher than even a large team of human analysts could manage manually.
When they first appeared, threat intelligence feeds constituted a huge leap forward, as they enabled security professionals to manage much higher levels of relevant information than had ever been possible before. But as they become more abundant, free feeds in particular became “noisy,” and filled with errors and/or false positives. These issues, coupled with the sheer volume of data available, started to pose problems, and it became too much for security teams to handle.
Contextual Threat Intelligence
Without context and prioritization, the true benefit of threat intelligence feeds is elusive. But with context and prioritization, feeds can become a tremendous source of value to your organization.
For this reason, we designed Recorded Future to play the role of threat intelligence provider.
Instead of simply ingesting a bunch of free feeds, Recorded Future actively produces threat intelligence that’s unavailable anywhere else, contextualized by your own internal risk indicators.
With Recorded Future Intel Cards, organizations now have the power to whittle truly massive volumes of data down to a manageable number of prioritized alerts. Not only does this free up analysts to concentrate on larger security issues, it enables security professionals to make real, measurable improvements to the security of their organizations.
In fact, a recent independent lab test found that real-time threat intelligence from Recorded Future boosted analyst productivity by 10 times, cutting the average time to triage a security event from three minutes to 1.2 seconds.
To see what Recorded Future can do for you, request a demo with one of our expert analysts.