Adapt and Tune: Adapt Searches
There are two types or searches within the Recorded Future App for Splunk, correlation and enrichment. Both types are described in further detail below together with instructions on how to modify them.
Recorded Future provides a compilation of sample searches on the support web Common Splunk Search Strings for Recorded Future Risk Lists which shows how to correlate Recorded Future Risk Lists with various log files.
The Splunk feature “Workflow actions” allows for context dependent menu entries for search results.
A description of the actions not covered in the default set can be be found in the following article on the Recorded Future web site: Pivoting to Recorded Future Enrichment Data in Splunk.
The base searches for the Correlation Dashboards follow the same format and is used as a base for more specific searches.
This is the default base search for the Recorded Future IP Log Correlation Dashboard:
index=main sourcetype="netscreen:firewall" earliest=-24h | eval Name=dst | join [|inputlookup default_ip_risklist.csv | table * ] | search Risk != "" | sort -Risk
- index=main sourcetype=”netscreen:firewall” earliest=-24h selects the ‘main’ index, sets the sourcetype to ‘netscreen:firewall’ with a time stamp within the last 24 hours.
- eval Name=dst renames the field ‘dst’ in the logs to ‘Name’ by using the eval statement.
- join [|inputlookup default_ip_risklist.csv | table * ] joins the results of the previous search with a table containing all the fields available in the IP_risklist.csv lookup file.
- search Risk != “” removes any results that do not have any Risk assigned to them by searching for the items which have a value assigned to the field ‘Risk’.
- sort -Risk sorts the results in descending Risk order, by adding a ‘-‘ before the field name, to make sure that the highest risk results are displayed at the top of the list.
A typical modification is to use a custom risk list. In the Recorded Future Risk Lists, the main element is called Name. Its content depend on the type of Risk List so for instance it denotes domain in the domain Risk List and CVE in the vulnerability Risk List. To use a custom risk list, modify the second row to match the field names of both the data in Splunk and the Risk List.
Older versions of the Correlation Dashboards used the lookup which in the context of Recorded Future App for Splunk is much slower and will not remove any non-matching events from the result. It is faster to use join and subsearch with inputlookup.
This is the search that determines the amount of unique rows in the base search. The first statement does a statistical distinct count, thus only counting the unique values in the field ‘Name’ and saves the result in the field ‘count’. This value is then colored by using the rangemap statement.
| stats dc(Name) as count | rangemap field=count low=0-0 evelated=1-1 default=severe
The following subsearch creates the last table in the Correlation Dashboards to display the number of occurrences for each value in the field ‘Name’. In comparison with stats, eventstats adds a new field ‘Count’ to each row in the results. All duplicate rows in regards to ‘Name’ are removed with ‘dedup Name’. The Recorded Future macro ‘format_evidence’ parses the JSON object in the field ‘EvidenceDetails’, thus populating new fields with the data contained there. Finally, the data is sorted in descending order and cleaned up by removing all decimals from Risk before creating a table showing the final information.
| eventstats Count BY Name | dedup Name | `format_evidence` | sort -Risk | eval Risk = round(Risk,0) | table Risk, Name, Count, RiskString, Rule, Evidence
The Enrichment Dashboards use API credits to get more information about the specific entity from the Recorded Future Connect API.
The base search for each of the enrichment dashboards contains a REST call to get the information about an entity. It is possible to customise the information received from the API. By default all available fields selected to be able to present as much data as possible to the user. The available fields can be found at the Recoded Future API page. The CDATA encapsulation, in other words “regular expression in this search. The spath statement is used to parse JSON objects so that the information can be used in searches. The last two statements convert the timestamps to a more conventional format.
.+)" | rename rulesMax TO risk.rulesMax | spath input=timestamps | eval FirstSeen=strftime(strptime('timestamps.firstSeen', "%Y-%m-%dT%H:%M:%S.%NZ"), "%b %d, %Y") | eval LastSeen=strftime(strptime('timestamps.lastSeen', "%Y-%m-%dT%H:%M:%S.%NZ"), "%b %d, %Y") ]]>
A lot of the subsearches in the enrichment dashboards are quite similar, especially when it comes to the related entities tables. One example is the one below for related domains:
| `unpack_relatedEntities(RelatedInternetDomainName)` | sort -count, name | rename name as Domain | rename count as References | table Domain, References
The macro ‘unpack_relatedEntities’ parses the input JSON object. A macro is used to ensure that all related entity tables are handled the same way. The result is sorted and renamed to get suitable headers. The last row creates the table which is displayed in the dashboard, showing the fields ‘Domain’ and ‘References’.
“Recorded Future App for Splunk” has been developed by Recorded Future.
Further information and support can be found on our Support web site: support.recordedfuture.com