Splunk ES TA Change Log

Change Log

All notable changes to the Recorded Future Splunk ES add-on will be documented in this file.

[4.0.0] - 2018-10-23

New

  • Adaptive Response has been added.
    • The Adaptive Response can be added to any correlation search yielding supported IOC types (IP, domain, hash and URL). A new notable event will be created if the event can be enriched.
    • Ad-hoc mode is available (ex from the Incidents review panel), once used a drilldown link will open a panel with latest information about the IOC.
  • Added URL risk information.
  • Improved display of risk evidence in the Incident Review dashboard.
  • Support for Custom risklist using Recorded Future Fusion was added. Any number of risklists can be added.
  • Support for retreiving alerts from Recorded Future has been added.
  • Help pages are included in the app (including this Changelog).
  • New reports:
    • A new report "Latest updates of all risklists" was added.
    • A new report that show all log events from the app was added.
    • A new validation feature has been added. This feature can be used to verify that the app can work or to gather information about potential issues.
  • New options to customize access to Recorded Future's API (non-standard URL and optional SSL verification).
  • Search head cluster synchronization:
    • Only one cluster member retrieves risklists before distributing them to the rest of the cluster.
    • Configuration is synchronized, ex the API key can be added to any node in the cluster, it will be propagated to all nodes.

Changed

  • The filenames of the risklists in the the lookups folder have changed. Ex: rf_ip_threatfeed.csv has become rf_ip_risklist.csv. The transform used to map between the name and the file name has been adapted to ensure backwards compatibility.
  • Complete rewrite of the scripts included in the app.
    • Updates of the risklists and retreival of alerts have been implemented as modular inputs to improve reliability and scalability. Updates are performed as soon as new versions of the risklists become available.
    • The setup GUI has been extended and leverages Splunk's framework.
  • Minor graphical changes to adapt to Splunk's GUI changes introduced in Splunk 7.1.

[3.2.3]

  • Added a config stanza to manually override management host and port.

[3.2.2]

  • Adjusted config files to comply with certification requirements.
  • Improved how SPLUNK_HOME is detected if the environment variable is unset.

[3.2.1]

  • Bug fix in verify_rf_app.py which failed to take default values into account in one of the verification steps.
  • Modified verify_rf_app.py to flag missing folders which are created when running the risk list retreival script as warnings rather than errors.

[3.2.0]

  • Moved python modules into the bin directory (requirement from Splunk).
  • Added a new script (| script verifyRFApp) that performs a number of test on the system and app environment to help troubleshoot any issues.

[3.1.4]

  • Bug fix: changed the workflow to lookup an IOC from encoding the URL.

[3.1.3] - 2017-10-10

  • Handle case when there is a UniversalForwarder running on standard REST endpoint and the Splunk Enterprise is running on a non standard port.

[3.1.2] - 2017-10-03

  • Handle when Splunk refuse to tell which version of ES is running.

[3.1.1] - 2017-09-22

  • Updated icons.
  • Improved implementation of CLI launch detection.
  • Added verification that any proxy added in gui is a https one.
  • Obfuscate the token in the Setup form.

[3.1.0] - 2017-09-04

  • Made sure the update intervals don't slip.
  • Improved the setup GUI.
  • Added detection and prevention of CLI launch.
  • Added instrumentation of Splunk and Splunk ES version.
  • Renamed the default stanza to logging (new Splunk requirement)
  • Replaced 0 and 1 with false and true in inputs.conf

[3.0.6] - 2017-08-16

  • Handle byte order marks (BOMs) in web.conf.
  • Fixed wrong default log level (should be INFO).

[3.0.5] - 2017-07-24

  • Detect and use non default management port configuration.

[3.0.4] - 2017-07-18

  • Change application log to $SPLUNK_HOME/var/log/TA-recorded_future/get-rf-threatlists.py
  • Removed Eventgen samples and config.
  • Log version and OS when starting.
  • Create directory for lookups if it doesn't exist (can be the case on search head clusters).
  • Updated information about deployment on clusters.

[3.0.3] - 2017-07-11

  • Added the possibility to run "| script updateRFThreatlists" in the web GUI. This will print some stats about the risk lists and if needed update them.
  • Added logging in many places.
  • Catch and log before exiting in most places.
  • Added specific exit codes in most places.
  • Test if the passwords.conf file exist if the program fails to optain a token.
  • Added unittests for api_key.py.
  • Updated installation instructions.

[3.0.2] - 2017-06-21

  • Added saved searches to purge the Threat intelligence framework of outdated Recorded Future data.
  • Added per risk list configuration of interval, max_entries and enabled.
  • The get-rf-threatlists.py script now runs every 5 minutes by default. During each run it checks whether a new download is requrired for any of the enabled risk lists.
  • Removed the algorithm field from the generated CSV for the Threat Intelligence framework since this wasn't parsed by the framework.
  • Some changes to make support running on Windows.
  • Modified correlation search for domain based events to properly extract the domain from a URL.

[3.0.1] - 2017-06-01

  • GUI enabled to allow access to Setup in a search head cluster.

[3.0.0] - 2017-04-19

  • Make use of new Recorded Future Python API endpoints and corresponding Python library.
  • Added the domain and hash risk lists.
  • Generates separate minimized CSV files for the Threat Intelligence framework.
  • Renamed threat_keys to have rf_ prefix.
  • Reduced the size of the lookup files.
  • Added blacklisting to minimize the size of the Knowledge bundle.
  • Improved workflows to be more robust.
  • Added support to limit the maximum number of entries in each risk list.
  • Added support to enable/disable specific risk lists.
  • Added support to change the loglevel. Improved logging.
  • Removed JavaScript from setup.xml.

[2.4.2] - 2017-02-19

  • Temporary workaround for issues with Splunk password store.

[2.4.1] - 2017-02-15

  • Added instrumentation for troubleshooting interaction with Splunk password store.

[2.4.0]

  • Updated the RF correlation search so that it piggybacks off of the ES correlation search, 'Threat Activity Detected'.

[2.3.9] - 2016-12-31

  • Corrected issue in config_file.

[2.3.8] - 2016-12-22

  • Reworked the threshold so that a target number of entries is specified, the system will then select a threshold that will yield a number of entries in the vicinity of that number.

[2.3.7] - 2016-12-19

  • Added a threshold which only included entries with a risk score above a certain level.

[2.3.6] - 2016-11-29

  • Cleaned unused searches.

[2.3.5] - 2016-11-22

  • Merge

[2.3.4] - 2016-11-17

  • Improved resilience of temporary2.0.5 file handling.

[2.3.3]

  • Fixed bug - input script hitting API every minute

[2.3.0] - 2016-10-31

  • Various fixes to meet the criterias for certification.

[2.1.3]

  • Removed unused import in python setup script.
  • Various file permissions updated to match Splunk guide lines.
  • File name conventions and paths updated to match Splunk guide lines.
  • Changed location of temporary files to within the app directory.
  • Added documentation about requirements and cluster considerations.

[2.1.2]

  • Force lookup on correlation search to run on the search head and not on any remote peers

[2.1.1] - 2016-09-22

  • Fixed bug with temporary files left behind.

[2.1.0] - 2016-09-16

  • Fixed bug
  • Updated get-rf-threatlist.py to make sure rfsetup.conf exists before trying to get API token
  • Removed inputs.conf stanza to run get-rf-threatlist.py every 30 min
  • Created commands.conf file and added a saved search to run every 30 min that will run get-rf-threatlist.py

[2.0.6] - 2016-09-06

  • Removed wrong drop-down menu for Title in Incident View.

[2.0.5] - 2016-09-02

  • Fixed issue causing Splunk error "A script exited abnormally"

[2.0.4] - 2016-08-26 Ess

  • Fixed some issues with character encoding.
  • Improved error handling and cleanup after an error.
  • Fixed issue with wrong correlation search in saved searches.

[2.0.3] - 2016-08-24

  • Improved how the Evidence Details are displayed.
  • Risk Score, Triggered Rules (previously Risk String) and Evidences Details are listed in that order.

[2.0.2]

  • RF risk score is considered in Splunk ES overall severity.

[2.0.0]

  • Changed from STIX feed to CSV feed
  • Added fields for 'Risk Score', 'Risk String', 'Evidence String'
  • Fixed bug (data not removing from KV store after disabling app)

2016-04-03

  • Initial release (Beaker)