The Big Data challenge of cyber security data analysis has been one of topics discussed at length during Cybertech 2016. The sheer amount of data generated by Security Information and Event Management (SIEM) systems is making it increasingly challenging for SOC (Security Operations Center) staff to access, query, and analyze their data in a timely and cost effective manner. Here comes Spark. Apache® Spark™, an open-source cluster computing engine with in-memory processing, allows to speed up significantly security log analysis, providing an operational advantage of low latency data access.
According to a recent market study Apache Spark Market Forecast 2017-2020 the cybersecurity log analysis is the fastest growing segment of the Apache Spark market, likely to double by 2020.
Photo from Cybertech 2016: Analyst briefing “Using Apache Spark To Power Real Time Cybersecurity Log Analysis”