Simply put, big data for threat detection employs the latest big data analysis techniques to make threat detection better and faster. Threat detection is much like solving a jigsaw puzzle, it requires assembling often oddly shaped interlocking and tessellating pieces. Each piece contains a small part of a complete picture. Only when all the pieces have been perfectly aligned, a meaningful image is revealed.

A threat develops through a series of stages known as the cyber kill chain. The most advanced and unknown threats proceed undetected through every step of the cyber kill chain from pre-attack information gathering in the reconnaissance phase to command & control at the peak of the breach. Breaking the attack kill chain requires deep visibility into every activity on an endpoint, including surreptitious activities, over a prolonged time frame. Big data techniques can be used garner and analyze endpoint data to accelerate threat detection. Carefully analyzing large data sets of events, like puzzle pieces, reveals trends and insights that help detect threats faster and more accurately than popular signature-based prevention technologies.

Cybersecurity Kill Chain  

Capturing the Minute Endpoint Details

The execution of a malicious program or a file can include several actions. For example, the malware may create a child process, delete a dynamic-link library file, change a registry value or change a network destination in the routing table. Threat detection forensic investigation aims to gather all relevant evidence. This requires blow-by-blow collection of activities and events on a user endpoint, spread across various components such as operating system, network, process, memory, file, registry and host.

The richer the data collected from an endpoint, the better the threat detection deductions that can be drawn from it. The sheer quantity of activities generated on end-user workstations and active security monitoring generates large volumes, velocity and variety of data. This deluge of data can overload security tools and personnel alike, but if we utilise big data and machine learning methods all this data gives us an upper hand against attackers.

Big Data Improves Threat Visibility 

Attackers invest heavily in developing malware capable of quietly penetrating networks and remaining active inside without detection. Malware authors focus on persistence for two reasons. First, they need time to slowly traverse the network and learn about it. They look for important assets that can be later exploited for blackmail, data poisoning, insider trading or disrupting business operations. Secondly, much like a traditional espionage operation, they wish to maintain access for future exploits.

Therefore, the ability to organize and analyse large amounts of diverse data quickly and effectively is critical to an organization’s security. The analysts working in a Security Operations Center (SOC) need to ascertain the entire threat kill chain quickly, but they often lack the data visibility to do so. Threat detection involves connecting the dots between files and running processes to paint a complete picture of a threat vector, its dimensions of activities and execution into a detailed attack story. The faster the attack story is completed, the faster the threat response MTTR (mean time to respond).

Using big data technology to cope with such substantial amounts of data allows us to quickly perform risk scoring, graph analysis, time series analysis and determine causal relationships between entities to assess alerts and determine if they are suspicious or benign. If we couple machine learning and big data, we can sift and analyse massive amounts of data to detect cyber threats in a quicker, fluent and much more precise manner than traditional methods.

Big Data Endpoint Detection and Response  

Managing and analysing data has always offered great benefits for organizations of all sizes and across all industries. Performing analytics and complex queries on large data sets used to be very inefficient because big data technologies were not leveraged. Many security tools commonly used today were not designed to effectively analyse and manage data, so it is still difficult for analysts to quickly build an attack story and they are unable to shorten response times.

Implementing big data threat detection tools allows security analysts to piece together cyber threat puzzles faster. Big data analysis tools arm analysts to quickly detect and investigate threats, reach root cause and respond accurately.

Learn more about Cyberbit EDR Kernel-Based Endpoint Detection vs. Whitelisting