Introduction
Security operation centers (SOCs) are at the core of monitoring, investigations and response for most enterprises. They are where security analysts drive the process for alert ingestion, triage, make decisions based on available information and respond to threats. What makes SOCs effective are the people who work day in and day out to improve the overall security of the organization. While being a security analyst has desirable compensation, it is no surprise that working at a SOC comes with a lot of responsibilities and challenges.
In this article, I’d like to highlight some of the common practical hurdles we observe with uncertain (aka non-conclusive, non-definitive) security alerts, and our recommendations to overcome them.
A typical day in the life of a security analyst
Outside of research, special projects, product evaluations and continuous training, the majority of a security analysts' work consists of handling different types of security cases. In practice, this means that your effort is split between multiple cases assigned to you by the team lead or a round-robin assignment flow.
An assigned case can be a specific request from HR, user reported issue or an audit/reporting requirement, in which case the analyst will focus primarily on collecting data to prove or disprove a hypothesis.
Up to 80% of an analysts’ effort goes towards cases based on security alerts from detection, monitoring and correlation systems. Unlike specific requests, these cases require discovery, scoping and prioritization by the analyst. In nine out of ten alert-based cases, the analyst’s job is to prove that there isn’t a serious security threat/issue. Although popular culture may suggest otherwise, Russia, China or EvilCorp are not behind every single alert or attack in the SOC.
Triggered by state actors or not, every case requires analysts to make decisions based on evidence available to them. The issue in most situations is that they don’t have access to all required information. For example, a SIEM may be used to create alerts and collect data, but multiple systems in scope may not be sending their logs to the SIEM. In these cases, analysts can be forced to reach a conclusion without having the complete context.
Uncertain alerting wastes cycles and causes fatigue
This is where the quality of signals and the volume of signals, in this case security alerts play an important role. False positives aside (more on those later on), most security alerts are verbose and far from being conclusive. They indicate interesting patterns that need to be looked into, instead of potential malicious activity or breaches.
On an average day, analysts need to comb through dozens to hundreds of alerts to make sure these interesting patterns are not indicative of malicious activity including attacks and insider threats. Obviously, not looking into these alerts can mean that significant security incidents can be missed. Yet, looking into each alert consumes valuable cycles which means more interesting alerts may be skipped or overlooked. (This phenomenon was recently highlighted in a ZDNet article, stating “More than half of security practitioners believe vendors are flooding them with ineffective alerts to avoid accountability should a breach hit.”)
For example, your productivity suite or DLP provider may generate alerts saying:
- Jack just downloaded 500 files.
- Maria just authorized an app that deleted 150 files in 2 minutes.
- Emma signed in from three new locations on the same day.
In these alerts, your security vendors don’t tell you that there is a breach. They’re giving the analyst a lot of places to look, asking for them to make a judgement. For these examples, an analyst would have to get up to speed on who these users are, what their usual behavior looks like and what these behaviors entail to make informed decisions on these alerts. Each of these steps take minutes to hours depending on the complexity of the tech stack and context. You need conclusive decisions to close a case. And every good decision is made using common sense and the specific context of the situation. And to get the complete context, analysts need the flexibility to move their investigation timeframe and widen their aperture.
Wasted cycles aside, these alerts also cause fatigue for analysts, increasing human errors and hurting motivation of these teams.
Keys to success: Identifying the right scope and collecting the right information
Even for tier-2+ investigators, one of the key challenges is quickly identifying the scope of an investigation and deciding where to focus their efforts. Determining where to invest time and where not to becomes a critical question. Analysts typically rely on their SIEM as the single source of truth, especially when tracking an IP or identity, but they often discover that systems within the case’s scope may not be connected to the SIEM. In other cases, the SIEM may contain only partial information, requiring further investigation into individual data sources.
For most cases, common sense and context are driven by job role and normal behavior of user(s). If the case is about a user sharing HR-related information with third parties, and this user is in HR managing contracts, that behavior may be normal. If they are in mechanical services, it may be unusual and needs more investigation. In many cases, behaviors trigger the alerts, but to reach a verdict, an analyst needs to understand how this behavior relates to their role, normal behavior and the additional context.
In short, we need to find ways to facilitate decision making for analysts to overcome the high volume of uncertain alerts.
Playbooks are good for known patterns, but what about everyday cases?
Playbooks do a great job of running pre-built programmatic steps for known case patterns. This is promising but in practice, you can’t fully rely on playbooks because many times, the cases you have in your pipeline are new. Security vendors introduce new alerts all the time and attackers create variants of known attacks to bypass playbooks.
When you have a case/alert in your pipeline, you have to take care of it then. So, there isn’t time to wait for the content team who write playbooks. This team often is not part of the SOC operational team who are responding to every case. It may take days or weeks until you have the right automation in place. Analysts need to figure out their steps and replay it time and time again. This is why they can’t rely on the SOAR/content team to build a new playbook for every case.
The other issue with playbooks is that when the alert data and the flow change (and they change often), their playbooks break – making this programmatic approach less useful for the case at hand. Playbooks are heavy, they break easily, and they are a pain to build/maintain.
Practically speaking, analysts need the right tools to create investigation flows on the fly. It's also realistic to expect that these same flows can be refactored into playbooks eventually, but something must fill the gap in the here and now. Every security system generates its own alerts daily and introduces new alert types frequently, and as more security vendors are introduced, the number of alerts will continue to increase—especially since these vendors are being paid to deliver more alerts.
Recommendations to optimize effectiveness of analysis
There are a lot of methods to optimize the output of security analysts, but two methods that we have seen work with multiple organizations are:
- Resist the temptation: Don’t just focus on the things you can easily understand
It is human nature to stick with what we know. So, it is tempting to only dig into the cases that involve familiar tools or behaviors. But the truth is, most interesting cases will have artefacts that are outside of an analyst’s comfort zone.
Another misleading factor can be how an analyst’s performance is measured. If you’re judged on case time to close, it is tempting to only look at the cases you understand or can close in a short time.
- Have a plan to block and tackle new case patterns
You may have programmatic methods (SIEM, SOAR playbooks or hyper automation) to triage and combine alerts. These can save valuable cycles and can reduce the number of cases that need analysts’ attention.
In practice, you will have to deal with new patterns on a daily basis. So instead of relying completely on programmatic methods, have a plan to explore new patterns on your own. Once explored, it is helpful to establish a temporary investigation path while the content team incorporates these new patterns into programmatic methods.
To start this plan, documenting the tools and knowledge available for data collection and interpretation can go a long way. Some modern SOC tools have good means to achieve these exploratory goals, Command Zero offers a pretty effective way with individual questions and facets too.
Inevitably, some of these new patterns will be false positive alerts. False positives are as old as security alerts as a concept, and they keep haunting our industry for a reason. As we all acknowledge they suck, we need to continuously look for ways to balance eliminating uninteresting findings and casting a broad net to avoid missing malicious patterns. Having access to historical context and the flexibility to expand date ranges to learn more about normal behavior are time-savers.
The combination of these methods helps analysts get up to speed more quickly on vague cases and get to a verdict more quickly.
- Accelerate with automation and GenAI where it makes sense
Automation and AI clearly have a lot of potential for improving SOC processes. The good news is that these methods are not replacing human analysts anytime soon, and they are generally better than humans only in tasks that humans already despise (yes, I’m thinking reporting!). What’s even better is that some of these capabilities can be easily integrated into current processes. For example, automation can help standardize and streamline remediation steps while GenAI agents can help with some basic queries and summarization during investigations.
Conclusion and call to action
Uncertain alerts are part of the current reality for SOC teams, but we see improvements in alert quality and solutions made available to analysts. I firmly believe we need to facilitate decision making for analysts to solve this problem. Improving data collection, context building and giving analysts flexible tools to widen their understanding of each case will be key. We are in the early days of effective purpose-made solutions for security analysis and I’m proud of our ongoing contributions to this space with Command Zero.
I encourage all oversubscribed SOC teams to get a demo to see how it can transform their day-to-day operations.
.png)