Introduction
This blog post is the third post of a blog series covering the key findings of our first research report “Top Challenges in Cyber Investigations & Recommendations for SecOps Leaders”, published on September 10, 2024. You can read the first two blog posts of this series here:
In this post, we will cover the second key finding of this research:
Current SecOps tools are hard to operate and investigate
EDR/XDR, SIEM and SOAR are the three most widely deployed SecOps tools today. These technologies are foundational pillars of information security programs, used by SOC and IR teams across the industry. Although EDR/XDR, SIEM and SOAR are powerful, they incur significant costs due to deployment and management challenges.
EDR/XDR is a robust and powerful tool in capturing endpoint data. However, investigators begin to experience challenges when tasked with correlating network and cloud telemetry. An even bigger issue with EDR/XDR is the hefty price tag. Often, it is cost prohibitive to deploy EDR/XDR at scale in cloud environments. This in turn, can lead to visibility gaps.
Key findings
85% of respondents considered EDR as the most heavily relied upon investigation tool. 76% of respondents reported ingesting security relevant data to a SIEM for investigations and GRC purposes, with EDR data being the primary data source. However, respondents also stated that it was prohibitively expensive to use SIEM effectively to cover collection and retention of all security data.
59% of respondents expressed concerns about staffing costs associated with running their SIEM. While at the center of detection, correlation, alert escalations and investigations, SIEM and SOAR technologies have proven to be highly labor-intensive when it comes to implementation, customization and operations. 75% of respondents cited the lack of resources and skills required for integrating data sources into SIEM and SOAR. Most respondents also expressed they are using a third party or dedicated security engineering resources just to keep SIEM and SOAR systems operational.
Highly specialized skills are required to deploy, customize and maintain a SIEM. This involves the complicated process of developing rules and scripts that integrate event/data flow. Further, the financial cost of data retention is a significant and growing barrier due to the explosion of data across the enterprise environment. The SIEM is too costly to be fully deployed (across heterogenous cloud environments), adequately integrated with numerous data sources, and properly maintained.
The final security product which warrants discussion is SOAR. Contrary to the initial promise of the concept, SOAR is difficult to deploy, maintain and integrate. Respondents who utilize a SOAR all emphasized the need for specialized resources to script and automate playbooks. As a result, most SOAR investments are limited to using default playbooks or mildly customized playbooks that require a lot of manual work when it comes to investigating cases.
Investigation teams often encounter additional difficulty incorporating data from non-security products (such as Active Directory, source code repositories, case/ticket management, document management systems, etc.). This information is often needed for application, user, and data loss probes. In turn, this challenge leads to more manual efforts for analysis, resulting in lengthy and costly investigations.
Data collection, processing and retention surfaced as other main obstacles for security operations. 76% of respondents were unsure if they had collected all the data necessary to adequately investigate breaches across all their computing platforms.
83% of respondents stated that access to SaaS log data is essential for incident response. However, less than 50% ingest SaaS logs into their incident response data platforms. Business applications and core SaaS applications are increasingly becoming high value targets since they can host IP and other sensitive company data.
Blind spots in investigations are common due to the narrow focus on security alerts and logs. Only 28% of organizations automate the integration of non-security data sources.
Similarly, 90% of respondents consider network data a crucial factor in investigations. Yet less than half of the organizations surveyed collected network traffic flow data, citing concerns over volume and retention times.
Command Zero’s perspective
Despite the early and sincere focus on search/investigations, modern SIEM and SOAR capabilities have evolved to satisfy compliance/regulatory requirements. These technologies do not provide dedicated investigation tools and the right user experience for an effective flow.
Most SIEM features and engineering effort focus on collecting more raw logs and data retention in an economically feasible way, while pushing these logs to data lakes for long term storage and archiving. SIEMs do an excellent job at ingesting high volume of raw logs, normalizing, indexing and storing these logs while running static correlation rules to surface alerts. Due to storage limitations, cost and the difficulty of ingesting custom data, centralized logging on SIEMs is commonly limited to security devices only, generating gaps in visibility.
SOAR is a concept invented to overcome the flood of SIEM alerts and automate response to known threats. SOAR excels at static pattern matching via playbooks and improves the fidelity (aka true positive concentration) of the alert funnel. It does a good job at pattern matching the known alerts, but any minor change in the pattern breaks the rigid playbook structure and SOAR becomes useless for these alerts. In practice, SOAR fails to understand the full context of alerts and adapt to variants of alert patterns.
Although SOAR is not the best solution to cyber investigations, there are a lot of benefits to using SOAR. SOAR automates repetitive tasks, responds to known threat patterns in a programmatic way and improves overall security while reducing the effort for the security operations team. But this comes at a cost. The most consistent industry feedback is that SOAR platforms require advanced security engineering and developers to setup, customize and maintain. This leaves SOAR users restricted with the handful of default playbooks or investing in a full-time content/security engineering team to keep SOAR operational. This brings up the obvious question: With limited resources, should security operations teams focus on engineering playbooks when they could be focusing on real security issues?
Combined, SIEM and SOAR deliver a necessary service for security operations. They help continuously monitor the environment for alerts, adhere to compliance and regulatory requirements and can identify interesting alerts/cases that need further investigation. But when it comes to handling escalated investigations, they do not provide a clear path to follow actors across complex environments.
EDR/XDR technologies have come a long way with search and investigation capabilities, relying primarily on the data from endpoint agents. The issue with this approach is that EDR/XDR provide no value for systems that do not have agents installed. This means fundamental systems including Identity Providers (IDPs), cloud components and SaaS can be out of scope for investigations run on EDR/XDR. As of August 2024, some EDR/XDR vendors are adding SIEM/SOAR offerings to their portfolio. It is yet to be seen if these efforts can succeed or if they will carry the same design limitations of SIEMs.
- Advanced training requirements for SIEM and SOAR mean that subject matter expertise will always be siloed within the team. Analysts running cases need to pull in other individuals to get full technical coverage. This also makes redundancy within the team more challenging since more team members need to get training on each platform. Additionally, platform user/admin training is a significant time investment.
Overall, security operations teams are left with SIEM, SOAR and EDR/XDR systems that do a satisfactory job collecting logs, generating alerts and triaging alerts. Yet, for escalated cases that require further investigation, tier-2+ analysts get little to no support. This means, investigators run investigations with a patchwork of open source, commercial and custom tools.
Recommendations
- Data collection and gaining visibility into your environment is key for security operations. Assume and accept that there won’t be 100% coverage of all IT systems nor enough content for detection across all systems. Identifying the gaps you have and fixing them can help improve security. For example, knowing you’re not collecting GitHub logs (or that bespoke web application) today, and creating a process for common GitHub investigation types in the future.
- Investing in conceptual and technology-based training for your security operations team will not only make them better at their job but will help with talent retention too.
- Implement layers of abstraction where possible to maximize the value received from individual solutions. Being able to build narratives using various data points across multiple platforms using a single solution is ideal to minimize technology expertise requirements for your team.
Conclusion & What’s Next
We covered the second key finding of this research on this blog post, we will dig into the final key finding and recommendations on our next blog post in this series.
If you’d like to read the full report, you can download a copy from the report overview page on our website.
