.png)
Introduction
Tackling complex cyber investigations at scale requires simplification of the investigation process and building an expert system that will support all users. In this post, I’ll cover the need for expert cyber systems and how the Command Zero security research team distills decades of knowledge into applicable content in the platform. Two of the key benefits of using an expert system for investigations are increasing speed to get to answers and improving the consistency of investigations across all analysts. Before I dive into the meat of this topic, I’d like to share my background which helped me understand the importance of pragmatic approaches in cyber investigations.
My background is fairly diverse in terms of roles and areas of responsibility. I officially got my start in the United States Air Force as a 3C, computer systems operator. Unfortunately, at the time the Air Force lacked Cyber Command, but I was fortunate my commanding officers saw my talent and put me into a Tier-3 operator position doing Active Directory and Exchange Administration. I went on to work as a contractor for the Air Force and other Government agencies for more than a decade after leaving Active duty, working information warfare and a few other jobs involving red teaming and offensive / defensive work. This early portion of my career shaped my understanding of networks and faults in system architectures and led me to become a better penetration tester and defense engineer as well.
Leaning on my past experiences and understandings, I fold this knowledge into how the team operates here at Command Zero. From the research labs to the methodologies we employ in creating content and building integrations for analysts to use daily.
Systems for experts continue to fail cyber as an industry
There are two types of systems used to conduct cyber investigations today. You have point (security) products built for the expert user, then you have centralized security data stores which are core to several functions, built for a specialized product expert... who is not necessarily a security analyst. The short of it is, we can’t keep building systems for expert users only while we all agree that there is a universal talent shortage in cyber.
In other words, there are two types of cyber systems built exclusively for expert users. First, security products such as EDRs which focus on security for your endpoint. Second, you have systems such as Identity Management whose focus is on providing a service around identity. What you find is the information between these two systems are presented and available in very different ways, and the systems behave differently in the methods in which data is available through API or the interface. This creates a disconnect between the administrators (or experts) running these systems and security analysts who are responsible for making sense of the data generated by these systems in the grand scheme of the environment.
Shortcomings of point security solutions
Let’s take two very different products as examples: an EDR and an Identity Provider.
The EDR will give you visibility into signature/behavioral and analytical-driven detections it determines to be malicious on endpoints. From here, analysts are expected to understand the nuance of the alert, and through what is displayed ascertain impact, scope and what to do with it. One of the critical steps in security operations (if not the most critical one) is understanding what happened with confidence. Alerts in isolation give an incomplete picture, incidents which serve as collections of alerts are better. An incident paints a narrative and includes more depth to what has happened, but the detection product is only as good as the visibility it has, in many cases limited to the endpoint. To obtain the full picture, analysts have to ask:
- Has the incident expanded beyond the endpoint?
- What users were involved?
- Is the attacker still there?
- Are any other systems impacted?
- What about the cloud applications the desktop has connected to?
The expectation of the analyst is to know how to ask these questions, but we also to have the following expertise in relation to the product itself:
- The investigative and technology specific expertise to ask all these questions to any system that might be involved,
- Where to get the data and has direct access to it,
- How to interpret that data from various systems in scope.
Shortcomings of non-security solutions
Another interesting aspect of these overly ambitious expectations is that analysts will have the same level of knowledge about non-security products, like identity providers or code repositories, as they do about security products. Non-security products are generally completely outside of the purview of security teams, or at least the analyst team. And while security products may be in or closer to their purview, each system in scope is still riddled with nuances and high barriers of entry to meaningful outcomes.
Let’s address the non-security products aspect with an example, an Identity provider such as Microsoft Entra ID or Okta Identity Cloud. These products are built to be identity providers, so the context in which information is provided is towards administrators. Navigating away from your investigation into one of these identity providers, the information is presented nicely, but it requires interpretation in the security context, or within the context of the case at hand. That’s if you have direct access to these systems. This is the case with many SaaS applications, they are not security products so investigating any cases that involve them is often difficult, both because of the data they make available and how it’s displayed.
In general, the shortcoming of platforms for experts is the overall assumption that the operator driving the console is an expert. Often, we find ourselves using a product which assumes the person interpreting the other end of the data being presented knows how to interpret it. Over the years, the industry has changed, knowledge has changed, and new products are being introduced at a faster pace. Expert knowledge is derived from experience and observation. If you look at most cyber products, they present a complex finding to you and you are expected to understand the totality of the alert.
What happens if you are unsure how to interpret, or even what you are looking for to validate it? This is a tricky situation even for organizations who can afford specialized teams for each product: Steve handles all the alerts for identity, so he is the only person to deal with risky user cases! What happens when Steve gets a competitive job offer or gets sick?
Centralized data stores and the painful retrieval of data
The flip side to the platform expert users is data stores, consider a SIEM or data lake. While these systems do a good job of centralizing and normalizing log data, running investigations on them can be tricky. Depending on how these systems were set up and some of the architectural decisions determine what investigations can even be performed. When crafting a query for data, are you certain it contains all the data you need? How can you be assured that the cryptic SIEM query you’ve been using, written by someone years ago, even contains all the data you were seeking? What happens when you have a complex query that you yourself need to write?
I’ve personally found times where I had to retrieve data, only to find myself spending an hour debugging the query, for it to return data and I forgot not only what I was looking for, but how I came to find myself looking for it.
“There are unknown, unknowns”
- Donald Rumsfeld
The same shortcomings apply to programmatic approaches like SOAR and SOC automation since these systems rely heavily on rigid patterns, proving little value for any minor variance from known flows. The 2023 SANS SOC Survey is a good read on some of the issues around SOAR. One of my favorite quotes from this survey is:
“Continual tuning of SOAR by skilled analysts is needed to obtain value—SOAR as a work style, not throwing a switch.” We all know by now how that work style fails to fit into the MO of most organizations.
Organizational burden of systems for expert users
The burden of systems built for the expert users falls upon the newer in role folks and operational cadence/consistency of organizations. When you consider the learning curve that exists within security, coupled with the acceleration of adoption of new SaaS services, it’s impossible for any individual to know every system, and it is increasingly difficult to keep up with the constant change in all systems. In this setting, how are new in the cyber field, new in role cyber analysts expected to learn and grow? During the middle of an incident is surely not the right time, and most organizations don’t have a platform that provides impactful growth paths for them to gain the expertise they need.
When you are responding to an incident or conducting a high stakes investigation, do you know all the questions to ask? Sure, there are always baseline questions like what processes are running, or what new files exist on the system... But what about more complex scenarios? Mimikatz executed on a machine... We know credentials were dumped. Do you know which ones? Who else has logged on to that machine? Where else have they logged on? What do they have access to? Do they have VPN access? Or MFA? Have they changed their password since the last login?
All these questions are common sense for a seasoned investigator, but what about the analyst new in the field? A relatively inexperienced analyst may not fully understand the nuances of Active Directory or identity access because they’ve never administered them?
The next piece of this of course is, how do I even go about finding out that information? In practice, this is challenging even if the data is all collected into a single point, like a SIEM (you are collecting all relevant data right? And active state information like their current MFA Setting?). There are several SIEM queries to extract that information and it’s then on the analyst, and their individual knowledge to make sense of the data received.
The other issue is personnel changes. It’s truly great for Jane’s job security when they’re the only one who knows how to query the data lake, or build queries for the detection team... But what does that do to the team, and the organization when Jane gets another job? The dependencies on a handful of individuals contribute to a variety of issues, from impeding job growth, to risking the ability of the security team to effectively investigate the data.
If by observation or direct evidence an analyst realizes the scope has expanded to a SaaS application such as GitHub, does that analyst have access to the system? Who do they need to contact to get access or help? What is the process to include this new system in the investigation?
By the time they run this down, other things can happen which allows them to lose their place in the investigation or forget the critical investigative context of what they were searching for. In many organizations, investigation steps aren’t tracked item by item, it’s up to the analyst to keep their place, and track their context. This clearly isn't a scalable solution to investigations, and Command Zero set out to solve exactly this problem.
Command Zero’s solution to deliver expertise to every user
We are building an expert platform for all users. Our goal is to enable teams to spend more time investigating impactful cases and less time security engineering.
When we develop integrations and content, it’s aimed at enabling analysts to spend more time in the investigation, reviewing content and establishing context versus running down system ownership, gaining access, figuring out how to craft a query, how to interpret the data. Time is of the essence!
Similar to an EDR or Anti-Virus providing signatures, Command Zero authors expert content for all users. Users are not expected to be experts in every product, and every area. The content is powered by experts in the background, with the goal of teaching and enabling analysts of all levels.
Expert content on the platform is shared across organizations, when new content is created based on user feedback or needs of an investigation, the entire community using the platform benefits. As new integrations and data sources are requested, we work closely with customers to address their use cases and design the integration and questions to maximize value to everyone who uses the platform and has that integration.
The real value of the platform is the expert methodology and method in which we approach building questions based on our understandings of threat actor mindset, forensics expertise, customer use cases and scenario-driven development of content.
How the Command Zero Security Research Team delivers expert content
One of the unique qualities of Command Zero is that it’s a question-based investigation platform. So, distilling our investigative and technology specific expertise into usable questions becomes the fuel for the platform. Expert content (including investigative questions) is created using scenario-based purple teaming in our lab environment. Our lab is a fully functional organization, with each of the security and non-security products integrated into it. We start by determining investigation/attack scenarios and researching them extensively.
Threat intelligence helps identify current and future patterns
We use threat intelligence to drive decisions like attackers carrying out these attacks. We pivot on the attack patterns observed in the wild and other potential methods that can be used to ensure more comprehensive coverage. Just because an attacker is using some of the possible methods today, doesn't mean they will keep using them tomorrow. So, we look for all the different ways this attack can be conducted, exploited, or otherwise used maliciously.
At one point when I was working with a hacker named MC, who was hands down the best exploit developer I’ve met, in reference to disclosed vulnerabilities: "Where one exists, several others are sure to be adjacent."
Thanks to MC, I’ve incorporated this concept into much of our research methodology. So, we look not only at what is known, but what the possible unknowns are as well.
Questions on active state and historical activities
Anytime we are creating content around a new integration source, we approach it by tackling the top-level categories of configuration/ settings, active state questions, and historical queries.
Configuration and settings for each data source are important for many reasons. Certain products have limitations based on licensing, while other times you can’t ask a question if the supporting service is not enabled. We will also create content that deals with changes to them.
Active state questions are important for context during an investigation. These are questions like “What applications are configured for my organization?” or “What Access Roles are assigned to Maria?”.
Historical questions on the other hand, deal with actions by the user, against the org et cetera. Good examples of historical questions are: “What applications consents have been granted in my organization?” or “What access roles were assigned to Maria?”. There’s a small nuance in the wording, but if you look closely into these question types, the active state deals with the current state, whereas historical questions paint the story of changes during the time range of the investigation.
Enriching investigations with advanced LLMs
Building upon the scenarios, we lean on Large Language Models (LLMs) and other semantic models to facilitate new avenues of investigation. Populating the LLMs with threat intelligence, forensic write ups and other content allows it to generate new question sets which may not have been covered under the original investigation scenario. Content researchers are still at the heart of the question creation process, but these methods enhance the brainstorming process while making sure nothing gets overlooked.
The other area where the LLMs come in exceedingly handy is generation of descriptions for questions. Since we want to not only aid in the investigation but also facilitate growth in analysts, each description contains a few key elements. A description of what you are asking, why you are asking this question, and most importantly what the analyst should be reviewing within the response. These descriptions are tedious to write, and this is an area where leveraging LLMs is beneficial.
By creating content in this way and standardizing it against broader stroke questions we ensure the availability of questions across integrations. This also helps organizations with consistency, they can see the available questions and customize facets to ensure they are always executed.
Conclusion
Universal talent gap is a challenge we must operate with in cyber. To combat this, we need to shift to expert platforms that augment all users. Command Zero does just that for cyber investigations.
Expert investigative questions and investigative flows (facets in our terminology) are the accelerators of the Command Zero platform. By leveraging this expert content, all tier-2+ users (tier-2, tier-3, incident responders and threat hunters) can deliver expert outcomes every time. In future posts, I will share some of our expert content and how it can translate into effective investigations for your organization.
