Thorough, independent tests are vital resources for cybersecurity leaders and their teams when evaluating vendors’ abilities to guard against increasingly sophisticated threats. One of the most trusted assessments is the annual MITRE Engenuity ATT&CK Evaluations: Enterprise.

This evaluation is crucial because it’s nearly impossible to judge cybersecurity vendors based solely on their own performance claims. Along with vendor reference checks and proof of value evaluations (POV) — live trials in their environment — the MITRE Engenuity results provide additional objective input to comprehensively assess cybersecurity vendors.

Let’s explore the 2023 results. In this article, we’ll delve into MITRE Engenuity’s methodology for testing security vendors against real-world threats, offer our interpretation of the results, and highlight key takeaways from Intelecis’ evaluation.

How does MITRE Engenuity test vendors during the evaluation?

The MITRE Engenuity ATT&CK Evaluation tests endpoint protection solutions against simulated attack sequences based on real-life approaches taken by well-known advanced persistent threat (APT) groups. For the 2023 evaluation, 29 vendor solutions were tested by emulating the attack sequences of Turla, a sophisticated Russia-based threat group known to have infected victims in over 45 countries.

Importantly, MITRE does not rank or score vendor results. Instead, the raw test data is published along with basic online comparison tools. Buyers use this data to evaluate vendors based on their organization’s unique priorities and needs. The participating vendors’ interpretations of the results are just that — their interpretations.

So, how do you interpret the results?

Interpreting the MITRE Engenuity ATT&CK Evaluations: Enterprise results can be challenging because they aren’t presented in a familiar format like a quadrant graph. Independent researchers often declare “winners” to simplify the process of identifying top performers. However, identifying the “best” vendor is subjective and depends on specific organizational needs.

With this in mind, let’s review the results to compare how participating vendors performed against Turla.

MITRE Engenuity ATT&CK Results Summary

The following tables present Intelecis’ analysis and calculation of all vendor MITRE Engenuity ATT&CK Evaluations: Enterprise test results for the most important measurements: Overall Visibility, Detection Accuracy, and Overall Performance. These metrics are considered most indicative of a solution’s ability to detect threats.

How did Intelecis perform?

Based on Intelecis’ analysis, our team is proud of our performance against Turla in this year’s MITRE Engenuity ATT&CK Evaluations: Enterprise, outperforming the majority of vendors in several key areas. Here are our top takeaways:

– Intelecis delivered 100% Detection (19 of 19 attack steps) with NO CONFIGURATION CHANGES
– Intelecis delivered 100% Visibility (143 of 143 attack sub-steps) with NO CONFIGURATION CHANGES
– Intelecis delivered 100% Analytic Coverage (143 of 143 detections) with NO CONFIGURATION CHANGES
– Intelecis delivered 100% Real-time Detections (0 Delays across all 143 detections)

Detailed Analysis

Intelecis was a top performer in both visibility and detection quality. This analysis illustrates how well a solution detects threats and provides the necessary context to make detections actionable. Missed detections invite breaches, while poor-quality detections create unnecessary work for security analysts or potentially cause alerts to be ignored, again inviting breaches.

Intelecis delivered 100% visibility and perfectly detected every one of the 143 attack steps using no configuration changes. The following chart shows the percentage of detections across all 143 attack sub-steps before vendors implemented configuration changes. Intelecis performed as well as two very large, well-known security companies, despite being a fraction of their size, and far better than some of the biggest names in cybersecurity.

Intelecis provided analytic coverage for 100% of the 143 attack steps using no configuration changes. The following chart shows the percentage of detections that contained important general, tactic, or technique information across the 143 attack sub-steps, again before configuration changes were implemented. Intelecis performed as well as Palo Alto Networks, a $76 billion publicly traded company with 50 times the number of employees, and far better than many established, publicly traded brands.

Overall Visibility is the total number of attack steps detected across all 143 sub-steps. Intelecis defines Detection Quality as the percentage of attack sub-steps that included “Analytic Detections” — those that identify the tactic (why an activity may be happening) or technique (both why and how the technique is happening).

Additionally, it’s important to consider how each solution performed before vendors adjusted configuration settings due to missed threats. MITRE allows vendors to reconfigure their systems to attempt to detect threats they missed or to improve the information they supply for detection. In the real world, we don’t have the luxury of reconfiguring our systems due to missed or poor detection, so the more realistic measure is detections before configuration changes are implemented.