MITRE CALDERA® Case Study

We accomplished a case study of our framework leveraging MITRE CALDERA®.This analysis has the objective of assessing the effectivness of the framework and the tool.

The data indicates that the tool's performance differs greatly based on the category. It has an excellent result (

90%

) in the "Organization" category, although this outcome is partially biased since this category was developed taking as reference MITRE ATT&CK, and Caldera leverages it too. The tool performs well in the Attack and Simulation categories (

69%

and

71%

), reflecting the software's satisfactory capabilities in helping red team operations. More than half of the features presented in "Architecture" are available, whereas the categories "Other" and "Enviroment" reveal results slightly below the passing mark.
Finally, the tool has a poor score on "Results" and "Information and Filtering" categories (respectively

28%

,

31%

). The unsatisfactory performance on the "Results" category was expected; this solution was born to help the red team's operations and is considered an Adversary Emulation Platform. For these reasons, its primary focus lies on the attack, while the support offered in analyzing the results and providing mitigations lags behind commercial BAS solutions. The sub- optimal outcome in the Information and Filtering category is because the tool's filtering is based on MITRE ATT&CK's, which permits filtering based on tactic, technique and platform but leaves out other valuable details like the APT correlation or the last update time. Furthermore, the reports are available in PDF, JSON, and CSV formats, but there are limited capabilities for analysing the results directly from the platform.

Overall, the tool offered 63 of the 145 analysed features, which can be considered a decedent result considering the intrinsic limitations of applying a Framework designed for BAS solutions to an Ad- versary Emulation software. The tool’s performance was notably affected by lower evaluations in categories that distinguish AE Tools from BAS Tools, which significantly impacted the final results.

Disclaimer:
The performance analysis and results presented herein are based on the use of CALDERA®, developed by MITRE Corporation. While this analysis has been conducted with diligence and objectivity, we do not assume any responsibility for the accuracy, reliability, or completeness of the tool's performance as evaluated. The findings and opinions expressed are solely those of the authors and are intended for informational and academic purposes only. We disclaim any liability for actions taken based on the results of this analysis. Users should perform their own assessments and consult the original developers for any issues, updates, or further information regarding the tool. Our analysis does not constitute an endorsement or criticism of the tool's capabilities or performance.