Blog

The Difficulties of Testing Threat Detection

In some respect, we are all new to cybersecurity. As threat vectors change and malware evolves or appears on a daily basis, security vendors continue to provide new and unique detection solutions to find these threats using some of the most cutting-edge technology like artificial intelligence, machine learning, behavioral analytics and more.

Yet, do they work? All this seems great if you plan on accepting the marketing hype. But hype is often greatly different from reality. Many vendors provide detection accuracies and false positive or negative rates, but how do you, as a customer, go about testing or validating these claims?

To start, you will need a large (thousands or more) number of samples of known and unknown malware. Additionally, you will want a good distribution of different types of malware both file-based and fileless threats which should include Windows Executables, DMG, Office, Adobe, Rich Text Format, Image Formats, Android APK and JavaScript.

Where can you get these samples? There are several online malware repositories and feeds available including VirusTotal for known malware. One note here, malware creators or organizations that have threat intelligence teams will likely have these kinds of “known & unknown” threats in their libraries.

Yet, once tested, “unknown” threats become known. So, a proposed solution would be to utilize an existing malware repository to create a test set which best exemplifies the threat profile of the organization and then control the testing environment to create “unknown” test set of malware. For unknown malware, we will discuss this in more detail in my next blog as this is a little more complex; stay tuned!

To understand the complexity and problem even further there is disagreement in the security community on “What is and isn’t malware?” In a test using VirusTotal to download a batch of 1,000 samples, it is likely that 30% or more of the samples will not be considered malware by the majority of VirusTotal’s 70+ antivirus scanners; meaning half of the antivirus engines will not identify the suspicious file as malware and the other half will. This is due to the fact that included in these samples are what is known as “Potentially Unwanted Programs and Applications” (PUPs and PUAs) and these types of suspicious files continue to be debated in the community as malware or not. If PUPs and PUAs are an area you want (or are required) to test then you should raise this with your vendor as many do not consider these to be malware, which will affect your false positive rate.

One recommendation for collecting known malware would be to use vendor consensus to confirm samples that are being collected. Secondly, if you download a sample set from VirusTotal, the majority of the samples will likely be PE32 GUIs. In my example of 1,000 downloaded samples, 90% were PE32 GUIs. Most feeds and repositories contain a range of file types including: Windows Executables, Android APKs, PDFs, Images, JavaScript and other file types. So build a set of samples for each file type.

Feel free to reach out and share your ideas on how to go about creating a test scenario for malware detection and curating known and unknown malware. In my next blog, I will talk about why sample size is important and the currently recognized methods for curating “unknown” samples.


Nick Arraje has over 20 years of technical sales experience working with companies in telecom, network visibility and cybersecurity. While working in the telecom market, he worked with Tier 1 and network equipment manufactures on network simulation tools and testing scenarios to validate product features and capabilities.  Nick has degree in Electrical Engineering from Northeastern University.

Nick Arraje has over 20 years of technical sales experience working with companies in telecom, network visibility and cybersecurity. While working in the telecom market, he worked with Tier 1 and network equipment manufactures on network simulation tools and testing scenarios to validate product features and capabilities. Nick has degree in Electrical Engineering from Northeastern University.


Tags: , , , , ,

Interested in learning about BluVector?Contact Us >