« May 2010 · November 2010 · February 2012 »

July 2010
MonTueWedThuFriSatSun
   01020304
05060708091011
12131415161718
19202122232425
262728293031 
August 2010
MonTueWedThuFriSatSun
      01
02030405060708
09101112131415
16171819202122
23242526272829
3031     
September 2010
MonTueWedThuFriSatSun
  0102030405
06070809101112
13141516171819
20212223242526
27282930   

Calendar:

Newest first Oldest first

Monday, 2 August 2010

Of Opinions and Anti-Virus Testing

There has been a very active discussion recently on the effectiveness of AV testing and the different organizations that conduct that testing. We very much enjoy any active discussion and will not usually shy away from anything that is open and honest.

Of course having anyone speak for us, especially when they are not knowledgeable about us, our methodologies, or our processes, only can exacerbate any issues brought up. So, in the interest of trying to get this discussion a little more open, I thought I would address a few of the questions or concerns that were brought up.

Q: Shadowserver is getting paid to conduct there tests and skew the results

  • Nope. We do get free licenses from all the vendors so that we can conduct our tests, and in return we send back to them anything they do not detect up to 2,000 samples each day.

Q: Shadowserver's testing is absolutely not a reliable antivirus comparison source.

  • Yes and No. We do not display our statistics for the purposes of trying to make one vendor look better than the next. We gathering in malware and test it. Your usage of the results will vary if you understand everything involved with the process. We only use Linux based AV test suites. This limits some of the possible results that we could get. Our tests are more in-line with what a gateway product would be able to accomplish, not an end-point client installation. There are a lot of other considerations and those are on our AV pages here and here.

Q: Shadowserver is using outdated software with unreliable configurations

  • Not really. We try to use the latest stable Linux based products and update the databases hourly for every vendor. If we get behind in a version those vendors are pretty quick to let us know to upgrade which we get to usually within a few weeks. As far as the configurations, we do post exactly what options we use so that you can duplicate any of our efforts, or better yet, tell us what we should be using. We have changed the options in the past from suggestions given to us from the vendors as well as our many constituents. We do not consider ourselves experts at any of this, but by being open and showing exactly how we test can make it clear on the usefulness of our results. And we enjoy getting comments back on what we should be doing better.

Q: Shadowserver have high rates of corrupted samples.

  • I am not sure the source of this. But in testing what we get in to the NIST Lists, we have a less than 0.001% overlap. We also pre-filter anything that is not a recognized binary, dll, or document file. So, operating on corrupted files is something that we are not seeing occur in our process.

Q: My favorite vendor is getting abysmal results in Shadowserver's statistics

  • That could be. What is different from our testing and any of the set tests that you see is that we are getting our data live. We do not have a set number of samples and types that we use against the different engines. We are showing the results from the different daily feeds. This means that what we get in on a specific day will be different from the next. We do not control what comes in. Yes, there are many sources, and each of these produces different types of binaries that get tested. It is possible that what we get in is weighted for certain types of malware, and that is acceptable. Once again, it is the openness of our process that allows you to see why our statistics are valid for our usage, but may not serve some specific need you might have for them.

Q: That's not the only inconsistency you can find.

  • We love controversy, but we also love consistency. And we have striven to make everything as consistent as possible within what we are doing and present. If you believe we missed the boat, please let us know. We would prefer to have what we show to be accurate and useful. If it fails on either count, we need to make corrections. We need everyone to let us know where we have failed, but please be specific and offer suggestions on how to improve.

Q: Linux verses Windows based Testing

  • Yes, we are only doing the Linux based testing. It is more of a resource constraint than anything else. If someone would like to donate a couple of blade servers so we would be able to run two hundred to three hundred virtual machines, we would be happy to add in Windows based AV testing as well.

I am sure there are more questions, and please drop us a note and we'll try to answer them as best as possible. The only thing we ask is that if there is an issue, please address it to us. We really would like to improve what we have for everyone else.

=>Posted August 02, 2010, at 09:21 PM by freed0