AV Testing

Testing anti-virus products is not just a case of finding a few files and scanning them to see if they contain viruses. Testing must be done in a rigorous, methodical manner to ensure that a correct result is achieved. This guide aims to point out the most important criteria for testing, and give some guidelines for testing the various different types of anti-virus scanner. As there are more and more inaccurate test results presented to the public, we are aware of the issue and would like to respond the following tests:

for http://www.virus.gr

For more general information on AV testing, please note the following:

Virus sample collections
The virus samples used in a test should be widely accepted by the AV industry. The wildlist (http://www.wildlist.org) is a premier source of information on which viruses are spreading In the Wild. The list is used by Virus Bulletin (http://www.virusbtn.com) and other anti-virus product testers as the definitive guide to the viruses found in the real world. It is not rare, however, to find that some test results are based on a “personal collection of virus samples”. Such tests produce misleading results which are hardly recognized by AV professionals.

False positives
It should be clear that the more genuine viruses that a product detects the better it is. More importantly, is that the product does not give false positives. That is, the product does not flag a file as infected while that file is, indeed, clean. It is surprising to see that some tests claim a file is infected, simply because one of the products says so. Certainly, it is unfair to blame a product if the product does not give a false positive. In some tests, ESET NOD32 can be ranked quite low. While looking at these results, we should take into account that ESET NOD32 would “fail to give false positives”.

Some basic guidelines for anti-virus scanner testing

1. Files should have their normal extensions, and not have renamed extensions.
   ESET NOD32 has a very sophisticated scan engine, optimised for real world usage, rather than “academic” testing. This means that it will perform best when used as a user would use it – in scanning a system for real, live viruses. Scanning a file that does not have its normal extension – for example, the file myscreensaver.exe renamed to myscreensaver.ex$ – this will give unpredictable and often false results. The .ex$ extension is not executable in a normal system, and so ESET NOD32 may, legitimately, ignore this file as being of low threat. Of course, with the correct extension, a real virus would be recognised as a threat to the system.
2. Files should be on a local hard drive. Unless testing scanning on network drives specifically, the virus files should be on a local hard drive. If not, then figures for scan speed and system impact will be incorrect. ESET NOD32 is extremely fast and has a minimal impact on system performance while scanning; this speed will not be measured correctly across a network, because the speed is limited by the network connection.
3. Settings should be checked. Some tests are done with “default” settings, and some with “best” settings. Please contact a member of the Eset team to determine which settings should be used in a non-defaults test.
4. Viruses should be real, unaltered samples that can be verified as viral through replication. See sample selection section later in this document.
5. Damaged, intended, or non virus files should not be tested against, including any so called virus simulator files – these are not viruses, and should not be tested for. Eset prides itself on only detecting real viruses, and has no wish to play the game of detecting non-viruses, just to pass tests.
6. Viruses should be unaltered from their normal In the Wild state, and should be already existent viruses. Eset will not be party to the creation of new viruses or the alteration of viruses to create new variants for any purpose, including the purposes of testing. Altering any virus (presuming that it is replicable afterwards) effectively creates a new virus. Therefore we consider such practice to be unethical and unprofessional and we will not participate in tests that are based, wholly or in part, upon altered or specially created viruses.
7. Clean files used in a false positive test should be normal files that would exist on an end-user’s system, rather than files that have been specially created to look like viruses or “trick” anti-virus scanners.
8. Statistical integrity. Sample selection should be made on a statistically sound basis, the number of samples used in a test is important. Testing against two or three samples, or even 10 or 15 samples does not constitute a statistically significant set of samples. The Wildlist contains 250 – 270 different viruses on its top list (the most significant list) of currently In the Wild (ItW) viruses. The smaller the set tested against, the more statistical error there will be in the test.
9. Tests against “zoo” viruses should be separate from tests against ItW viruses. Zoo viruses are viruses that do not and have not historically appeared on the Wildlist. Therefore, there is a very small chance that these viruses will ever appear on a normal user’s system). This means that there is little significance to detection of these samples, and statistically, detection of these is far less important than the detection of ItW viruses.
10. Any virus that a product misses, or fails to detect correctly, or any clean file that a scanner mistakenly detects should be made available to the product manufacture for verification, and if necessary, rectification.

Conclusion
ESET NOD32 is tested by several well respected organizations such as Virus Bulletin, ICSA and West Coast Labs, and continues to achieve the best results possible in these tests. These testers are recognized to have carried out scientifically based tests against verified virus samples, and their results are respected throughout the industry.