Testing Firm NSS Labs Declares War on Antivirus Industry
25.9.2018 securityweek Analysis

Simmering Tensions in the Antivirus World Erupt Again

NSS Labs, a security product testing and validation firm, has effectively declared war on the entire antivirus (AV) industry. On September 18, it filed an antitrust law suit against CrowdStrike, Symantec, ESET, the Anti-Malware Testing Standards Organization (AMTSO), and Does.

The ‘Does’ are described as endpoint protection (EPP) vendors (that is, AV vendors) and members of AMTSO.

AMTSO is a non-profit organization established in 2008 with the stated purpose of improving anti-malware testing. It is open to academics, reviewers, publications, testers and vendors, and its current 51 members include the named defendants, the plaintiff NSS Labs, and most – if not all – of the major EPP vendors.

NSS Labs claims that AMTSO has organized a conspiracy against the EPP product testing industry – and specifically NSS Labs – to prevent independent testing of EPP products. It claims that this conspiracy (the complaint describes the defendants as the ‘EPP Vendor Conspirators’) is enforced by an agreement within AMTSO that only allows testing in accordance with AMTSO’s published testing protocol (PDF).

The effect is that if testing procedures are not considered to be in conformance with the guidelines, the AMTSO members will not use that testing company. This removes a major part of NSS Labs income generated through paid EPP tests and sold EPP test reports.

NSS Labs’ group tests, with no charge to the vendors, are then further disrupted by the inclusion of ‘no testing without agreement’ provisions within the EPP product end-user license agreements.

“They [AMTSO] claim to try to improve testing but what they’re actually doing is actively preventing unbiased testing,” claims Vikram Phatak, CEO of NSS Labs, in an associated blog post. “Further, vendors are openly exerting control and collectively boycotting testing organizations that don’t comply with their AMTSO standards – even going so far as to block the independent purchase and testing of their products.”

The Complaint

The complaint alleges that AMTSO and its members have conspired to ensure that product testing is effectively controlled by the vendors being tested. Specifically, it alleges, “that CrowdStrike, Symantec and ESET conspired with each other and the other EPP Vendor Conspirators to license their products under terms of use or end user license restrictions that purport to prevent competitive or comparative testing of their products, and purport to prohibit their customers from allowing their copies or ‘instances’ of EPP products to be used for competitive or comparative testing.”

NSS Labs is seeking a jury trial, and damages and costs.

Background

AMTSO was founded because anti-virus testing is profoundly difficult. With multiple testing agencies testing different products by different methodologies, the potential of introducing innocent bias is high. And with no external certification required for testing, the potential for fundamentally flawed methodologies is ever present.

AMTSO’s intent was to develop a set of testing standards that would eliminate bias and deliver comparable test results regardless of the products tested or the testing companies involved, provided they use the same testing standards. It believes this is of benefit to both anti-virus users and to anti-virus vendors.

Its difficulty is that this only works if the AMTSO testing standard is used. Any attempt to enforce or impose its use can be verbally interpreted as a conspiracy to force its use – and NSS Labs seems intent on testing whether it is legally a conspiracy under the Sherman Antitrust Act and the Cartwright Act.

AMTSO is no stranger to such accusations. In recent years a fresh generation of EPP vendors (generally known as second-gen AV) relying primarily on machine learning algorithms to detect malware – rather than the malware blacklists originally used by the early AV vendors – has challenged the market’s status quo.

These new vendors have been at times aggressive in their marketing, claiming to block malware that 1st gen products could not detect. They found that AMTSO’s testing standards – at that time – could not compare 1st gen and 2nd gen products, and sometimes resorted to their own testing approaches.

While bad feeling between the two parties was strong, nevertheless AMTSO found a way to bring many of them onboard to develop new standards that would be fair to all parties. This process involved both stick and carrot. The stick came in Virus Total’s own suggestion that it would restrict access to its malware database for vendors and testers who do not sign up to AMTSO.

(NSS Labs claims this was part of the conspiracy. The complaint alleges, “The AMTSO EPP vendor members and AMTSO itself agreed, among other things, that access [to VirusTotal] should only be available to EPP vendors who are AMTSO members and whose products are only tested by EPP testing services who are also AMTSO members. In addition, both the EPP vendors and the EPP Testing services would be required to have agreed to adhere to AMTSO’s ‘Fundamental Principles of Testing’…” But AMTSO told SecurityWeek at the time that the initial suggestion came from VirusTotal, which was increasingly concerned that 1st AV vendors would desert VirusTotal.)

The carrot was that in joining AMTSO, the 2nd gen vendors would get a seat at the table able to influence new standards that would cater for both approaches to malware detection. This is precisely what has happened, with many of the 2nd generation EPP vendors having joined AMTSO. The implication is that AMTSO itself prefers collaboration to controversy.

NSS Labs and the EPP industry

Just as AMTSO is no stranger to controversy, nor is NSS Labs. In February 2017, CrowdStrike sued NSS Labs to prevent the publication of its product test results following an NSS group test. The lawsuit failed to prevent publication, but CrowdStrike blogged at the time, “Taken in total, NSS’ failure to conduct the most basic of fact checking during the private testing and the well-publicized history of problems with NSS testing ultimately gave us no confidence that NSS Labs could conduct accurate testing of our security products. Therefore, we declined to participate in the public test.”

Similarly, Tony Anscombe, ESET global security evangelist, blogged on April 17, 2018: “When or if you read the NSS Labs test results document, we hope you find it belongs in the circular grey filing cabinet under your desk, the same place I put my copy of the report.” Earlier in the blog he had said, “In the test results published in 2017, we experienced numerous issues and NSS Labs failed to publicly correct all the inaccuracies despite their agreement to remedy them at a meeting in April 2017.”

Plaintiffs’ response

SecurityWeek approached a number of the EPP vendor plaintiffs for their views on the complaint. All except CrowdStrike declined to comment because of the sensitivity of the issue. CrowdStrike sent the following statement:

“NSS is a for-profit, pay-to-play testing organization that obtains products through fraudulent means and is desperate to defend its business model from open and transparent testing. We believe their lawsuit is baseless.

“CrowdStrike supports independent and standards-based testing—including public testing—for our products and for the industry. We have undergone independent testing with AV-Comparatives, SE Labs, and MITRE and you can find information on that testing [online]. We applaud AMTSO’s efforts to promote clear, consistent, and transparent testing standards.”

AMTSO also responded. In an emailed statement, it registered disappointment in NSS, and categorically denied all claims against it. “AMTSO was founded in 2008 as an international non-profit association that focuses on addressing the global need for improvement in the objectivity, quality and relevance of security testing methodologies. Our membership is 50+ security vendors and testers. AMTSO provides a forum to discuss, engage, and communicate practices that will advance ethical, transparent and standards-based security testing.”

The statement points out that NSS is a member of AMTSO, and that one of its employees was a member of the working group that developed the standard. “Rather than trying to use the legal system to tear down what we all built together, we encourage NSS to bring its concerns back to the table and engage with the rest of AMTSO membership to make our industry better.”

The NSS response to this is likely to be similar to its Complaint: “While providers of EPP testing services, including NSS Labs, are allowed to and do participate in AMTSO, they constitute a small minority of AMTSO members and are easily outvoted by EPP product vendor members as indeed they were in the adoption of the AMTSO Testing Standard.”

Looking forward

It’s difficult to see the path forward. If the complaint reaches trial, it will take the legal system to decide whether a conspiracy exists. If NSS prevails, it is equally difficult to predict AMTSO’s future – it will be denied its very purpose. It will be able to continue developing testing standards, but will find it impossible to ensure they are used.

AMTSO’s problem is that on the surface it looks like a conspiracy and acts like a conspiracy even if it is not a conspiracy. Again, if the matter goes to trial, AMTSO will likely need to prove the necessity for what it does. The probable route would be to denigrate NSS Labs’ non-AMTSO testing – and frankly it appears that numerous vendors will be willing to testify on that.

This complaint is going for broke. If NSS succeeds, it will have few friends in the EPP industry. It may be able to buy EPP products and test them privately, but revenue will be dependent on corporations buying the reports. It will likely get little cooperation from the vendors who have spent a decade in developing the AMTSO standards.

If AMTSO prevails, NSS will either lose the EPP side of its market – or will eat humble pie and adopt the AMTSO standards. There are no winners here.

The best outcome would be an out of court agreement preventing the case going to trial. While AMTSO’s emailed statement appears to offer that possibility, a separate blog post by AMTSO President Dennis Batchelder makes no mention of working together in the future. Instead it simply refutes the NSS claims.

“Our testing standard holds both testers and vendors accountable to ethical and fair practices, including ensuring that competitive tests are fair to all participants,” he writes. “It does not tolerate backroom deals, “fitted” results, or offering private, pay-to-play, undisclosed advantages to vendors who happen to pay more than others. This change is critically important to the broader cybersecurity community, including testers, vendors, and most importantly customers.”