Earlier this month, Consumer Reports touched off a firestorm of controversy when it downgraded four of Microsoft's laptops from "recommended" and predicted that a quarter of the devices will incur issues within two years. The four devices:
- Microsoft Surface Laptop, 128GB and 256GB versions
- Microsoft Surface Book, 128GB and 512GB versions
Consumer Reports said it could not recommend the Microsoft laptops or tablets because of poor reliability compared to other brands.
Microsoft quickly defended the reliability of its laptops and tablets in a statement to Consumer Reports.
"Microsoft's real-world return and support rates for past models differ significantly from Consumer Reports' breakage predictability," Microsoft said. Consumer Reports' findings do not "reflect Surface owners' true experiences or capture the performance and reliability improvements made with every Surface generation."
Panos Panay, corporate vice president of Microsoft Devices, said in a blog he was "disappointed" in Consumer Reports' decision to downgrade the devices and included some survey results.
Surface "ranks highly in customer satisfaction," said Panay. Of users surveyed, "98% of Surface Pro 4 users and Surface Book users say they are satisfied with their device, and our Surface Laptop and new Surface Pro continue to get rave reviews."
So, who's right and who's wrong? There is a clear discrepancy between the two reports and it turns out, it may simply be a matter of who was asked and when.
The evolution of a product
Who was asked? Consumer Reports determines quality through testing and surveys. The surveys are self-selecting and largely made up Consumer Reports subscribers.
"This means they don't even accurately represent the subscriber base, let alone the full population of product users," said Rob Enderle, president of the Enderle Group.
The other issue is timing. Consumer Reports says it based its decision on feedback from subscribers who bought 90,741 new tablets and laptops between 2014 and the start of 2017. That's a pretty broad timeframe, especially considering the Surface products have evolved substantially since the devices' first release.
Last week, technology journalist and blogger Paul Thurrott obtained a leaked Microsoft internal memo that revealed some problems in the original Surface Book released in 2015. The memo showed that the original Surface Book saw a 17% return rate. The Surface Pro 4 that launched at the same time also saw a 16% return rate after launch.
Microsoft executives believe those return rates significantly impacted Consumer Reports' findings, according to the report.
Consumer Reports tends to "report on things about one to two years back, so it isn't uncommon to have the reports point to products that haven't been sold new for a year or more," said Enderle. "Worse, they can highlight and blacklist a product or vendor for a problem that has already been fixed."
While Microsoft did have some initial issues with early Surface hardware and software, the company also addressed those problems. The return rates for both products quickly dropped below 10% just two months later. As of April 2017, with the exception of the Surface Book, return rates are all under 4%.
It appears Microsoft is now paying the price for problems it has fixed, causing Panay to question how the survey was conducted.
"We track other indicators of quality such as incidents per unit (IPU), which have improved from generation to generation and are now at record lows of well below 1%," he said in his blog post.
Complaints of the past
Consumer Reports' testing methods often come under scrutiny. Last year, the nonprofit was criticized when it refused to recommend Apple's new MacBook Pros. The company said testing indicated the products had inconsistent battery life.
After Apple pushed Consumer Reports for more information about its testing processes, Apple realized a bug had caused the poor battery results. Following a software update, Consumer Reports ultimately recommended the MacBook Pro.
Both cases illustrate how old or incomplete information can damage a brand. Though Microsoft addressed the issues, the lag between the survey and the problem means Microsoft still took a hit. The report was a knock on Microsoft's device reputation, which is especially damaging since its hardware sales have already depreciated in the last year.
"In the end this both points to a problem with Consumer Reports methodology and Surface products in the 2015 timeframe," said Enderle. "Of the two, the Consumer Reports issue is actually the bigger problem because it isn't helpful to be steered away from a vendor for a problem that no longer exists. You want to avoid problems that exist during your buying cycle.
"Every vendor needs to have high quality and that includes those that report on the quality of others," he said. "And problem reporting needs to be both accurate and timely. It does you no good to find out you shouldn't have bought something a year after you bought it."
The key now is how Microsoft handles the situation.
"In terms of how you deal with a problem like this, what not to do is attack the individual or organization reporting the problem," said Enderle.
"The right response is to listen, disagree and then address the problem, not make the dispute so visible people become afraid to buy the product," he said. "The best response is to focus on improving customer care and talking about the improvements that were and are going to be made."
