R-STUDIO Review on TopTenReviews: Conflict or Confusion?

We have been developing R-STUDIO for over 15 years, and to this day, we maintain a rigorous development and release schedule. Each major public release is preceded by countless hours in the laboratory, testing the features and robustness of our intermediate builds. To ensure we are putting out the best product on the market, our testing also includes analyses of our competitors' products. We compare our performance to our competitors and pay close attention to any features that are not currently available in R-STUDIO but may be useful for our users.

This is healthy competition in action.

We receive a lot of positive feedback from our end users. We also get top ratings in the data recovery software charts-for years, we remained in the top three at TopTenReviews.com.

All of a sudden in the end of January 2015, our rating at TopTenReviews drastically changed. When we checked the charts earlier this year, we saw that R-STUDIO had gone from being #2 on TopTenReviews' data recovery software comparison to #8. The new top three products were Data Rescue PC3, Ontrack EasyRecovery, and Recover My Files.

TopTenReview_DataRecovery_01.png
Click image to enlarge
TopTenReview: Data Recovery Software Review, February 2015

We took a closer look at the review and realized that there was something wrong. There were technical errors in the comparison itself, and several factual misstatements about R-STUDIO as well. Here are a few of the troublesome statements TopTenReviews was making:

  • R-STUDIO does not have a boot utility. It does - it's called R-STUDIO Emergency and it has been available for years.
  • R-STUDIO does not have a "pause" button. It does, except it's labeled "stop."
  • R-STUDIO does not offer a lab recovery service. True, but irrelevant to the performance of the software-plus, even those companies that do offer lab recovery services don't include this in the cost of the license. It's misleading to list this as a software feature.
  • Data Rescue PC3 outperforms R-STUDIO when recovering "damaged files." This criteria includes recovering "overwritten files." It is technically impossible to recover overwritten files. There are utilities that can open a corrupted file of a known type (such as an Excel .xlsx spreadsheet or an Outlook .pst file) and reconstruct it in order to remove partial data from it. But none of the utilities in this comparison include that feature.
  • The comparison rated each utility's ability to recover "formatted files." You can format a partition or a disk, but you cannot format a file. Surely TopTenReviews meant that the disk was formatted, but how was it formatted? Was it a quick format or a full format? This is an important distinction.
  • Data Rescue PC3 was 4-5x faster than R-STUDIO. We were interested in this metric. We wanted to know about the test that TopTenReviews performed and how they got their numbers. That is, how were these tests benchmarked?

All told, R-STUDIO ended up taking a huge hit in the Scanning Features category (previously called Recovery Features), suddenly dropping from a 10 to a 5.

TopTenReview_DataRecovery_02.png
Click image to enlarge
TopTenReview: Data Recovery Software Review, February 2015, Scanning Features

In the speed category, R-STUDIO got a 7.18.

TopTenReview_DataRecovery_03.png
Click image to enlarge
TopTenReview: Data Recovery Software Review, February 2015, Scan Speed

We wanted to understand how we could have been beaten by our competitors and we did our best to duplicate TopTenReviews' results in our own test lab. We took the top three utilities to our lab and subjected them to the same performance tests and benchmarking that we use when developing and improving our own software. While we recognize that some aspects of software can be subjective - for instance, price and user interface-factors such as data recovery speed, accuracy, and features are not. If TopTenReviews' new 2015 rating was accurate, then we should see the top three products drastically outperforming R-STUDIO in these areas.

In these areas, R-STUDIO far outperformed the top three products on TopTenReviews in almost all of our tests.

We sent our findings to TopTenReviews.

TopTenReviews Responds to Our Findings

From January 2015 to March 2015, we corresponded with various staff and managers at TopTenReviews. Our issue was first fielded by Sharyl Minter-DeJesus in Product Submissions, and we were eventually escalated to Reviews Manager Jacob Rogers and finally Director of Editorial Operations Lecia Monson. TopTenReviews responded quickly to our emails, and seemed to be taking our concerns very seriously. But the actual action they took was underwhelming.

TopTenReviews quickly conceded the oversight about the boot utility feature and agreed to take another look at the review. We waited several weeks and nothing changed on the published review.

Finally, by March 20, 2015 TopTenReviews made some minor corrections. They gave us our boot utility check mark on the comparison chart and they reworded one of the tooltips to specify that "reformatted files" referred to files recovered from a reformatted drive (but still did not specify quick format or full format).

Regarding our questions about how they were testing damaged file recovery and overall speed, they said:
"We consider further details of testing processes to be proprietary information."

This is not only evasive, but also contrary to the principles of hardware and software testing.

When it comes to software and hardware, the benchmarks matter. Take TomsHardware.com, a sister company to TopTenReviews. For each piece of hardware they rate, they give a detailed explanation of how they got their performance ratings. They give it to you in a way that lets you re-perform the test on your own. This is how tests work in the scientific community as well. This is what Consumer Reports does. This is what peer-reviewed journals require. This is the standard of testing.

So, how come TopTenReviews does reviews differently? Why is their staff so responsive and willing to listen to our concerns, but so unable to alter their ratings and reviews, even after we've identified several inaccuracies?

These are the questions we continue to grapple with. We've dug as deep as we can on the technical side. This clearly isn't a matter of R-STUDIO falling short as a data recovery utility. We did not suddenly become a second rate software developer overnight. And we suspect that the other programs didn't suddenly become miracle tools overnight either. We think something else might be afoot.

Independent Testing by DigiLabs

We get the sense that the staff at TopTenReviews want to see us satisfied. But it seems like something is holding them back. It's something that is irrelevant to the technical performance of our software. It has to be - the numbers don't lie.

We wanted to back up our own technical claims with testing performed by an independent laboratory. We wanted someone to do the kind of scientific, methodical testing that we hoped TopTenReviews was doing, but could never get a straight answer on due to their testing methods being "proprietary."

For our independent review, we wanted someone experienced in the data recovery field. TopTenReviews prides itself on selecting expert reviewers with relevant technical experience, and we agree with that approach. If a reviewer is going to be recommending data recovery software products, he or she should know what it's like to use them in real life scenarios-in the field, for a business, or at home.

The reviewer who authored the original data recovery software comparison for TopTenReviews was Chris Snoke. Mr. Snoke appears to have several years of experience writing about software and a good grasp of the technical aspects that matter most. Surprisingly, Mr. Snoke was also on the byline for the February 2015 update, where Data Rescue PC3 was placed at #1 and R-STUDIO was dropped to #8. But Mr. Snoke appears to have been reassigned after the February 2015 review. As of October 2015, there is a new reviewer: Jeph Preece. According to his LinkedIn, Mr. Preece appears to be a gifted fiction writer, poet, and musician, but lacks the same level of technical background as Mr. Snoke.

We think the background and expertise of a software reviewer matters.

That's why we reached out to DigiLab Inc., an Illinois-based technology company specializing in research and development in data recovery, data migration, and digital forensics. The three principals - Andrey Petrov, Artem Rubtsov, and Oleksandr Shashkov - are experienced and well-known in the data recovery field. DigiLab Inc. took the top three products according to TopTenReviews and compared them to R-STUDIO. In a later update to the review, Stellar Phoenix Windows Data Recovery Professional 6.1 entered the top three, so DigiLab Inc. compared that utility as well.

DigiLab performed a rigorous battery of tests with fastidious documentation of their hardware test environments, test criteria, and testing methodology. They broke down their ratings into two major categories: data recovery performance and features.

You can read the full Data Recovery Software Comparative Analysis Report by DigiLab, and if you are so inclined, repeat their testing and methodology exactly in order to verify their independent findings.

The results confirmed what we found in our lab.

R-STUDIO outperformed all four competitors in both categories. But more interestingly, DataRescue PC3 unequivocally underperformed in both categories compared to R-STUDIO, Ontrack EasyRecovery, Stellar Phoenix, and Recover My Files.

That R-STUDIO was superior didn't surprise us. The fact that TopTenReviews' #1 product was actually the worst did.

It seems that DataRescue PC3 is holding the #1 position for reasons other than its data recovery performance and features. What gives?

We think we have an idea of what is going on here.

What's Different about TopTenReviews?

Founded in 2003, TopTenReviews gained a reputation as an authoritative, objective resource in the software and hardware reviews sector. They did this by enlisting trusted experts to do their reviews and by providing thorough, in-depth, and accurate analysis of all the top products on the market. This is the website and reputation that people think of when they see the TopTenReviews logo. And until roughly 2013, this is what the website was.

In 2013, changes started to happen. The two principals that had made TopTenReviews a household name moved on to other endeavors at precisely the same time. Jerry Ropelato became CEO of WhiteClouds, a startup in the 3D printing industry. That same year, Stan Bassett moved on to Alliance Health Networks. Then, in 2014, the parent company of TopTenReviews rebranded itself from TechMediaNetwork to Purch.

Purch has quite a different business model than TopTenReviews did when it first started. Purch, which boasted revenues of $100 million in 2014, has been steadily acquiring numerous review sites over the years, including TopTenReviews, Tom's Guide, Tom's Hardware, Laptop magazine, HowtoGeek, MobileNations, Anandtech, WonderHowTo and many, many more. According to a Fortune write-up of Purch's latest round of VC funding:

The company earns money from affiliate fees on sales of products it reviews; a big source of revenue is referrals from desktop antivirus software. The company also makes money on an advertising network it owns. One source said Purch's revenue is growing between 10% and 15% per year.

There's one other company that Purch owns that may be of interest to anyone visiting TopTenReviews.com: Herman Street. You might recognize Herman Street from the various ads that accompany the reviews on TopTenReviews, particularly the data recovery software review.

In fact, if you click on any of the links to purchase the #1 rated software, DataRescue PC3, it brings you to the product page on HermanStreet.com.

TopTenReview_DataRecovery_04.png
Click image to enlarge

TopTenReview_DataRecovery_05.png
Click image to enlarge

TopTenReview_DataRecovery_06.png
Click image to enlarge
TopTenReview and Herman Street: Data Rescue PC

This seemed interesting to us. We took a look at the archived versions of the review, back when Stellar Phoenix was the #1 product, and we noticed that the link to buy Stellar Phoenix also went to HermanStreet.com.

TopTenReview_DataRecovery_07.png
Click image to enlarge
TopTenReview and Herman Street: Stellar Phoenix

In the updated 2015 review, where Stellar Phoenix is no longer #1, the link to buy the product goes directly to Stellar Phoenix's website and no longer goes to the HermanStreet.com product page.

So, from about 2012 to 2015, Stellar Phoenix was #1 on TopTenReviews. This time period coincides with the time when Stellar Phoenix was being promoted by Herman Street.

Now, DataRescue PC3 is being promoted by Herman Street at a deep discount. And now, DataRescue PC3 is the #1 rated product.

We are not suggesting that any of the very courteous, very professional people we contacted at TopTenReviews are engaged in anything improper. The willingness of these individuals to try to help us in any way they could is proof that they were exercising good faith. But is it possible that their ability to correct the underrating of our product in comparison to DataRescue PC3 might have been limited by their parent company?

This relationship is not transparent to the casual web reader. People visit websites like TopTenReviews because they want fast answers from an authoritative source. They don't want to have to engage in investigative journalism. They don't want to spend hours fact-checking and verifying. And they shouldn't have to. When they visit TopTenReviews, they assume that they are visiting the same website that they've known and trusted for over a decade. They don't know about the relationship between TopTenReviews, Purch, and Herman Street.

Update:

It's worth noting that at the time of our dispute on January 20, 2015 R-Studio was #3 on a very similar comparison published under "Disk Recovery Software".

TopTenReview_DataRecovery_08.png
Click image to enlarge
TopTenReview: Disk Recovery Software Review, January 20, 2015

As a result of our correspondence, TopTenReviews said they would accelerate their update to this page and take our feedback into consideration. When the update rolled out in May 2015, unsurprisingly, Data Rescue PC3 was shifted to #1 and R-STUDIO was dropped from #3 to #5. This update was authored by Jeph Preece, the musician and fiction writing MFA.

TopTenReview_DataRecovery_09.png
Click image to enlarge
TopTenReview: Disk Recovery Software Review, May 2015

Recently, the Data Recovery Software Review comparison was updated to be identical to the Disk Recovery Software Review page:

TopTenReview_DataRecovery_10.png
Click image to enlarge
TopTenReview: Recent Data Recovery Software Review

So, we went from being #2 for Data Recovery Software and #3 on Disk Recovery Software to being #8 for most of 2015 before settling on #5 for both Data Recovery Software and Disk Recovery Software.

We consider #5 better than #8, but our questions about how TopTenReviews generates their ratings and how they select their #1 product remain. According to our testing and DigiLab's testing, R-STUDIO far outperforms all the top three products. We still consider the comparison chart inaccurate.

Conclusion

R-Tools Technology, Inc. is an independent company of software developers. That's all we do. It's all we know how to do and it's all we want to do. We got to where we are today by developing the best software we can and facing the facts when competitors give us a run for our money. When the competition heats up, we hit the lab. We build, we test, we improve. Being one of the best at this has allowed us to become highly respected in the field of data recovery. Frankly, building better and better software is the only trick we have up our sleeve. It frustrates and saddens us that this strategy might not be good enough anymore.

We honestly don't know what to do from here.

According to their website, Purch has massive reach and influence over what people buy. They have 100 million unique monthly visitors to their online properties and they drive over $1 billion in commerce transactions every year. They have acquired many of the brands that readers have grown to trust over the years, and they own advertising channels and storefronts as well.

This is what Purch says about its brands:

Purch provides shoppers with relevant information on today's purchases and tomorrow's innovations with hundreds of daily articles from respected product reviewers, journalists, and community members. A deep understanding of current trends, consumer needs, and intent ensures that every piece of content on our sites provides great value to marketers and their audiences.

We have a question for software developers like us. It's the same question we pose to readers and reviewers and consumers shopping for software or electronics.

Is this what you've experienced from Purch's brands in recent years?

If things have truly changed for TopTenReviews and the other companies Purch has acquired, the impacts are widespread. If you haven't checked your own product's rankings lately, then you might not even know that you've been affected. If you've made a purchase based on information from TopTenReviews recently, you might not know what other factors are influencing the rankings.

We are inviting you to take a look at the information from TopTenReviews and compare it to the full report of the same products from DigiLab and decide for yourself. Does something seem strange to you? Please fill out our feedback form with your opinions and comments and help us set the facts straight. We also encourage you to refer to this article or even republish it. We are small compared to Purch, but perhaps we can still influence their marketing strategy in a positive way. Maybe we can show them that readers truly want the transparent, professional, and objective information that their mission statement promises.

© Copyright 2000-2017 R-Tools Technology Inc.