How were the Best Channel Vendors survey and special report generated?
For the eighth consecutive year, we relied on the wisdom and experience of Penn State University to help ensure the survey’s statistical accuracy. Back in 2009, Penn State agreed with our philosophy of conducting a Web-based survey (as opposed to a phone-solicitation survey) of our subscribers to capture significant data from our most active resellers. The technology categories were determined by the Business Solutions magazine’s (BSM) editorial staff.
Who participated in the survey? How did they vote?
From September 2015 to mid-October 2015, BSM VAR subscribers were asked to participate in the Best Channel Vendors survey. We asked subscribers to rate their vendor partners on a scale of 0 (worst) to 5 (best) in seven categories. We also provided some guiding questions to ensure consistency in how VARs interpreted each category:
How many VARs participated? Did you throw away any votes?
A total of 2,737 people participated in this year’s survey, but we eliminated ballots from voters who did not qualify as resellers. In all, 5,401 validated votes were cast in the survey.
How were the Best Channel Vendors determined?
An overall average score for each vendor was determined by adding the cumulative scores of the seven categories and dividing that score by the number of VAR subscribers who rated that product. From that list, our editorial staff chose the Best Channel Vendors by applying a “Scree Test.” We won’t go into the full definition here — we’ll leave that to the Penn State statisticians to explain — but basically our editors applied reasonableness to the results and drew the line where there was a clear break between the highest scorers and the “rest of the pack.” Our goal was to reflect the sense of the survey respondents and let our readership know who they felt were the Best Channel Vendors. Those familiar with our past Best Channel Vendors surveys might remember that our previous results included the top 15 percent of vendors in each technology category and that last year we tightened the grading criteria and chose the top 5 percent of channel vendors based on reader feedback. We continued that tradition for this year’s results.
How are the Best Channel Vendors listed?
They are listed first by technology category (Business Continuity through VoIP) and then by technology subcategory. Best Channel Vendors are listed in alphabetical order. We did not rank them according to their overall average score. One of the main purposes of this special report is to give our resellers a guide to who their peers think are Best Channel Vendors. Ranking vendors according to their overall average score would mislead readers and defeat that purpose.
Did you “fudge” any scores to make sure your advertisers were included on the list? Did any vendors buy their way onto the list?
No. Other magazines have skewed their surveys to include current advertisers or entice new advertisers. We believe engaging in that behavior is not only a disservice to the industry, but also it’s dishonest. We would not compromise the reputation our magazine has built over the past 29 years for the extra revenue a few ads would generate.
OK, so the survey is accurate. But is it perfect?
Certainly not. Penn State’s statisticians stressed to us that all surveys have their limitations. This survey is not perfect, but it should be an excellent tool for the channel because it indicates products and vendors that have highly satisfied VARs.