RMM Product Testing Methodology
Prior to starting the RMM test, I had several calls with Eric Brown and his team, which included Kevin Danis, Diana Piesak, and George Renz. We discussed the criteria we wanted to use for the tests to ensure every product was tested fairly. I provided RTM with a form that included fields for key criteria we wanted to capture, which the MSP filled out for every vendor.
I followed up with RTM right after the testing to get a few more details about the methodology they used:
What were the basic steps you followed to test each RMM product?
Brown: The first thing we did, prior to testing any of the products, was set up a call with a representative from each vendor. We explained that we were testing their product and we wanted access to a product expert, so that if we ran into any technical problems or had questions, we could quickly get our questions answered.
This turned out to be very helpful, especially considering we had about four weeks to test all the products. Without the help from each product expert, it would have taken quite a bit longer to figure out how to set up each product. With Level Platforms, for example, the first install with their server we did together. They walked us through the process of downloading and installing the product from their portal and they set up a WebEx to help us set up a virtual machine. With their help this was an hour and a half process; without their help, it would have been an all-day experience.
What safeguards did you put in place to ensure each product would be tested consistently?
Brown: We used the same equipment for each test, including the same two servers, two laptops, and virtual machines for all tests were used.
Considering that multiple people were involved in the test, how did you minimize individual biases/margins of error?
We did a lot of group testing. Kevin and I did a lot of tests together. Also, in situations where one team member already had prior experience with a particular product, as was the case with Continuum, I had another member of the team who wasn’t as familiar with that product conduct the testing, and we compared notes afterwards. Having the spreadsheet ahead of time helped create consistency, too.
What was the biggest challenge you ran into with the test?
We underestimated the time it would take to test each product by 8 hours each. In the end, it took two full days per product per person.