RMM Survey - May 2018

Introduction

This survey was taken in May 2018 and posted on the /r/msp subreddit. Around 380 responses were collected.

Let's get the obvious out of the way. This is data from an online survey open to the public, it is for entertainment and discussion purposes only. Further, there are many elements which were not considered (e.g., price) which would normally be important factors when choosing an RMM platform.

Only RMM platforms that received more than 5 ratings have been included - 5 ratings would represent less than 1.5% of the total responses, which I believe is a reasonable cut off.

Results

Respondents, Platforms, and Survey Overview


RMM Platforms

This is a breakdown of the RMM platforms included in the results, along with the number of responses for each platform, and what percentage of the total that number represents.

Region of Origin

As expected, Americans consume the majority of the doughnut, followed by Europeans and Australians. There was one respondent from South Africa, and one from Texas (yes, Texas).

Average Weighted Rating

The ratings were calculated using Bayesian estimate which normalizes scores based on the number of ratings. The more ratings submitted, the more trustworthy the score is considered. This ensures a product rated Excellent (5) by a small number of respondents does not look better than a product rated 4.5 by 100 respondents - we can be much more confident of the cumulative wisdom of 100 ratings than the opinion of a just a few.

This can be potentially perceived as a bias against smaller RMMs that don't have many users, but unweighted (raw) data is of little use and is easily manipulated when there are few respondents. If you want to read more on this topic, this is a good summary.


The One Chart to Rule Them All

SolarWinds N-Central came out as the overall leader, but only by a feather, following closely behind are NinjaRMM and ConnectWise Automate. All platforms with the exception of Atera scored in the 3s indicating there wasn't a clear leader. From here we can drill down and view the ratings by category.

Categorised Weighted Ratings

Let's dig into the details.

You will notice that many charts have bars which are a different shade, the shade represents the consensus across the ratings. For instance, 5 ratings resembling [3,3,3,3,3] would have a higher consensus (darker bar) than 5 ratings resembling [1,1,3,5,5], which would indicate lack of consensus/divisiveness (lighter bar). High divisiveness would indicate different expectations and experiences. This is one of the more interesting aspects of the data as we often see a large spread in the ratings given the same product and category. There are many factors which could account for this:

  • Seat count, different RMM packages may be more/less suitable to a given MSP size (more on this below)
  • Time spent on RMM configuration and maintenance. Some RMM packages may require more tuning and work to get right, companies that were willing to invest the necessary time and resources may have better experiences. On the other hand, the inverse can also be true - there could be some expectation that things 'just work'.
  • Past experience, coming from an RMM which does one thing exceptionally well may leave you feeling let down when another platform does it less well, someone without that experience may rate the product differently.

    • Stability & Reliability A measure of how reliable, stable, and bug-free the platform is

    Stability appears to be relatively solid across all platforms, with none scoring below 3. NinjaRMM has the lead, followed by a AEM and N-Central. AEM and NinjaRMM have equal consensus ratings and N-Central has the widest range of ratings (lowest consensus) of the three.

    Remote Access A measure of the remote access/control capabilities of the platform.

    Remote Access had some of the highest scores, with two platforms breaking the 4 line. Automate came out as the clear winner, with the highest score and a low distribution of scores. Following behind were NinjaRMM and N-Central. NinjaRMM had the highest consensus, while Automate had the lowest. Honourable mention to Continuum for scoring 0.01 behind N-Central, and almost equaling third place.

    Scripting A measure of the scripting capabilities of the platform

    Automate again came out on top, following behind was AEM, and N-Central. Consensus was fairly even across all three, with AEM having the highest by a small margin. Kaseya sits in the middle, between the top 3, and the remaining platforms.

    Patching A measure of the software patching capabilities of the platform

    Patching is where things started for fall apart, with no platforms hitting 4, and several below 3. Best performer in this category is NinjaRMM, barely edging out Continuum, with Ninja having a higher consensus rating, and N-Central coming in third, followed closely by AEM and then Kaseya. These four are the only products to score above 3.

    Reporting A measure of the reporting capabilities of the platform

    The downward spiral continues with reporting being one of the worst performing areas, not a single platform breaking past 3.5. Continuum scored the highest, followed by Pulseway and N-Central. Consensus was low across all three.

    Inventory & Auditing A measure of how well the platform can audit and inventory systems

    Avast Managed Workplace edged out the rest in its first win despite having a low number of respondents, followed by Kaseya and AEM. Managed Workplace had the highest consensus of the thee.

    Integration A measure of how well the platform integrates with 3rd party software

    Another first, SolarWinds RMM was the best performer here, followed by Automate and Continuum. SolarWinds RMM and Continuum had solid consensus ratings, while Automate had a lower consensus.

    Support A measure of the quality of support provided by the platform vendor

    Ninja came out as the clear winner with solid consensus, followed by Continuum and Pulseway, with Continuum having the lowest consensus of the three. Automate had its lowest score here, it's interesting to think that had their support been rated higher they could have potentially beaten N-Central in the overall ratings.

    MSP Size

    Some RMM platforms may be suited to smaller companies, and other, potentially more complex and flexible platforms, may be suited to larger MSPs where dedicated resources can be employed to manage the system. Let's start off with the median MSP seat count.

    MSP Seat Count Distribution

    Here the seat counts are split into into four groups so we can get an idea of the distribution. We can see almost 50% of respondents worked for MSPs with 101-1000 seats, followed by 1001-5000 seats.

    RMM Popularity by Seat Count

    Seat count would be a definite factor in RMM choice, and while SolarWinds N-Central was the overall winner, let's see how the size of an MSP determines RMM platform choice and satisfaction.

    1 - 100

    101 - 1000

    1001 - 5000

    5001+

    It's immediately clear that Automate and N-Central are not as popular with smaller MSPs. Here, Ninja, SolarWinds RMM, and Atera dominate.

    Moving up to the 101 - 1000 seat range, N-Central doubles its slice, while Automate almost triples in popularity. AEM has a bit of a jump as well, while Ninja drops a few percent along with SolarWinds RMM. Atera and Comodo One take a considerable dive in popularity.

    In the 1001 - 5000 range, Automate gains even more ground, with a substantial leap by Kaseya, and N-Central makes some gains again. Several other platforms take a beating here, with Ninja and SolarWinds RMM barely hanging on, while several other platforms have dropped off completely.

    Finally, in the 5000+ seat group, we're down to even fewer contenders and see N-Central and Kaseya overtake Automate as the most used platforms.

    Overall N-Central, Kaseya, and Automate seem to cope with growth as they have good presence in all groups and only appear to get stronger as we move up. Continuum and AEM show similar patterns, though overall they are less popular than the aforementioned three.

    RMM Platform Average Rating per Seat Count

    Having the ratings split into four groups presents several challenges. Firstly, each group would have a smaller number of responses than the total, which makes use of the Bayesian estimate impractical. Following on from this, it means we have fewer tools available to combat 'outlier' responses, allowing the results to be easily skewed. Looking at the data, I've found a reasonable compromise. Using normal averages instead of Bayesian estimates, but only including responses in the 95th percentile produced results which did not have unusual spikes caused by outlier votes, but also did not punish platforms with smaller presence in any particular group.

    Further, only platforms with three or more responses per group have been included, this may mean that a platform is on the doughnut charts above, but is missing from the bar charts below.

    Consider the popularity of each platform in the doughnut charts to get an idea of the number of responses per platform, and keep in mind that the shade of the bars determines how consistent or varied the responses were. A darker bark means higher consensus, and a lighter bar means more divisiveness.


    1 - 100

    101 - 1000

    1001 - 5000

    5001+

    NinjaRMM, AEM, and Pulseway have the top three spots for the 1-100 seat count. Ninja has the highest consensus of the three, which is impressive considering it also has the biggest share of users.

    In the 101-1000 range Continuum has a large jump to take the top spot, followed by Managed Workplace, and a two way tie between NinjaRMM, and N-Central. Consensus is fairly even between all platforms, though not particularly high.

    Moving onto the 1001-5000 group, many platforms drop off, leaving N-Central, AEM, and Continuum in the lead, though Automate and Kaseya are not too far behind. Continuum has the highest consensus, but this is possibly due to the small share of users in this seat group.

    Finally, in the 5000+ space, only 3 platforms with 3 respondents or over remain, out of those N-Central has come out on top, with Automate and Kaseya lagging behind. Consensus was lowest for N-Central, and highest for Automate.

    RMM Platform Longevity

    Half of the responses are from MSPs that have been on their current RMM platform for around 2.5 years, while 36% said that they are considering a change. Lets dig a bit deeper.

    Considering a change by RMM platform

    SolarWinds RMM, Atera, and Avast Managed Workplace are at highest risk of losing MSPs, all thee with over 50% of respondents claiming they would look at alternatives, while Pulseway, Autotask, and NinjaRMM are relatively safe.

    Considering a change by years on current platform

    We can look at the same data a little differently, this shows what percentage of responses said they were considering a change, and how many years they've been with their current RMM on the X axis. It seems the most popular time to change RMMs is when you've been using your existing platform for 2-4 years.

    Unfortunately we don't have an answer to the most important question - why? Are MSPs rapidly increasing seat count and outgrow the product after 2-4 years? Or it is human nature and a case of the grass being greener on the other side? Why is there a decline after 4 years - have they surrendered to, or mastered the platform?

    Further, as we have no historical data it's important to be wary - we can't tell whether the 2-4 year "hot zone" is a constant (related to MSP growth, human nature, etc), or whether it changes based on RMM performance. I.e, the 2-4 band has high numbers because that's where the unpopular RMMs sit.

    Putting it together

    Let's explore that last comment.

    The question is, are MPSs looking to move away from Atera, Avast, SolarWinds RMM, etc because they perceive them to be bad products, or because they've reached that 2-4 year period where you're likely to start looking anyway, and, is that period a constant?

    This is the same chart as above, but with additional bars which represent the average rating, and the average years responders have used a particular platform, and two purple lines which show the 2 year and 4 year band.

    What we're looking for are platforms that don't fit the mould. Atera and Comodo have a relatively young user base (outside of the 2-4 year zone), but they don't seem too happy, reinforcing the poor average rating. AEM on the other hand appears to be doing well, it's entering the hot zone, but is still keeping MSPs (mostly) happy. SolarWinds RMM is in the zone, but also well over the average percentage of users looking at alternatives.

    Thank you!

    That's it for now, thanks for reading and thanks to all that participated in the survey. If you'd like to contact me, you can send a PM on reddit (though I don't check that account often), or fill in the form below. More of my work can be found at xkln.net.


    Creative Commons License