Articles

Can UX Metrics Predict Software Revenue Growth?

Does better usability lead to more revenue?

What about positive word of mouth? Is it tied to revenue growth?

Are UX metrics for usability and intent to recommend able to track future revenue growth?

Many UX researchers who work for software companies or on software products collect UX metrics. In fact, we strongly advocate for it. As part of implementing a plan to improve UX, you need to start with UX metrics.

But is there any evidence that UX metrics are tied to business metrics such as revenue growth in the software industry?

SUS and NPS

To look for a relationship between UX metrics and growth in the software industry, we started with two of the most popular UX metrics: the System Usability Scale (SUS) and the Net Promoter Score (NPS). Both metrics are widely collected and reported, which make them good candidates for looking for a relationship with what UX teams measure (usability and likelihood to recommend) and what the business cares about (revenue growth and future revenue growth).

NPS Predicts Revenue Growth in Some Industries

I’m not aware of any studies associating SUS scores with revenue. We have examined the relationship between the  NPS and future revenue growth. We found that using the original NPS data reported by Reichheld was a good predictor of growth within the industry. Across the seven industries studied we found the NPS can predict around 38% of future revenue growth.

However, this likely represents a “best case” scenario for the predictive ability of the NPS as it was pre-selected to make the case for the NPS as a proxy for growth in 2003. Those industries included rental cars, life insurance, airlines, and grocery stores where experiences are often a mix of service, product, and company interaction. Does the same relationship apply in the software industry where the focus is often dominated by a product experience?

Finding SUS & NPS Data

As we did in our NPS replication study, we needed to go back in time far enough to examine future growth rates. For NPS data we used two data sources: a 2014 software report purchased from Satmetrix (12 products) and our (MeasuringU) NPS benchmark report. For SUS data we also used our 2014 software benchmark report (20 products).

There were some differences between how the data was reported and what products were included in both reports. SUS scores were only collected from the MeasuringU report. Also, in the MeasuringU report we broke out NPS and SUS scores by product, even when the product was part of a suite. Satmetrix reported only the NPS for MS Office and Adobe Creative Suite, while we provided NPS and SUS scores by product; Word, Excel, PowerPoint, and Photoshop.

To see how similar these independently collected sources of NPS data were, we needed to find the products in common. We approximated MS Office scores by averaging together Word, Excel, and PowerPoint scores but only had one of the products (Photoshop) in the Adobe Creative Suite. Satmetrix also included ride-sharing services Uber and Lyft, which we didn’t consider software and didn’t collect, so they aren’t included in this analysis.

Together that left us with four products in common:

  • TurboTax
  • Microsoft Office
  • McAfee Antivirus
  • Mint.com

Despite the small number of products, we still found a strong correlation between our NPS scores and the Satmetrix NPS scores (r = .84) suggesting that despite the different data collection methods and sources we have good agreement. This is an encouraging point of corroboration.

SUS Strongly Correlates with NPS

We’ve consistently seen a strong relationship between the SUS and NPS across many data sets. Typically, attitudes toward usability as measured by the SUS explain between 30%–50% of likelihood to recommend scores (the NPS). We also see that strong relationship with this data. The correlation between the 20 product SUS scores and NPS scores in the MeasuringU benchmark report is r = .81, meaning attitudes toward usability explain a substantial 66% of the variability in consumer software NPS. While these two concurrently collected attitudinal metrics have a strong correlation, do either of these predict revenue growth for their product?

Finding Growth Rates

To look for a relationship between metrics and growth we used a similar approach to our NPS analysis. With the 2014 NPS and SUS data, we collected financial data for the immediate years to predict growth (2014, 2015, and 2016). We again combed through financial statements of consumer software companies to find growth rates (not the easiest task).

Not all products have clear revenue because they’re free (e.g. Adobe Reader, Google Calendar, iTunes, Google Docs) or we had insufficient or no SUS data (e.g. Adobe Creative Suite, Google Docs, Norton Antivirus), or the revenue was not reported or associated to the product that we could find (e.g. Webex, Mint.com, Google Drive). We were able to find clear financial data for eight products (linked to sources of financial data so you can replicate):

UX Metrics Predict Growth

We found a strong positive correlation between the SUS and NPS for both 2014–2015 and 2014–2016 future growth years. The correlations using 2014 NPS and 2014–2015 revenue were strong (r = .65; p = .08) as well as with 2014–2016 revenue growth (r = .74; p = .03). The magnitude of the correlations is similar to what we found with Reichheld’s data, where NPS correlated r = .62 with immediate two-year revenue growth.

For SUS we also found a strong positive correlation between 2014 SUS scores and 2014–2015 revenue growth (r = .59; p = .12) and between 2014–2016 revenue growth (r = .74; p = .04).

Visualizations of the relationship between SUS and NPS can be seen in Figures 1 and 2. On average we’re seeing that attitudes toward usability (SUS) and users’ likelihood to recommend (NPS) are able to explain (predict) at least 50% of two-year future revenue growth rates. This is even larger than the relationship we found at the company level from Reichheld’s data.

Figure 1: Relationship between 2014 SUS scores and 2014–2016 revenue growth.

 

Figure 2: Relationship between 2014 Net Promoter Scores and 2014–2016 revenue growth.

When conducting a regression analysis, especially with a small sample size, one data point can have a large influence on the statistical relationship—either making or masking the underlying correlation.

In both figures we can see Dropbox had a high growth rate and corresponding high NPS and SUS scores. In contrast, AutoCAD had a low SUS score and lower growth for this period. Both products have a large influence on the regression equation and correlations. To see how robust the relationship is between these metrics and growth rates, we removed both products and reran the correlations. Encouragingly, even after removing both products, the correlations remained for SUS (r = .64) and NPS (r = .40). Not surprisingly though, with only six products, neither correlation was statistically significant—again illustrating the challenge of looking for relationships with only a few data points. Only very large correlations will be significant, but the non-significant correlations are still meaningful as part of a meta-analytic approach across studies and industries. This finding is nonetheless quite impressive as it shows a clear link between something that’s relatively easy to collect (user attitudes toward usability and recommending) and something that’s hard to collect, but important (future revenue growth). We’ll look to continue to corroborate these findings with future datasets and analyses.

 

Summary & Takeaways

In this article, we looked at the relationship between common consumer software UX metrics (NPS and SUS) and future revenue growth. We found:

Attitudes toward usability and likelihood to recommend explain (predict) future revenue growth. We found both SUS scores and Net Promoter Scores collected in 2014 had strong correlations to the revenue growth in the immediate year and two years (2014–2016).

Attitudes correlate with outcomes. Given the similar correlations for both the NPS and SUS and future growth, it’s likely that other attitudinal measures (e.g. satisfaction, UMUX-Lite, TAM) aggregated at the product level will correlate with future revenue. Future research should examine other measures with future revenue growth to see whether the SUS and NPS are “special” or just one of many attitudinal measures that companies can track.

UX metrics are valid leading indicators: This analysis suggests tracking common UX metrics such as the SUS and NPS in the consumer software industry may be good leading indicators of revenue growth. This is encouraging because many organizations already collect this data for their products. It’s also encouraging that popular UX metrics that measure customer attitude (which are easy to collect) are tied to future business outcomes (that are often hard to measure).

Free and bundled software may mask relationships. It’s quite challenging trying to isolate both attitudes toward products and then associate those attitudes to financial metrics. There’s a lot of free and bundled software (e.g. G Suite) that makes it hard to isolate the relationship between attitudes and business outcomes. A future analysis can include examining more success metrics for software that doesn’t have clear financial data.

Additional analysis is needed (always). If you’ve read my articles you know I’m a big fan of corroborating data. The more independent data sources that are able to replicate findings, the stronger the claims should be. I’m not aware of other data sets that have attempted to find the relationship between UX metrics and revenue and encourage others to look for this relationship (please share with us). We’ll be sure to keep investigating the relationship and post our findings.

from MeasuringU https://measuringu.com/uxmetrics-growth/