Voice of the Customer Methodology

Gartner Peer Insights Voice of the Customer Methodology

Gartner Peer Insights avatar
Written by Gartner Peer Insights
Updated over a week ago

For Nov 2022 publish and after. Click here for Oct 2022 publish and earlier.

Last Updated: July 2022

Gartner Peer Insights is a free peer review and ratings platform designed for enterprise software and services decision makers. Reviews are organized by products in markets that are defined by Gartner Research in Magic Quadrant and Market Guide documents.

The “Voice of the Customer” is a document that applies a methodology (documented here) to aggregated Gartner Peer Insights’ reviews in a market to provide an overall perspective for IT decision makers. This aggregated peer perspective, along with the individual detailed reviews, is complementary to expert-generated research such as Magic Quadrants and Market Guides. It can play a key role in your buying process, as it focuses on direct peer experiences of buying, implementing and operating a solution.

Understanding the “Voice of the Customer” Methodology

Vendors in the “Voice of the Customer” document are categorized into four quadrants based on User Interest and Adoption (X-axis) and Overall Experience (Y-axis). Vendors from any of the four quadrants may be the best fit depending on your business needs.

User Interest and Adoption (X-axis)

User Interest and Adoption scores incorporate three factors, each given equal weight: review volume, user willingness to recommend, and review market coverage across industry, company size, and deployment region.

  • Review volume is calculated using the total number of eligible reviews (see below for eligibility details) for a vendor over the time period. An asymptotic function is applied to reduce sensitivity to small changes in review count, especially for vendors with a large number of reviews.

  • Willingness to recommend is calculated based on the responses from the reviewers. Reviewers are asked, “How likely is it that you would recommend this product or service to a friend or colleague?” The reviewer can select a response on a rating scale of 0 to 10, where 0 is “not at all likely” and 10 is “extremely likely.” The response is considered “willing to recommend” if the rating is greater than or equal to 8 and “not willing to recommend” if the rating is less than 8. A vendor’s willingness to recommend score is the percentage of “willing to recommend” responses for both questions as defined above out of the total number of responses to the question.

    • For reviews first submitted in mid-2018 or that came from the Magic Quadrant Reference Survey, the question “Would you recommend this product to others” was asked instead, with responses including “yes,” “yes, with reservations,” “I don’t know,” and “no.” The response is considered “willing to recommend” only if the reviewer chose “yes.”

  • Review market coverage is calculated by counting the number of different industry types, company sizes, and deployment regions for which the vendor has at least 5 eligible reviews. An asymptotic function is applied to this count so that low market coverage results in a lower score but returns diminish for very high coverage. If an eligible review does not have industry, company size, or deployment region information, it will not contribute to the calculation for that bucket but will be included for all other calculations.

    • Industry of the reviewer is based on the reviewer profile industry. Reviewers can select from one of the standard industry categories:

      • Communications

      • Construction

      • Education

      • Energy and Utilities

      • Finance

      • Government

      • Healthcare

      • Manufacturing

      • Media

      • Retail

      • Services

      • Transportation

    • Deployment region is a question in the survey that each reviewer must answer. They can select one or multiple regions:

      • Asia/Pacific

      • Europe, Middle East and Africa

      • Latin America

      • North America

    • Company sizes are grouped based on reviewer profile details as entered by reviewers in one of the following categories:

      • 10B+ USD

      • 1B – 10B USD

      • 50M – 1B USD

      • Public Sector, Gov’t, Edu

Market Average: The midpoint of the axis labelled as “Market Average” in the Voice of the Customer graphic references the mean value calculated from all vendors directly included in this analysis. This may not represent the overall market as represented on Peer Insights. A vendor must meet or exceed this value for User Interest and Adoption to be placed in the right-hand quadrants.

Overall Experience (Y-axis)

A vendors’ Overall Experience is a composite score that incorporates the Overall Rating average and two sub-rating averages: ‘Capabilities’ (Product Capabilities for software markets; Service Capabilities for service markets) and ‘Support/Delivery’ (Service & Support for software markets; Delivery & Execution for service markets). These components are combined into the Overall Experience score based on Gartner Peer Insights determination of the importance of each component in an end-user buying decision. This calculation is subject to the conditions specified in the “Review Eligibility and Weighting” section below.

The ‘Overall Rating’ is the rating question on the survey that reads: “Please rate you overall experience with this vendor.” This question is required and survey respondents must give a 1 to 5 rating (no “N/A” option).

The ‘Capabilities’ rating is within the ‘Overall Summary’ section of the review survey. For software markets, it reads: “Overall rating of product capabilities.” For service markets, it reads: “Overall rating of services capabilities.” This question is required and survey respondents can either give a 1 to 5 rating OR select “N/A” if they feel they are not suited to answer the question.

The ‘Support/Delivery’ rating is also within the ‘Overall Summary’ section of the review survey. For software markets, it reads: “Overall rating of service and support.” For service markets, it reads: “Overall rating of delivery and execution with the vendor.” This question is required and survey respondents can either give a 1 to 5 rating OR select “N/A” if they feel they are not suited to answer the question.

Market Average: The midpoint of the axis labelled as “Market Average” in the Voice of the Customer graphic references the mean value calculated from all vendors directly included in this analysis. This may not represent the overall market as represented on Peer Insights. A vendor must meet or exceed the mean of all eligible vendors’ Overall Experience scores to qualify for the top-half quadrants.

Note that ‘Overall’ vendor ratings used in calculation (and displayed in vendor specific graphics in VOC) may not exactly match the star ratings displayed on the Gartner Peer Insights site, where the default star rating is based on all published reviews, with reviews older than 12 months weighted progressively less (decreased by half every 12 months).

Understanding the Quadrants

Upper-Right Quadrant: Customers’ Choice

Gartner Peer Insights recognizes vendors who meet or exceed both the market average Overall Experience and the market average User Interest and Adoption score through Customers’ Choice distinction.

Vendors in this quadrant received strong ratings for overall experience, product capabilities, and service & support relative to the market. This is a good indicator for satisfaction with the vendor’s products in this market.

Vendors in this quadrant also exceed the market average User Interest and Adoption, which includes review volume, review market coverage, and willingness to recommend. These factors help distinguish vendors that generated the above average Overall Experience scores required for this quadrant within large sample sizes and diverse coverage. High willingness to recommend is an overall measure of users’ interest and enthusiasm based on their experience with the vendor. Vendors in this quadrant may not have performed equally highly in all three of these components.

Product selection processes that value mainstream adoption and broad coverage would find vendors in this quadrant a likely fit. Processes that prioritize specialized, verticalized, customized or novel solutions should also consider other quadrants when formulating a short list.

Upper-Left Quadrant: Strong Performer

Vendors who are rated equal to or above the market average Overall Experience but do not meet the market average User Interest and Adoption score.

Vendors in this quadrant received strong ratings for overall experience, product capabilities, and service & support relative to the market. This is a good indicator for satisfaction with the vendor’s offerings in this market.

However, vendors in this quadrant are below the market average User Interest and Adoption. There are several possible reasons for this: the vendor may have received a lower volume of reviews during the 18-month period captured in this document; the vendor’s business model may be focused on a narrower set of regions, industries, or company sizes; or users may be unwilling to recommend the vendor.

Vendors in this quadrant can be worth investigating further as they score well on Overall Experience, may have a strong willingness to recommend, and may excel at specific use cases, in specific industries or regions, or for specific organization sizes.

Lower-Right Quadrant: Established

Vendors whose ratings are less than the market average Overall Experience but have higher-than-average User Interest and Adoption.

Vendors in this quadrant received below-average ratings for overall experience, product capabilities, and service & support relative to the market.

However, vendors in this quadrant also exceed the market average User Interest and Adoption, which includes review volume, review market coverage, and willingness to recommend. High review volume is an indicator of high user adoption and confidence in the vendor’s placement due to a larger sample size. High review market coverage indicates the vendor’s coverage across a diverse range of industries, regions, and company sizes. High willingness to recommend is an overall measure of users’ interest and enthusiasm based on their experience with the vendor. Vendors in this quadrant may not have performed equally highly in all three of these components.

It is important not to overlook these vendors because they have generated high user interest and users are willing to recommend them. Additional investigation may need to be done to determine if the reasons for the user interest represented in this report align with your organization’s needs.

Lower-Left Quadrant: Aspiring

Vendors in this quadrant are below the market average on both Overall Experience and User Interest and Adoption. However, vendors in this quadrant can be worthy of consideration for users, depending on specific use cases. Like all vendors in this report, their products align to this market and they have met the minimum criteria to be included. If a vendor is newer to the enterprise space, unique in some ways, or innovating rapidly, they could appear in this quadrant. However, it is also possible that vendors who are struggling to reach long-term viability also appear here. Due to these possible mixed results, users must delve into the review details and cross-check with other Gartner research resources to make better decisions.

Review Eligibility and Weighting

Eligible reviews used for the “Voice of the Customer” are only those reviews published on the Gartner Peer Insights site that were submitted:

  • In the 18-month period specified in the “Voice of the Customer” document.

  • From end users of companies with >$50M in annual revenue.

  • For products that are actively being sold by vendors. Reviews for legacy products (no longer being sold by the vendor) will be excluded from all calculations.

  • And included in the ratings calculations on the Peer Insights site (excluding vendor partner reviews, etc.)

Additionally, reviews from the first 6 months of the 18-month period will be given 50% weight in all calculations for both axes. This means that one review from the first 6 months will have half the impact of one review from the next 12 months in determining a vendor’s X-axis (including all 3 components: review volume, willingness to recommend, and review market coverage) and Y-axis scores. The 18-month review period allows for a larger sample size of reviews and greater confidence in each vendor’s placement on the quadrant. The 50% weighting for reviews in the first 6 months allows more recent reviews to factor more heavily in determining a vendor’s placement.

Vendor Inclusion Criteria

The “Voice of the Customer” report includes vendors who have products listed in a particular market (see Gartner Peer Insights Listing Guidelines here), and have received 20 or more eligible reviews along with 15 or more ratings for the ‘Capabilities’ and ‘Support/Delivery’ sub-ratings over the 18 month analysis period.

Currently, Gartner Peer Insights limits the number of vendors included in the document to 25. If more than 25 vendors meet the inclusion criteria, the 25 vendors with the highest number of eligible reviews will be included. If two or more qualified vendors with the same number of eligible reviews tie for the 25th position, all vendors in the tie will be included in the document.

“Voice of the Customer” Segment View

Understanding the “Voice of the Customer” Segment View Quadrants

In response to end-users’ desire to evaluate solutions based on specific attributes of their organization, such as industry, company size, or deployment region, Gartner Peer Insights now provides analyses for those categories where there was sufficient review volume. The full list of industries, company sizes, and deployment regions can be found above in the “User Interest and Adoption” section. In each segment graphic, vendors are positioned in different quadrants based on their reviews within the specified segment. As with the overall market “Voice of the Customer,” the X-axis represents User Interest and Adoption and the Y-axis represents Overall Experience. Gartner Peer Insights recognizes highly-rated vendors in categories through the Customers’ Choice Segment Distinction.

The review inclusion criteria, vendor inclusion criteria, y-axis, and four quadrants for the Segment View follow the same logic as the overall “Voice of the Customer” (see above), except that only reviews within the segment are included in the calculations. For example, for a vendor to be included in the Finance Segment View, the vendor must have at least 20 reviews (with 15 or more ratings for the ‘Capabilities’ and ‘Support/Delivery’ sub-ratings) that meet all Review Eligibility criteria listed in the relevant section above and are from reviewers in the Finance industry.

The calculation of the X-axis (User Interest and Adoption) for the Segment View differs slightly from the approach for the overall market “Voice of the Customer.” For Segment View, X-axis scores incorporate two factors, given equal weight: review volume and user willingness to recommend. These are calculated in the same way as specified in the relevant section above, except that only reviews within the segment are included in the calculations. Because the Segment View focuses on a single segment, it does not incorporate review market coverage across industry, company size, and deployment region.

Mentioned in a Voice of the Customer document and want to use in your external marketing? Check out the Marketing Tools for Vendors page.

Frequently Asked Questions

About VOC and Markets

What is the Gartner Peer Insights ‘Voice of the Customer’?

The Gartner Peer Insights ‘Voice of the Customer’ is a Gartner research document that synthesizes Gartner Peer Insights’ content in a specific market for a specified 18-month period. This peer perspective along with the individual detailed reviews are complementary to expert research and was developed as an additional resource for end-users in their buying process.

What will you do to recognize vendors?

We will announce the vendors online on the Gartner Peer Insights platform. We also encourage vendors to drive awareness of their achievement through various marketing channels. We will provide digital marketing collateral and templates to help with these efforts.

Does this replace the Magic Quadrant (MQ) view of a market?

No, Gartner Peer Insights complements Magic Quadrant and Critical Capabilities reports by offering perspectives solely about what a vendor’s customers think of its products. Based on end-user feedback, our clients value having a holistic view of Gartner expert opinion and end-user perspective when evaluating vendors and their products in a market.

How will this data be used in the MQ? Will receiving the distinction put us in the Leaders’ quadrant?

The raw data from Peer Insights reviews is made available to our analysts, but it is only one factor among many that they consider when creating an MQ.

Publication/Schedule Related

Will I get this document to review before it publishes on the site (like we do for the MQ)?

The data used in the report is entirely based on end-user reviews and ratings that are available on the Peer Insights site. As a courtesy heads-up, we will share a draft directly with the vendors included in the document approximately 2 weeks ahead of report publication.

Is there a minimum criteria required in order to publish a Voice of the Customer document for a market?

Gartner reserves the right to selectively publish a Voice of the Customer document based on our determination of the value of that document to our end users. Currently we will only consider a Voice of the Customer document in markets where there is a certain level of market maturity (representing vendors and number of reviews) on the Peer Insights platform, and 5 or more vendors represented in the document. Only vendors with 20 or more reviews during the 18-month submission eligibility period are included in a Voice of the Customer research document. For more details, please go to the Voice of the Customer Methodology page.

Which markets are/will be covered by the Gartner Peer Insights ‘Voice of the Customer’ report?

Gartner reserves the right to selectively publish a Voice of the Customer document based on our determination of the value of that document to our end users. Currently we will only consider a Voice of the Customer document in markets where there is a certain level of market maturity (representing vendors and number of reviews) on the Gartner Peer Insights platform. For more information, see the detailed methodology and Voice of the Customer Publication Calendar.

How do I access the Publication Calendar of markets for Voice of the Customer and Customer's Choice?

The Publication Calendar for upcoming Voice of the Customer markets can be found here. Kindly note that this list and schedule are tentative and subject to change at the discretion of Gartner Peer Insights. It is updated frequently, so be sure to check often.

Are Voice of the Customer documents published for all markets? How do you determine which markets will go for Voice of the Customer?

Gartner reserves the right to selectively publish a Voice of the Customer document based on our determination of the value of that document to our end users. Currently, we will only consider a Voice of the Customer document in markets where there is a certain level of market maturity (representing vendors and number of reviews) on the Gartner Peer Insights platform, and more than 5 vendors represented in the document. Only vendors with more than 20 reviews during the 18-month submission eligibility period are included in a Voice of the Customer research document. For more information, see the detailed methodology and Voice of the Customer Publication Calendar.

Is the Voice of the Customer still an annual document under this updated methodology? Is there any opportunity for a Voice of the Customer document to refresh more often?

The calendar for the Voice of the Customer document is based on the research publication calendar. After the Magic Quadrant or Market Guide publishes, the Gartner Peer Insights team will determine if the market meets the minimum review requirements for a Voice of the Customer document. Typically, the Voice of the Customer document publishes 5 months after the corresponding Magic Quadrant or Market Guide publication. If the Magic Quadrant or Market Guide publication is delayed, the Voice of the Customer will be delayed accordingly. See the Voice of the Customer Publication Calendar for a full calendar of upcoming markets.

How do I know when the 18-month window starts for my markets?

The start and end dates for upcoming markets are noted within the Voice of the Customer Publication Calendar. Alternatively, the 18-month period is retrospective from the deadline displayed on the Standings Page dashboard for the market.

Reprints/Quote Requests

Can we leverage this document as a reprint? What is the price for the reprinting this document?

The Gartner Peer Insights ‘Voice of the Customer’ document is able to be licensed as part of the Gartner reprint program to support your sales enablement, lead generation, and content marketing strategies. This Peer Insights research document reprint follows the same policies and pricing guidelines as any other research note reprints (e.g., MQ, critical capabilities). Existing Reprints clients may use an existing document “slot” for Peer Insights content. Please contact your account executive for more details.

The document provides valuable synthesized insights from our reviews that complement the analyst perspective in a format your clients could easily share with their investment committees.

If I can get this information from the free Peer Insights site, why should I buy the reprint?

We developed the PI research note as we heard consistently from end-users that they need a better shortlisting tool in a format they could easily share with their investment committees. The research note synthesizes the key insights from all the information available on the site using our proprietary methodology.

Although the data is available on site, the images that synthesize the information is not available. For example, the User Interest and Adoption axis in the main quadrant (Figure 1) is only available through this document.

It includes only a smaller set of eligible vendors, and shows their ratings over a fixed period of time (last 18 months) vs. the PI site which is continuously changing as reviews come in.

Many vendors who license the reprint of this report, often use it for lead generation. You cannot do that with Peer Insights site.

Purchasing a reprints license gives you access to a dedicated program manager who will handle all of the legwork for quote request approvals and help you to track the reprint impact


Methodology Related (Including CCD)

Which VOC methodology is currently used? Will the methodology change from year to year?

Our goal is to be analytically sound and transparent. To meet these goals, we expect our methodology will evolve over time as we learn from the process. We are committed to being completely transparent about our criteria for this distinction. View latest methodology and historical list of all methodologies. The new methodology will go into effect for all Voice of the Customer documents published in November 2022 and later.

What are the inclusion criteria in a VOC?

To be included in the “Voice of the Customer” report for a Gartner Peer Insights market, a vendor must have a product that aligns to that market as determined by Gartner Peer Insights and have 20 or more eligible reviews during the specified 18-month submission period. Please refer VOC methodology for full details.

How and when is market average calculated?

The average market rating is determined by all eligible reviews submitted in the 18-month review period and therefore can change until the end of the review period as reviews are moderated and published. Vendors can view whether they are above, below, or in-range of market average for both axes on the Standings Page (updated daily). For more information, see the detailed methodology.

Does the overall market rating on the Y-axis use the same calculations as what we will see with the new site scoring updates?

Voice of the Customer Y-axis calculations use a specified 18-month time period with a lower weight for the oldest 6 months of that period. The star rating displayed on the Gartner Peer Insights reviews page shows either past 12 months or an all time average, with a lower weight for earlier years. Thus, the time periods for the Voice of the Customer and the Gartner Peer Insights reviews page will typically be different. For more information, see the detailed methodology.

Is there a limit to the maximum number of vendors that can be listed in a Voice of the Customer document for a market?

Yes, there is a limit. The Voice of the Customer document was specifically created to help end-users shortlist vendors in their buying/selection process. Currently, Gartner Peer Insights limits the number of vendors included in the document to 25. If more than 25 vendors meet the inclusion criteria, then 25 vendors with the highest number of eligible published reviews will be included. If two or more qualified vendors with the same number of eligible reviews tie for the 25th position, all vendors in the tie will be included in the document.

For more details, please go to the Voice of the Customer Methodology page.

What are the inclusion criteria for Category Cuts?

A vendor’s inclusion in category cuts will remain independent of representation in the overall Voice of the Customer i.e, in a scenario where a vendor is excluded from ‘Voice of the Customer’ due to the maximum vendor limit, the vendor will continue to exist in category cuts if eligibility criteria for the category are met. See the category distinction methodology here.

What is User Interest & adoption?

Please refer to the Voice of the Customer methodology here for full details

How are vendors/products included in a VOC report?

A dedicated Peer Insights content team evaluates whether a technology provider’s products and/or services align with the Peer Insights markets based on a series of market guidelines defined by Gartner experts. The Peer Insights markets are periodically reviewed throughout the year to ensure alignment is maintained. For more details, refer to the Peer Insights listing guidelines.

Are reviews for products that are no longer sold included in the distinction analysis? What about reviews for products that are in beta test?

Reviews for products that are in production / general availability, along with beta products are included in calculations for the Voice of the Customer. Reviews for legacy products (that are no longer sold by the vendor) are not included in these calculations.

Why are you excluding reviews from companies that are less than 50M USD in annual revenue?

Gartner Peer Insights was developed with the global enterprise customer in mind. We define enterprise customers whose company sizes are greater than 50M USD in annual revenue.

For how long is the distinction current?

The Voice of the Customer document and Customers’ Choice distinction will be current for 15 months or until the next version of the Voice of the Customer for the market is published. Documents that are no longer current will remain visible on Gartner.com with an “Archived” tag. Our schedule for Voice of the Customer publication is determined based on the publish date of the corresponding MQ/MG and is subject to change based on the research calendar at the discretion of the Peer Insights team in conjunction with Research.

Right now the reviews come sourced by Gartner as well as from vendors’ sourcing links. Aren’t vendor-sourced reviews inherently biased because vendors select their best clients and offer them incentives to write reviews?

Our interviews with reviewers suggest that most of them are motivated by a desire to give back to the broader IT community. We’ve decided to include honest vendor-incentivized reviews as part of the relevant data set used to determine the distinctions. Our vetting process for reviews is extensive (see our FAQs for a summary), and we investigate any anomalies to ensure that users can trust the reviews in Gartner Peer Insights.

Can I see my status for eligibility towards Customers’ Choice in a specific market?

When a specific Peer Insights market is about 2 months out from a Voice of Customer sourcing deadline, the vendor Dashboard on the Technology Provider Tools portal will update to show eligibility towards inclusion in the Voice of the Customer and Customers’ Choice in that market. This information is dynamic, updating daily. Please note, the information shown in the Dashboard is not final and may change after sourcing periods close as part of our moderation and auditing process.

Are vendor ratings rounded/truncated in the VOC?

Yes. The market average Overall Rating is calculated by averaging all eligible vendors’ Overall Ratings. All vendor ratings and the market average Overall Rating are then truncated to the tenths place (e.g., ‘4.512’ becomes ‘4.5’). Vendors whose truncated scores meet or exceed the truncated market average are positioned in the upper-half quadrants, while vendors below the truncated market average are positioned in the bottom-half quadrants. These truncated scores are used to position vendors in the Voice of the Customer quadrants and are listed in Figure 2 in the Voice of the Customer document.

Are more recent reviews weighted more heavily in VOC calculations and scores?

Reviews from the first 6 months of the 18-month submission eligibility period will be given 50% weight in all calculations for both axes. This means that one review from the first 6 months will have half the impact of one review from the next 12 months in determining a vendor’s X-axis and Y-axis scores. The 18-month review period allows for a larger sample size of reviews and greater confidence in each vendor’s placement on the quadrant. The 50% weighting for reviews in the first 6 months allows more recent reviews to factor more heavily in determining a vendor’s placement.

How is sensitivity to review volume managed?

An asymptotic function is applied to reduce sensitivity to small changes in review count and review market coverage, especially for vendors with a large number of reviews. This means that for a vendor with a low review count (e.g., 20 reviews) who adds 10 more reviews will see a larger increase in review volume and review market coverage than a vendor with a high review count (e.g., 100 reviews).

What if I have more questions?

If you have any further questions, send us an email at PeerInsights@gartner.com and we will get back to you as soon as we can.

Resources for Users

Other Helpful Resources!

Need additional assistance? Reach out to your Program Manager or email peerinsightsvendorsuccess@gartner.com for assistance!

Did this answer your question?