Analyze your copilot's customer satisfaction metrics

Important

Power Virtual Agents capabilities and features are now part of Microsoft Copilot Studio following significant investments in generative AI and enhanced integrations across Microsoft Copilot.

Some articles and screenshots may refer to Power Virtual Agents while we update documentation and training content.

The Customer Satisfaction tab of the Analytics page provides a detailed view of customer satisfaction (CSAT) survey data, including the average CSAT score, primary user query themes, and actionable insights on drivers of satisfaction or dissatisfaction with your copilot's responses.

By default, the page shows you key performance indicators for the last seven days. To change the time period, use the date pickers at the top of the page. You can retrieve data for any period within the last 45 days.

Customer satisfaction score

The Customer satisfaction score chart provides a graphical view of the average of CSAT scores for sessions in which customers respond to an end of conversation request to take the survey. The CSAT survey asks customers to rate their experience on a scale of 1 to 5. If an end user responds to more than one survey in the same session, only the most recent is used.

This chart also provides a period-over-period change indicator. For example, when you select a three-day period, the indicator shows the percentage change relative to the three days prior to the selected period. Please note that the period-over-period indicator will only be shown if your copilot has CSAT survey data available for the prior period. If there are no CSAT survey data available for the same trailing period of time relative to what is selected in the filter, then the period-over-period indicator will not be shown.

CSAT survey response rate

The CSAT survey response rate chart shows the number of end of conversation CSAT surveys that were presented and the percentage of surveys that were completed.

Customer satisfaction breakdown

The Customer satisfaction breakdown chart shows that percentage of sessions that were satisfied, dissatisfied, or neutral in the selected time period. The Customer satisfcation status pane provides more detail on the various signals used to determine session satisfaction status.

Customer satisfaction status

The Customer satisfaction status chart provides key insights on the themes users have searched for, and the satisfaction level of the users about the copilot's responses. Sessions with similar themes are grouped together. The chart shows the number of sessions for each theme during the selected period, and the percentage of these sessions that were satisfied or dissatisfied. Sessions that were neither satisfied nor dissatisfied are considered neutral sessions and don't appear on this chart.

The theme of a given session is derived using ML models. Before being sent to the analytics dashboard, themes are processed to remove any personally identifiable information (PII) or sensitive information, such as phone numbers. Additionally, if themes contain profane or harmful language, these themes are masked.

Hover over each segment of the chart to see the specific satisfaction or dissatisfaction drivers. If any of the criteria are true for a given session, then the session is classified as satisfied or dissatisfied:

• A session is considered dissatisfied if:

  • The user gave two stars or less at the end of conversation survey.
  • The user was asked to rephrase their query twice or more than twice in the (system fallback topic).
  • The user abandoned the session.
  • The user escalated the session to a live agent.
  • The overall sentiment of the user about their conversation with the copilot is classified as negative. Sentiment is determined using a publicly available ML model finetuned for sentiment analysis.

• A session is considered satisfied if:

  • The user gave four stars or more at the end of conversation survey.
  • The user wasn't asked to rephrase their query more than once in the (system fallback topic).
  • The session was resolved.
  • The overall sentiment of the user about their conversation with the copilot is classified as positive.

Sessions that don't meet any of the above criteria are considered neutral sessions and don't appear on this chart.

On the Summary tab of the Analytics page, you can use the Information icons to learn more about engagement, escalation, abandon, and resolution rates.

Theme and session sentiment extraction

Copilot Studio uses natural language processing (NLP) techniques to extract themes and assign sentiment to a given copilot session.

For every session, Copilot Studio extracts themes from the first user utterance. Individual sessions with similar themes are aggregated, and appear as a single item on the Customer satisfaction status chart.

To evaluate session sentiment, the underlying NLP model is trained on public English language datasets. This involves analyzing the text of the session to determine whether the overall sentiment is positive, negative, or neutral, and preprocessing user queries to remove false positives. For example, this preprocessing ensures that a query such as "what is the best option?" isn't classified as positive solely because the word "best" appears in the query.