Building a Robust Virtual Agent (VA) Rollout Strategy for DSPs

Proven methods to increase VA containment & customer satisfaction

The virtual agent’s market is at an all-time high and is garnering more and more interest with each passing day. It is establishing itself as “the must-have” solution for digital service providers (DSPs) seeking to improve customer experience, reduce call center costs, and optimize time to serve.

But are these virtual agents living up to the hype they are creating? Gartner has placed them in “trough of disillusionment” in its hype cycle. Deployment of VAs are not done in a proper way, which is the primary reason for this situation. As a result, they don’t reach the required confidence levels and are not able to capture the right customer intent.

This article describes in detail about building a robust VA roll-out strategy for DSPs. It provides the top 10 considerations that can help DSPs containing their customers within VA interaction and increase overall customer satisfaction.

1. Choose the right use cases for roll-out – kickoff roll-out with self-service flows

The inbound calls that DSP’s call center receives are categorized as customer service inquiries, technical troubleshooting, or sales based. The use case that these calls invoke can be any one of the following flows:

Self-Service, Data-Driven, or Transactional Flows.

Recommendations:

1. NLP models get trained eventually and accuracy increases over time. Hence, start with simpler use case such as Self-Service for rollout and then gradually move to more complex use cases such as Data-Driven and Transaction services
2. Generate confusion matrix to measure the accuracy, precision, and recall of the NLU model.
3. Improve precision by training NLU with relevant intents and recall by adding more training utterances.

2. Consider the complexity of intents by analyzing the length of conversation and time taken by the agent to complete the conversation

Understanding the complexity of intent is crucial for programming or training the VA. The complexity of intent can be assessed by length of the conversation, the average time taken by an agent to complete the conversation, and the hierarchy of intents & sub intents.

Recommendations:

1. Develop a machine learning-based ‘intent analyzer tool’ shown as below: 

2. Kickoff roll-out with high-volume and simple intents for quick ROI.

3. Consider variations of intents by analyzing its scope, lifecycle, and precursor

From the clusters identified, representative examples are taken for training the NLU, which doesn’t cover the entire scope of that intent.

Take examples from the cluster based on the following criteria to holistically account all the variations of intent across the spectrum.

1. Scope of Intent: Understand where the intent could be applicable within the scope of the service.
2. Intent Lifecycle: Understand and include variations from before, during, and post scenarios for the intent.    
3. Intent Precursor: Consider the reasons or background issues leading to that intent.

This improves containment by increasing precision and recall.

4. Introduce fine-grained, context, and customer data-based routing guide

The routing guide is the first set of navigational options that VA presents to the customer on the invocation. Customers may not always know where to find the information they are interested in. Through a series of qualifying questions, users are routed to the appropriate location.

Recommendations

1. Present fine-grained options: The optimal number of questions that are neither too broad nor narrow.
2. Present questions based on customer account information and relevant to the context.
3. VA response flexibility: VA should be able to answer questions which customer asks outside of the scenarios mentioned above.
4. VA should be trained for different scenarios such as spelling mistakes in customer response, questions asked by customers even before VA presents options etc.

5. Design a chat interface which minimizes strain on users and is also visually appealing

Chat interface with UI components that are functionally and visually appealing enable and enhance the conversation flow between VA and users.

Recommendations

1. Minimizes ambiguity: By providing UI components like date picker, radio options, lists, tree views, and reduces back & forth messages.
2. Ability to undo & redo: VA builds context with every user response and to clear misinterpretation.
3. Enable visual response: Reduces strain on user and burden of interpretation on NLU.
4. Conversation route: Gives visibility into the direction of chat & goal and should be designed to enable the user to ‘recognize’ instead of ‘recall’.

6. Measure the impact of VA interaction on customer experience to improve its responses

A mechanism should be in place to continuously monitor and improve NLU performance even before it’s rolled out. Manually performing this task is not enough. Develop a “conversation rating engine” to perform programmatic reviews on a high volume of VA chats in real-time and also re-train the model as shown below

Conversation rating engine can improve NLU intent classification confidence by 2-4% weekly, review 200K chats per day and score the chats on the scale of 1-10 (1- worst, 10 – best)

7. Throttle traffic to minimize the negative impact on customer experience

It is crucial to decide the proportion of customer chat traffic that would be diverted to the VA while rolling out. In the early stages of the roll-out, the VA containment will remain low. i.e., most of the conversations will be diverted to live agents by VA.

Recommendations

1. Off- business hours: Start the roll-out in off-business hours to take advantage of relatively less traffic, optimizing limited customer exposure.
2. Agent- driven roll-out: Launch VA in an agent-driven mode and increase the traffic when VA accuracy increases.
3. Agent availability: Plan for all scenarios (Agent available, agent not available, business hours, non-business hours) when VA invokes agent.

8. Provide adequate training time to increase NLU’s response effectiveness

It is necessary to provide enough training time to expose NLU to highly varied human conversations. It is rare to see single-digit containment efficiency in the first few weeks of VA roll-out.

Recommendations

1. Re-train to remove confusion between existing intents.
2. Correct the intent errors i.e., Overlapping, High–recall, and low Precision Intents.
3. Add new intents and out-of-scope intents. Also, add training to improve precision and confused intents- to clarify their boundaries.
4. Combine the confused intents and distinguish using entities.
5. Monitor the NLU threshold confidence – by plotting intent confidence vs. error graph.

9. Perform agent-driven roll-out to give on-the-job training to the VA without negatively impacting customer experience

In the conventional rollout approach, right after the training VA is deployed directly to face customer questions. This will have a high error-rate. Hence, it is recommended to perform an agent-driven rollout.

Recommendations

To follow the Agent-Driven roll-out method where the live agent can observe and alter if needed, the responses coming from NLU. Eventually, this should advance to a stage where the agent is observing multiple chats and intervening only in the cases of escalation.

10. Measure performance – technical, qualitative and business metrics for holistic improvement

A metrics dashboard should be implemented to provide VA key performance indicators and other strategic data at a glance. It should include three types of metrics – system/technical, quality, and business, providing a holistic view of VA State.

These key considerations can help DSPs in achieving:

1. 30-35% increase in customer containment within 10 weeks of roll-out.
2. 15-20% increase in NPS and CSAT scores within 3 months of roll-out.
3. 12% reduction in average handling time in 6 months of roll-out.

By Sathya Ramana Varri C – Senior Director – Delivery, Prodapt 

Sathya is a Senior Director & heads the AI/ML and Intelligent Automation delivery for the largest US telecom service provider player in US for Prodapt, a global leader in providing software, engineering, and operational services to the communications industry. He has 20+ years of experience spread across various domains/technologies. Sathya is instrumental in several customer experience, intelligent automation & digital transformation initiatives.