Chatbot Order Selection Revamp

@Shopee, 2022

About Chatbot & Order Selection

Shopee Chatbot serves 250k users per day in 8+ regions, answering all customer inquiries about orders, logistics, post-sale service, and so on. Chatbot saves an excellent amount of the cost for customer service.

In this project, we aimed to bring down the drop rate, transfer to live agent rate and raise the resolution rate and CSAT of Order Selection flow.

A Bit Background

Over the past six months, the core metrics have been stagnant and remain at the industry average level. Chatbot team tried adjust the model, edited articles in Knowledge Base, but all seem not to be effective.

Choose a Research Method

Chatbot is characterised by large amount of data and small amount of information per piece of data. The product team has accumulated a lot of available data for research.

In contrast, the product team pays less attention to individual user experience or feedback than macro data fluctuation.

Given the above information, we decided to adopt a research method based on data analysis and supplemented by usability test.

Step 1 Mapping the User Journey and Find Out The Data We Need

Chatbot is a product with complex and interlaced flow, I mapped out the user journey to visualise the user flow and help us find out what data we need.

After mapping out the user journey, I asked BI team for the traffic,drop rate,resolution rate,and CSATof each node based on different flow. It also became key metrics that we used to measure the design outcome later.

Step 2 Form Hypothesis from Data and Journey Mapping

Data Anomalies




High drop rate at Order Selection node.







Drop rate at order selection node is higher for RR bots (11%) than Order bots (7%) for Click methods.




Lower Customer satisfaction (CSAT) at Return/Refund bot than Order bot.








arrow_forward







arrow_forward






arrow_forward
Possible Reasons

  1. the order card information confused user and result in hesitation;

  2. the user don’t know how to find other order;

  3. the user can’t find the wanted order because we don’t have it at all(hard to believe but possible)

  1. the user might not want to inquire about a specific order;

  2. our copywriting make the user believe that clicking on an order card means raise RR for that order.

  1. we send nonsense messages;

  2. the nature of that topic;

Step 3 Verification

Pair each hypothesis with one or more ways to verify:

At this step, we conducted several user testing, checked agent-customer chat history exported from live agent admin, communicated with BI team for more detailed data, and conducted fast A/B testing on UX writing. Full verification data here.

Step 4 Actions

Along with the movement of our research, solutions came quite naturally for each pain point.

Craft Design Solutions

1. Change information to help users recognise their order

2. Explore UI options to enhance affordance

3. Alternative flow to allow users back to any possible enquiries

4. Advice on UX writing

Result

With all A/B test, the final design we chose generates the highest performance of +3.71% in resolution rate, - 6.82% in drop rate, -2.73% in transfer to live agent rate. Link

Based on the traffic of the order bot in the last three months, the labor cost saved by bringing down the transfer to live agent rate is around 1 million USD/ year( based on ID salary).

We also did further analysis to understand why the final design outperforms other options. Link

What's More?

If you are interested in Chatbot , we can talk about some other related products such as:

  • Task Flow Editor: An Integrated operation management platform for customer servers to create & manage all chatbot flows;
  • Annotation Portal: A place for customer servers QA team to review and correct chatbot answer;
  • Data Insights: A data dashboard for Chatbot product team to check chatbot performance;