Can we get the benefits of Generative AI without replacing workers? Let’s see what this paper on GenAI productivity impact says. This is my second article on Generative AI studies, see the first one here.
Summary
GenAI Productivity Impact: Study reveals a 14% average productivity increase in contact centers.
Segmented Gains: Novice staff experiences a substantial 35% productivity jump.
Customer Sentiment Boost: Introduction of GenAI tools correlates with a significant increase in customer sentiment.
Manager Escalation Reduction: Study shows a noteworthy decrease in manager escalation instances.
Holistic Metrics: Emphasizes the importance of measuring factors beyond time savings, including customer sentiment.
Strategic Tool Deployment: Recommends a thoughtful approach, directing GenAI tools toward less experienced staff.
Business Implications: Businesses heavily reliant on contact center support may find substantial benefits from GenAI tools.
This paper by Erik Brynjolfsson, Danielle Li, and Lindsey Raymond (authors from Stanford and MIT), it’s focused on contact center productivity gains using GenAI tools.
You can probably transfer lessons from this study to IT contact centers, Retail contact centers, Financials contact centers, etc. So it’s applicable to multiple industry horizontals (CIO offices, Customer Support, etc). It’s also fairly representative as it covered the work of about 5000 agents. This is in contrast with some GenAI studies that have very small representation (50-100 subjects with control and experiment group).
Conclusions
Let me skip to the chase (if you don’t know the expression, it comes from a Steve McQueen movie!). The conclusion was that contact center staff gained a 14% productivity bump on average, but this result is masking an uneven distribution.
Low skilled staff (with less than 2 months experience on this job) gained a 35% jump while very experienced staff gained a negligible improvement. This is a huge impact given that in this industry, the authors estimate, there is a substantial turnover rate in staffing in contact centers, so there’s a large number of new on-the-job staff.
Low skilled staff (with less than 2 months experience on this job) gained a 35% jump while very experienced staff gained a negligible improvement.
The theory behind this difference in contact center agent segments is that inexperienced staff are being augmented with what to say (based on prior successful contact resolutions) while experienced staff are already very good at knowing what to say and need no help.
Surprising Results
One of the surprising results for me was the substantial increase in sentiment for customers. The authors studied the sentiment in the chats from customers and used sentiment measuring software to score it. Customer sentiment scored half a standard deviation higher when agents started using the GenAI support compared to when agents were not using it (before introduction). In addition there was a substantial decrease in manager escalation. (surprisingly, NPS scores didn’t budge).
So the takeaways are that the productivity impact is substantial (14% across the contact center), and even bigger if one focuses this on low productivity novice users (35%). One needs to be thoughtful of what segment to dedicate these tools towards (less experienced staff). And that one shouldn’t measure purely time savings, but also count customer sentiment, manager escalation, and other such measures.
For enterprises where contact / call center support is a substantial part of the business (b2c vs b2b) the benefits from introducing GenAI tools appears to be substantial.
What do you think about this study? Interesting? Was it what you expected?