How to Capitalize on Generative AI

Source: https://hbr.org/2023/11/how-to-capitalize-on-generative-ai

Business leaders are struggling to understand how seriously they should take the latest phenomenon in the world of artificial intelligence: generative AI. On one hand, it has already displayed a breathtaking ability to create new content such as music, speech, text, images, and video and is currently used, for instance, to write software, to transcribe physicians’ interactions with their patients, and to allow people to converse with a customer-relationship-management system. On the other hand, it is far from perfect: It sometimes produces distorted or entirely fabricated output and can be oblivious to privacy and copyright concerns.

Is generative AI’s importance overblown? Are its risks worth the potential rewards? How can companies figure out where best to apply it? What should their first steps be? To provide guidance, this article draws on our research comprising studies of specific generative-AI projects and broad analyses of how the technology will affect tasks and jobs throughout the economy.

A large enterprise-software company that one of us (Erik) studied along with Lindsey Raymond and Danielle Li of MIT illustrates that there are ways to both reap the benefits of generative AI and contain its risks. The company’s customer-service agents, who assist people via online chats, faced a common challenge: New hires needed several months to get up to speed on how to answer technical questions and deal with confused customers, but many quit before they became proficient. The company saw generative AI as a solution. It engaged Cresta (which Erik has been advising), a generative AI start-up, to implement two kinds of artificial intelligence. The first was a large language model (LLM)—designed to understand and respond to humans in their own words—that “listened in” on the chats. It was fine-tuned to recognize phrases that led to good customer-service outcomes in various situations. But because of the risk of confabulations, or plausible-sounding but incorrect responses, the system also used a machine-learning technique called in-context learning, which drew answers from relevant user manuals and documents.
The LLM monitored the online chats for specific phrases, and when one of them occurred, it based its responses on the information in the in-context learning system. As an additional safeguard, it didn’t respond to queries directly. Instead human agents were free to apply their common sense in deciding whether to use or ignore the LLM’s suggestions.
After a seven-week pilot the system was rolled out to more than 1,500 agents. Within two months multiple benefits appeared. Both the average number of issues resolved per hour and the number of chats an agent could handle simultaneously increased by almost 15%; the average chat time decreased by nearly 10%; and an analysis of the chat logs showed that immediately after the new system was implemented, customer satisfaction improved. Expressions of frustration declined, for example, as did TYPING IN ALL CAPS.
It’s especially interesting that the least-skilled agents, who were also often the newest, benefited most. For example, resolutions per hour by agents who had been among the slowest 20% before introduction of the new system increased by 35%. (The resolution rate of the fastest 20% didn’t change.) The generative AI system was a fast-acting upskilling technology. It made available to all agents knowledge that had previously come only with experience or training. What’s more, agent turnover fell, especially among those with less than six months of experience—perhaps because people are more likely to stick around when they have powerful tools to help them do their jobs better.