Let’s go over some Gen AI productivity studies. I figured we could double click on some of them as there haven’t been tons and it’s worth taking a look. This is particularly useful for leaders making plans to deploy GenAI in their enterprises and are prioritizing the opportunities from near term to medium term.
Let’s start with one of the early Gen AI productivity studies of 2023: “GPTs are GPTs: An early look at the labor market impact potential of large language models“.
This study is not useful for prioritizing and organizing a program of work at a Business Unit or Company level. However, it gives a broader swath of the opportunity at the level of an entire economy. Let’s call it the 30,000 feet view of productivity. Later, I will talk about more detailed empirical studies to see what productivity looks like when it’s amped up.
In this study, the researchers were trying to capture the potential impact of LLMs (text based AIs) across the entire economy. The way they did this is by using the “onet” database which deglosses the entire economy into 1000 professions. On average, each profession carries out about 20 tasks, resulting in 20,000 tasks across the entire economy.
Then, each task was judged by an analyst in terms of whether it would be made more productive by LLMs alone or by LLMs with some enhancing technology. Here is where some imagination is required as the analyst had to imagine the task being carried out and whether LLMs would make these tasks easier to do.
The results
What they found out when they tabulated all this analysis is that 80% of professions could be 10% more productive with LLMs. That is, imagine if a task is to write an email, then with AI tools embedded in the mail software’s editor, the task would be at least 10% easier. They also found that about 20% of professions could have about 50% of their tasks impacted. This is huge!
80% of professions could be 10% more productive with LLMs
To put things in perspective, normal annual productivity gains tend to be around 1.5-2% a year. This make numbers like 50% gains to take about a generation to achieve (25 years or so at 2% per year). Productivity measures the increase in output compared to the increase in hours worked. If you produce more while working less hours, then that is an increase in productivity.
One thing to remember is that this study measured the impact of LLMs on tasks and professions. It did not include multimodal models such as those we are seeing today which have vision, speech, or listening capabilities. With those more extensive capabilities, the productivity opportunity would be greater.
Digging into Gen AI Productivity Studies: a Key Takeaway
Some take aways for CxOs looking to implement GenAI in their organizations:
- consider impact by tasks and by roles,
- prioritize introducing it to those roles that are most exposed to GenAI productivity opportunities. This way, the prioritization aspect of your program of change will matter.
FYI, the authors are OpenAI researchers Tyna Eloundou, Sam Manning, Pamela Mishkin, and Daniel Rock (UPenn). Very well structured work.