I learned about the results of a fascinating BCG study conducted by many social scientists from the world’s top universities and institutions. Here is my big take away from the report and findings: Workers who use AI produced 40% higher quality results than those who didn’t. This was the first prominent research I’d seen on how ChatGPT can help to innovate how we work and be more productive.
The study was conducted in collaboration with Boston Consulting Group (BCG) and analyzed 100s of consultants across 18 different work tasks. Those using AI finished 12.2% more work tasks.They also completed tasks 25.1% faster. Those who used AI also produced 40% higher quality results than those who didn’t. Consultants with the lowest scores at the start of the study saw the biggest jump in performance when allowed to use AI.
Researchers who conducted the study included social scientists from Harvard, MIT, and Warwick University, as well as researchers from BCG. The test took hundreds of consultants and randomized whether they were allowed to use AI. The consultants with permission to use AI were given access to GPT-4.
Here is a link to a pdf of the report published by Harvard Business School. This report published on September 22, 2023 is titled “Navigating the Jagged Technological Frontier: Field Experimental Evidence of the Effects of AI on Knowledge Worker Productivity and Quality.”
Here is our AI summary of the key insights from the report.
The BCG AI study, conducted in collaboration with Boston Consulting Group, offers a detailed analysis of the impact of AI, specifically ChatGPT (GPT-4), on the productivity and quality of work among professional consultants. Here’s an article that summarizes the key findings and implications of this groundbreaking study:
Revolutionizing Consulting: The BCG AI Study’s Insights
Introduction: The Advent of AI in Professional Services
With the public release of Large Language Models (LLMs) like OpenAI’s ChatGPT, the professional world has been abuzz with the potential of AI. This study, involving 758 consultants from BCG, delves deep into how AI, especially GPT-4, impacts complex, knowledge-intensive tasks.
Experiment Design: Assessing AI’s Impact
The study’s unique approach involved two randomized experiments to assess how integrating GPT-4 into traditional workflows might augment or disrupt the work of highly skilled professionals. The participants were divided into three groups: those without AI access, those with GPT-4 AI access, and those with GPT-4 AI access plus a prompt engineering overview.
Inside the Frontier: Boosting Quality and Productivity
Tasks deemed within the AI’s capability (“inside the frontier”) saw significant productivity and quality improvements. For example, in a task involving the development of a new footwear concept, consultants using AI were more productive by 12.2% and completed tasks 25.1% faster. The quality of work was also higher, with a more than 40% increase compared to the control group. Interestingly, consultants below the average performance threshold improved their performance by 43%, while top performers saw a 17% increase.
Outside the Frontier: The Challenges
Conversely, a task designed to fall outside of AI’s current capabilities (“outside the frontier”) showed different results. Here, the study used a business case involving a spreadsheet and company interviews. In this scenario, consultants using AI were 19 percentage points less likely to produce correct solutions, highlighting the limitations of AI in complex, nuanced tasks.
Navigating the AI Frontier: Centaurs and Cyborgs
A fascinating outcome of the study was the emergence of two distinctive human-AI integration patterns: “Centaurs” and “Cyborgs.” Centaurs strategically divided labor between themselves and AI, while Cyborgs integrated their efforts intricately with AI, often at a granular sub-task level. These behaviors underscore the varying effectiveness of AI integration strategies in professional contexts.
Conclusion: The Future of AI in Professional Services
This BCG study offers a nuanced view of AI’s role in high-end professional tasks. It demonstrates that while AI can significantly enhance performance and quality within its capabilities, it also poses challenges when applied to tasks beyond its current scope. The study’s insights into Centaur and Cyborg behaviors provide a roadmap for organizations to effectively integrate AI into their workflows, maximizing its benefits while being mindful of its limitations.
For some great commentary on this study, check out Ethan Mollick’s blog post about it. Ethan was one of the authors of the report.
See this ChatGPT Prompt Tips Training post for more on how to utilize ChatGPT or Generative AI tools.