Harvard Study: GPT-4 Boosts Work Quality by Over 40%

Consultants that use OpenAI’s GPT-4 language model are “significantly more productive and produced significantly higher quality results” than those who do not, according to a new study from the Harvard Business School.

The study compared the performance of BCG consultants − those who used AI and those who did not. Researchers found that consultants who used GPT-4 produced 40% higher quality work, according to the paper, ‘Navigating the Jagged Technological Frontier,’ by researchers from Harvard, MIT, Wharton, BCG and Warwick Business School.

Those who used GPT-4 saw a 25% increase in speed and 12% rise in task completion.

According to the paper, consultants of all skill levels were found to have benefited from their use of AI, but the lowest performers had the biggest boost - 43% increase versus 17% for top performers.

However, while the quality of work got a big boost, the results also had less variety. “While GPT-4 aids in generating superior content, it might lead to more homogenized outputs,” the researchers said.

These findings were for tasks “within the frontier” of GPT-4, meaning they are easily done by AI. For the experiment, consultants were tasked with creative product innovation and development – to brainstorm new beverage concepts, pick the most viable product and create a go-to-market plan.

Related:The 'Remarkable' AI Model Behind ChatGPT’s New Multimodal Powers

Jagged frontier

The researchers also tested “outside the frontier” tasks, those which the AI model would struggle to accomplish. In the paper, this was the capability to pick up subtle clues in interview notes that accompany data on a spreadsheet.

In the experiment, consultants were asked to come up with actionable strategies to help boost the business of a company. Their tasks were to analyze the company’s channel performance through interviews and financial data, and then make recommendations to the CEO.

The metric for performance was coming up with the right strategy. Here, consultants using AI were 19 percentage points less likely to get it right. “Professionals who had a negative performance when using AI tended to blindly adopt its output and interrogate it less,” the authors observed.

Why is this important? Professionals using AI must know how to skillfully use this technology to make it pay off in spades. Otherwise, using AI for tasks it is not good at can actually hurt employee performance. “Outside of the frontier, AI output is inaccurate, less useful, and degrades human performance,” the authors wrote. They call this uneven performance, the “jagged technological frontier.”

Related:Intel Taps BCG to Bring Generative AI to the Enterprise

The Harvard study covered some 758 consultants, around 7% of BCG’s workforce. Tasks analyzed spanned a consultant’s daily work, including creativity, analytical thinking, writing proficiency and persuasiveness.

Centaurs and Cyborgs

The paper grouped AI users into two groups – ‘Centaurs,’ who divided and delegated tasks between themselves and the AI, and ‘Cyborgs,’ who integrated their workflow with the AI.

The centaur approach is named after the mythical creature that is half-human and half-horse, with the researchers opining that AI could also be a mixture of human and machines.

“Users with this strategy switch between AI and human tasks, allocating responsibilities based on the strengths and capabilities of each entity,” the report reads. “They discern which tasks are best suited for human intervention and which can be efficiently managed by AI.”

Whereas for cyborgs – named after hybrid human-machine beings as envisioned in science fiction literature – this approach was more about integrations, such as having the AI complete a work they started.

“Cyborg users don’t just delegate tasks; they intertwine their efforts with AI at the very frontier of capabilities,” they wrote. “This strategy might manifest as alternating responsibilities at the subtask level, such as initiating a sentence for the AI to complete or working in tandem with the AI.”

The concept of centaurs and cyborgs shows that there are multiple paths to effective collaboration with AI and that adopting the right practice for the right use case can spread the benefits of AI more widely. Different use cases will require different needs, and adopting either a centaur or cyborg approach may form as part of customized integration strategies that businesses can tailor to their specific needs and capabilities.

“It is clear that the best approaches to using AI are not fully understood and need to be deeply examined by scholars and practitioners,” the researchers wrote.

However, an immediate risk is that employers will stop delegating work that AI is good at to junior workers, “creating long-term training deficits,” they said. Expertise needs to be built through formal education, on-the-job training and upskilling of employees.