Geyto Research · Mariano D'Angelo · Evidence review · · 2 min read
Generative AI at work. What the first field studies show
Two field studies on generative AI show real but uneven gains. Adoption depends on the task, on experience and on the judgement to use it.
In brief: the first field studies of generative AI at work show real gains, but not even ones. They depend on the task, on the experience of the person using it, and on knowing when the tool helps and when it does not. For an organization, that means AI is an adoption problem: access to the tool is the beginning, not the result.
A customer support center
Erik Brynjolfsson, Danielle Li and Lindsey Raymond studied the introduction of a generative AI assistant in the customer support operation of a software company, with more than five thousand agents. Access to the tool increased productivity, measured as issues resolved per hour, by 14% on average. For novice or less-skilled workers, the improvement was 34%. For the most experienced, the effect was minimal.
One possible reading is that the tool spreads the practices of the best agents to those who are just starting. It is a learning effect, not only a speed effect.
A jagged frontier
Fabrizio Dell'Acqua, Ethan Mollick and their colleagues ran an experiment with 758 consultants from Boston Consulting Group. On tasks within what AI does well, those who used it completed more tasks, faster and with better quality. On a task chosen to fall outside that frontier, those who used AI were less likely to reach the correct answer than those who did not.
The authors called it a "jagged technological frontier": AI is very good at some tasks and poor at others of similar difficulty, and the boundary is not obvious to the person using it.
The old irony of automation
In 1983, Lisanne Bainbridge described the "ironies of automation": the more automated a system is, the more important and the harder human intervention becomes when something goes wrong. The person supervising loses practice in exactly what they have to do in the exceptional cases.
The studies on generative AI show a current version of the same problem. The value of the tool depends on the user's judgement about when to trust it and when to check.
What changes in the adoption question
AI does not change the underlying question: what does this change ask, and what capacity is there to sustain it? It makes the question sharper:
- The demand is less visible. It is not about learning new screens, but about judging outputs that look correct.
- Capacity is uneven. Novices and experts get different effects from the same tool.
- Handoffs change. New steps appear between people and systems: reviewing, correcting and deciding who answers for the result.
What to do with this
- Map the tasks before deploying: where AI helps and where it does not, in this specific work.
- Build judgement, not only use: exercises with cases in which the tool gets it wrong.
- Define who reviews and who answers for what the AI produces.
More in AI makes the question sharper and in Geyto Learning.
Geyto Research publishes essays, evidence reviews and market notes on what happens after go-live.
Frequently asked questions
Does generative AI increase productivity?
In field studies, yes, for certain tasks. In a customer support center, access to an assistant increased issues resolved per hour by 14% on average, and by 34% for novice or less-skilled workers (Brynjolfsson, Li and Raymond).
What is the jagged technological frontier?
It is the idea that AI performs very well on some tasks and poorly on others of similar difficulty, without the boundary being obvious. In an experiment with 758 consultants, AI improved results on tasks inside that frontier and worsened them on a task outside it (Dell'Acqua et al.).
What does it mean for adopting AI in an organization?
That adoption is not solved by access to the tool. It requires judgement about when to trust it, changes in work handoffs, and learning on the job.
Sources
- Brynjolfsson, E., Li, D. and Raymond, L. R. (2023). Generative AI at work. NBER Working Paper 31161.
- Dell'Acqua, F., McFowland III, E., Mollick, E. R. et al. (2023). Navigating the jagged technological frontier: Field experimental evidence of the effects of AI on knowledge worker productivity and quality. Harvard Business School Working Paper 24-013.
- Bainbridge, L. (1983). Ironies of automation. Automatica, 19(6), 775–779.
Share