Geyto Research · Juan Ferreiro · Essay · · 2 min read

Trusting the system. Use, misuse and disuse of automation

Trusting a system too much or too little has costs. What research on trust in automation says and what it means for adopting AI.

In brief: when an organization brings in a system that suggests or decides, the issue is not only whether people use it. How much they trust it also matters. Trusting it too little leads people to ignore a useful tool. Trusting it too much leads them to accept errors without seeing them. Research on automation proposes a different goal: calibrated trust, which matches what the system can actually do.

Four ways of relating to a system

In 1997, Raja Parasuraman and Victor Riley sorted the relationship between people and automation into four words:

  • Use: turning to the system when it helps.
  • Misuse: trusting it too much, accepting its outputs without checking or no longer monitoring it.
  • Disuse: not using it when it would help, for example out of distrust after a few errors or false alarms.
  • Abuse: automating without considering the effect on the people who work with the system.

The classification is useful because it shows that adoption is not a matter of "more or less use". A system that is used a lot can be misused.

Calibrate, don't maximize

John Lee and Katrina See reviewed the research on trust in automation and put forward a central idea: the goal is appropriate trust. That is, trust that matches the system's real capability in each situation.

For that, people need to understand what the system does, how it reaches its outputs and under what conditions it fails. Without that information, trust is formed by intuition, by isolated experiences or by what others say.

Why AI makes this more urgent

Generative AI tools produce outputs that look correct even when they are not. That makes errors harder to detect and over-trust easier. At the same time, one bad experience can push an entire team into disuse.

Lisanne Bainbridge warned forty years ago that automation leaves people with the hardest tasks: supervising, detecting the error and acting in the exceptional case. If the tool does the routine work, practice for those moments is lost.

Trust as a situated capacity

Calibrated trust is not an attitude you can ask for in an announcement. It is a capacity built where the work happens:

  • knowing the cases in which the system gets things wrong;
  • practising verification with real examples;
  • knowing who decides when the system and the person disagree;
  • reviewing errors together without looking for someone to blame.

What to do with this

  • Show the system's limits from the start, not only its benefits.
  • Include cases where the tool fails in training.
  • Measure misuse and disuse too, not only frequency of use.
  • Define clearly who is accountable for each result.

More in Generative AI at work and in Geyto Learning.

Geyto Research publishes essays, evidence reviews and market notes on what happens after go-live.

Frequently asked questions

What is automation misuse?

It is relying on a system more than its performance justifies: accepting its outputs without checking them, or no longer monitoring what it does. Parasuraman and Riley distinguished it from disuse, which is not using the system when it would help.

What is well-calibrated trust?

It is trust that matches what the system can actually do in each situation. Lee and See argued that the goal is not to trust more, but to trust to the right degree.

How is that trust built?

By showing how the system works and where it fails, by practising with real cases, and by giving people the means to verify the results.

Sources

  1. Parasuraman, R. and Riley, V. (1997). Humans and automation: Use, misuse, disuse, abuse. Human Factors, 39(2), 230–253.
  2. Lee, J. D. and See, K. A. (2004). Trust in automation: Designing for appropriate reliance. Human Factors, 46(1), 50–80.
  3. Bainbridge, L. (1983). Ironies of automation. Automatica, 19(6), 775–779.

Share