Geyto Research · Juan Ferreiro · Essay · · 2 min read
How to measure technology adoption beyond logins
Usage dashboards count logins, not changes in the work. What research on measuring system use says and which questions to add.
In brief: usage dashboards show how many people log in to a system and how many transactions they record. That is useful information, but it does not say whether the work changed. A technology can have good usage indicators and coexist with the parallel spreadsheet, the paper form or the decision that is still made outside it. Measuring adoption also means looking at the task, the routine and time.
The problem with "usage"
For years, information systems research used "usage" as a central variable: frequency, duration, number of features used. Andrew Burton-Jones and Detmar Straub pointed out a problem: usage measured this way does not distinguish between using a system superficially or in depth, out of obligation or because it improves the work.
They proposed measuring usage by considering three elements together: the system, the user and the task it is used for. Two people can record the same number of transactions and be doing very different work with the system.
Success is not only usage
William DeLone and Ephraim McLean proposed a model of information systems success that separates several dimensions: system quality, information quality and service quality, use, user satisfaction and net benefits. The useful idea is that usage is one part of success, not success itself.
Signals dashboards do not show
- Parallel routines: the spreadsheet that is still filled in "just in case".
- Compliance use: entering data at the end of the day to comply, while the real work is coordinated through another channel.
- Exceptions outside the system: difficult cases are resolved by phone or by message.
- Differences between sites: one site adopted the change and another, with the same system, did not.
- Decay: usage rises after launch and falls when support is withdrawn.
Questions to add to measurement
- Which tasks are done differently today than before the change?
- Which previous routines are still alive, and what do they solve?
- Where are the decisions the system was meant to support being made?
- Does the change hold once the project team is gone?
- Are the differences between teams or sites explained by the capacity available, or by what the initiative asks for in each place?
What to do with this
- Keep usage indicators as an early signal.
- Add observation of the work and conversations with the people who do it.
- Measure over time, not only at project close.
- Compare sites to find where the change takes hold and why.
More in Same technology, different outcomes and in the glossary.
Geyto Research publishes essays, evidence reviews and market notes on what happens after go-live.
Frequently asked questions
How do you measure the adoption of a system?
Usage indicators (logins, transactions, active users) are a starting point. To measure adoption you also need to know whether the work changed: which tasks are done differently, which previous routines are still alive and whether the change holds over time.
Why is usage not enough as a measure?
Because a system can be used superficially, under obligation or in parallel with the old way of working. Burton-Jones and Straub showed that a measure of usage has to take into account the system, the user and the task.
What is the DeLone and McLean model?
It is a model of information systems success that distinguishes system quality, information quality and service quality, use, user satisfaction and net benefits.
Sources
- Burton-Jones, A. and Straub, D. W. (2006). Reconceptualizing system usage: An approach and empirical test. Information Systems Research, 17(3), 228–246.
- DeLone, W. H. and McLean, E. R. (2003). The DeLone and McLean model of information systems success: A ten-year update. Journal of Management Information Systems, 19(4), 9–30.
Share