Testing & Evaluation
Error Identification & AnalysisSystems can be designed in ways where the likelihood of the occurrence of human error is increased or decreased. Human factors professionals seek to reduce the number of errors and the criticality of errors in order to improve safety and efficiency and to increase the likelihood of achieving the goals the system was designed to meet. This creates a need for tools to identify and classify errors. Categorizing errors by type makes it easier to understand what might be generating these errors and to focus on improving specific areas of a system.
|
What is Error?
Reason (1990) defines error as "all those occasions in which a planned sequence of mental or physical activities fails to achieve its intended outcome, and when these failures cannot be attributed to the intervention of some chance agency".
Two major categories of error are 1) slips that result from the incorrect execution of an action sequence and 2) mistakes that result from the correct execution of an incorrect action sequence
Norman's action theory (errors can occur at any stage, examples here for programming a medical device)
Types of slips (Zhang et al., 2004)
Goal slips
loss of activation - goal forgotten because of high memory load, delays, or interruptions
concurrent and sequential cross talk - correct goal distorted because of more common goal, activity with multiple tasks could have goals for different tasks mixed up
overflow of goal stacks - goals are too numerous to be kept in working memory
Intention slips
Same problems as goal slips but at the level of specific features of functions of a device
Action specification slips
associative activation - activation of similar but incorrect knowledge
description - incomplete or ambiguous specification of an intended action that is similar to a familiar action
failure of retrieval
situated activation (strong environmental stimulus replaces intention)
cross talk (interference from other parts of the task or other tasks)
Action execution slips
capture slip - routine overrides intended activity
double capture slip - unintended activation of a related strong action routine
perceptual confusion, deviation of motor skills, misfiring of action rules
Evaluation slips
Perception slips - lack of perception, misperception, mis-anticipation
Interpretation slips - default knowledge, confirmation bias, information overload
Action evaluation slips - loss of memory of goal, insufficient or ambiguous information, mixing up evaluations of different goals
Types of Mistakes
Execution mistakes
Goal mistakes - incorrect goals set by some means, can be caused by incorrect knowledge, incomplete knowledge, misuse of knowledge, biases and faulty heuristics, information overload, incorrect intention of how to achieve goal
Action specification mistakes - procedural mistakes, could be caused by lack of correct rules, overgeneralization, encoding deficiencies
Action execution mistakes - can be caused by misapplication of good rules and the dissociation between knowledge and rules
Evaluation mistakes
Perception mistakes - expectation-driven processing
Interpretation mistakes - incorrect interpretation of feedback caused by incorrect or incomplete knowledge
Action evaluation mistakes - incorrect knowledge leads a person to erroneously judge the completion or incompletion of a goal
Two major categories of error are 1) slips that result from the incorrect execution of an action sequence and 2) mistakes that result from the correct execution of an incorrect action sequence
Norman's action theory (errors can occur at any stage, examples here for programming a medical device)
- establishing the goal (set volume to be infused at 1000 cc)
- forming the intention (use keypad to enter 1000)
- specifying the action specification (press 1 0 0 0)
- executing the action (physically pressing 1 0 0 0)
- perceiving the system state (volume: 1000cc, with 1000 highlighted)
- interpreting the state (1000cc is displayed, but what does highlighting mean, has value been accepted?)
- evaluating the system state with respect to goals and intentions (determine if system has accepted volume, press key to start infusion)
Types of slips (Zhang et al., 2004)
Goal slips
loss of activation - goal forgotten because of high memory load, delays, or interruptions
concurrent and sequential cross talk - correct goal distorted because of more common goal, activity with multiple tasks could have goals for different tasks mixed up
overflow of goal stacks - goals are too numerous to be kept in working memory
Intention slips
Same problems as goal slips but at the level of specific features of functions of a device
Action specification slips
associative activation - activation of similar but incorrect knowledge
description - incomplete or ambiguous specification of an intended action that is similar to a familiar action
failure of retrieval
situated activation (strong environmental stimulus replaces intention)
cross talk (interference from other parts of the task or other tasks)
Action execution slips
capture slip - routine overrides intended activity
double capture slip - unintended activation of a related strong action routine
perceptual confusion, deviation of motor skills, misfiring of action rules
Evaluation slips
Perception slips - lack of perception, misperception, mis-anticipation
Interpretation slips - default knowledge, confirmation bias, information overload
Action evaluation slips - loss of memory of goal, insufficient or ambiguous information, mixing up evaluations of different goals
Types of Mistakes
Execution mistakes
Goal mistakes - incorrect goals set by some means, can be caused by incorrect knowledge, incomplete knowledge, misuse of knowledge, biases and faulty heuristics, information overload, incorrect intention of how to achieve goal
Action specification mistakes - procedural mistakes, could be caused by lack of correct rules, overgeneralization, encoding deficiencies
Action execution mistakes - can be caused by misapplication of good rules and the dissociation between knowledge and rules
Evaluation mistakes
Perception mistakes - expectation-driven processing
Interpretation mistakes - incorrect interpretation of feedback caused by incorrect or incomplete knowledge
Action evaluation mistakes - incorrect knowledge leads a person to erroneously judge the completion or incompletion of a goal
Why Use Error Identification & Analysis?
Human error introduces a large number of problems into a system. Reducing the amount of errors that occur through changes in design can improve efficiency, safety, usability, and user satisfaction. In some situations, the number or criticality of errors is so great that the tasks the system was designed to achieve become impossible. Error reduction (or ideally elimination) is obviously more important in some domains than others. Safety critical domains like nuclear and medical fields have a greater focus on human error than entertainment software.
When Use Error Identification & Analysis?
Error can be assessed at multiple points in the design cycle. Early on, when conducting a task analysis, designers may predict points of error and try to eliminate them through their initial designs. Inevitably, not all error will be predictable, and more errors will be found during the user testing stage. Ideally a design will be iterated numerous times and the number of errors will drop with each design. However, it is always possible that a change in the design that improves some aspects of usability ends up generating new errors.
How to Identify and Analyze Errors?
There are a tremendous number of ways to identify and analyze errors. Everyone seems to want to create their own system and develop an acronym to refer to it. Human factors professionals will likely find a way that works best for them.
Many error identification systems tie errors to the process of task analysis (often hierarchical task analysis), analyzing errors at every step of the task. Errors are sometimes classified based on type, as this can assist with removing the error from the system. Some error identification systems have been developed for specific domains, such as nuclear power.
The following table from Stanton et al. (2013) provides some examples of human error identification methods:
Many error identification systems tie errors to the process of task analysis (often hierarchical task analysis), analyzing errors at every step of the task. Errors are sometimes classified based on type, as this can assist with removing the error from the system. Some error identification systems have been developed for specific domains, such as nuclear power.
The following table from Stanton et al. (2013) provides some examples of human error identification methods:
References & Resources:
1. Reason, J. (1990). Human Error. Cambridge: Cambridge University Press.
2. Stanton, N.A., Salmon, P.M., Walker, G.H., Baber, C., & Jenkins, D.P. (2013). Human error identification methods. In. N. Stanton (Ed.), Human factors methods: A practical guide for engineering and design (pp. 139-212). Burlington, VT: Ashgate Publishing Company.
3. Zhang, J., Pate, V.L., & Shortliff, E.H. (2004). A cognitive taxonomy of medical errors. Journal of Biomedical Informatics, 37(3), 193-204.
2. Stanton, N.A., Salmon, P.M., Walker, G.H., Baber, C., & Jenkins, D.P. (2013). Human error identification methods. In. N. Stanton (Ed.), Human factors methods: A practical guide for engineering and design (pp. 139-212). Burlington, VT: Ashgate Publishing Company.
3. Zhang, J., Pate, V.L., & Shortliff, E.H. (2004). A cognitive taxonomy of medical errors. Journal of Biomedical Informatics, 37(3), 193-204.