Testing & Evaluation
Workload AnalysisWorkload is hypothetical construct that represents the mental cost incurred by a human operator to achieve a particular level of performance (Hart & Staveland, 1988). Workload level a human would experience during executing task is influenced by many factors including operator capability and required performance goals in addition to objective demands imposed by the task. The evaluation of mental workload is a key point in the research and development of human-machine interfaces, in search of higher levels of comfort, satisfaction, efficiency, and safety in the workplace (Rubio, Díaz, Martin & Puente, 2004).
|
What is Workload Analysis?
A task for a human operator within human-machine interactions should neither underload or overload an individual, and workload analysis is important to assess workload during task completion. A number of tools for the evaluation and prediction of mental workload were developed and currently used (Rubio, et al., 2004). There are three categories of mental workload measurements; subjective measures, performance-based techniques and physiological measures
1. Subjective measures
Subjective workload measures are commonly used tools in many fields and task scenarios. Subjective measures are known to be sensitive, more global and also diagnostic. Subjective measures are often preferred because they are easy to implement and users are usually accepting of these methods.
There are many subjective workload scales such as Modified Cooper Harper [MCH], Subjective Workload Assessment Technique [SWAT], NASA Task Load Index [NASA-TLX], Overall Workload Scale, Bedford Scale, Verbal Online Subjective Opinion [VOSO], Workload Profile [WP] and Subjective Opinion via Continuous Control [SOCC]. Commonly used subjective measures include NASA-TLX, SWAT and WP. SWAT, NASA-TLX and WP are all multidimensional. See the figures for the rating scale dimensions.
Subjective workload measures are commonly used tools in many fields and task scenarios. Subjective measures are known to be sensitive, more global and also diagnostic. Subjective measures are often preferred because they are easy to implement and users are usually accepting of these methods.
There are many subjective workload scales such as Modified Cooper Harper [MCH], Subjective Workload Assessment Technique [SWAT], NASA Task Load Index [NASA-TLX], Overall Workload Scale, Bedford Scale, Verbal Online Subjective Opinion [VOSO], Workload Profile [WP] and Subjective Opinion via Continuous Control [SOCC]. Commonly used subjective measures include NASA-TLX, SWAT and WP. SWAT, NASA-TLX and WP are all multidimensional. See the figures for the rating scale dimensions.
2. Performance-based techniques
Performance-based measures can be categorized in primary task measurements and secondary task measurements. Primary task measurements evaluate user’s performance (e.g., speed or accuracy) on primary system functions as indexes of operator workload. It is expected that user’s performance (speed and/or accuracy) will decrease as workload increases. Secondary task methodologies normally utilize secondary tasks which are not part of normal system functions (Wierwille & Eggemeier, 1993). Such tasks typically involve memory, mental math, interval production, reaction time, time estimation, and tracking. The potential issues regarding to sensitivity and intrusion have been suggested in previous literatures. It has been suggested that a secondary task is expected to demonstrate greatest sensitivity when there is considerable overlap in the processing/resource demands of the primary and secondary tasks (Wierwille & Eggemeier, 1993). Alternatively, embedded secondary task measures can be used. Embedded secondary task will be less artificial, thus minimizing intrusion. And embedded secondary task can also be diagnostic if demands overlap with primary task (example: communicating on the radio while flying). |
3. Physiological techniques
There are various physiological techniques that can be used as index of mental workload.
- Heart rate: index of arousal or physical work
- less-intrusive; good for continuous monitoring of workload)
- Brain activity: e.g., EEG recordings
- Difficult to synchronize stimuli with EEG record; lots of noise in data from body movement)
- Eye activity: e.g., eye-blink latency, eye-blink duration, eye-blink rate
- typically best used to measure workload of visual tasks
- Others: respiration, eye point-of-regard, epoch analyses of EEG
Why Use Usability Testing?
Workload analysis would allow assessing human operators’ experience of workload during executing tasks, regulating workload demands would ensure safety, health, comfort, and long-term productive efficiency of the operators (Rubio et al., 2004). A goal of workload analysis would be to regulate task demands not to be neither underload or overload an individual. Although the dangers of overload have long been recognised, many of our recent concerns are with the stress of underload and boredom (Becker, Warm, Dember, & Hancock, 1991; Hancock & Warm, 1989), particularly as operations become the subject of progressively increased automatio
When Use Workload Analysis?
Workload analysis often used in the later design stage of evaluation. Human operators are typically assessed during they are interacting with targeted systems in a real-world or simulated environments. Workload analysis also can be implemented at early stage of the design cycle such as task analysis. Designers or human factors evaluators may predict points of overload or underload to find potential error points.
How to Use Workload Analysis?
There are criteria to consider selecting suitable workload assessment methodologies for your targeted task/system/procedure (Eggemeier, Wilson, Kramer, & Damos, 1991):
1. Sensitivity: A tool’s power to detect changes in task difficulty or demands.
2. Diagnosticity: This involves not only the identification of changes in workload variation but also the reason for those changes.
3. Selectivity/Validity: The index must be sensitive only to differences in cognitive demands, not to changes in other variables such as physical workload or emotional stress, not necessarily associated with mental workload.
4. Intrusiveness: The measure should not interfere with the primary task performance, the load which is the actual object of evaluation.
5. Reliability: The measure must reflect consistently the mental workload.
6. Implementation requirements: Including aspects such as time, instruments, and software for the collection and analysis of data.
7. Subject acceptability: This refers to the subject’s perception of thevalidity and usefulness of the procedure.
In addition to the primary considerations, there are additional things that human factors researchers might need to consider to select workload analysis measures (Wierwille & Eggemeier, 1993).
1. Sensitivity: A tool’s power to detect changes in task difficulty or demands.
2. Diagnosticity: This involves not only the identification of changes in workload variation but also the reason for those changes.
3. Selectivity/Validity: The index must be sensitive only to differences in cognitive demands, not to changes in other variables such as physical workload or emotional stress, not necessarily associated with mental workload.
4. Intrusiveness: The measure should not interfere with the primary task performance, the load which is the actual object of evaluation.
5. Reliability: The measure must reflect consistently the mental workload.
6. Implementation requirements: Including aspects such as time, instruments, and software for the collection and analysis of data.
7. Subject acceptability: This refers to the subject’s perception of thevalidity and usefulness of the procedure.
In addition to the primary considerations, there are additional things that human factors researchers might need to consider to select workload analysis measures (Wierwille & Eggemeier, 1993).
- Time: short-term (several seconds) vs. Long-term (hours). It is important to identify if you want a momentary workload assessment or an overall workload assessment
- Multiple measures: some measures may be sensitive to different parts of a task; thus employing more than one measure may be more helpful for diagnostic purposes. However, multiple measures may also interfere with one another
References & Resources:
1. Becker, A.B., Warm, J.S., Dember, W.N., & Hancock, P.A. (1991). Effects of feedback on perceived workload in vigilance performance. In Proceedings of the Human Factors Society Thirty-Fifth Annual Meeting (pp. 1491–1494). Santa Monica, CA: Human Factors and Ergonomics Society.
2. Eggemeier, F.T., Wilson, G.F., Kramer, A.F., & Damos, D.L. (1991). General considerations concerning workload assessment in multi-task environments. In D.L. Damos (Ed.), Multiple task performance (pp. 207–216). London: Taylor& Francis.
3. Hart, S.G., & Staveland, L.E. (1988). Development of NASA-TLX (Task Load Index): Results of empirical and theoretical research. In P.A. Hancock & N. Meshkati (Eds.), Human mental workload (pp. 139–183). Amsterdam: North-Holland.
4. Rubio, S., Díaz, E., Martin, J., & Puente, J. M. (2004). Evaluation of subjective mental workload: A comparison of SWAT, NASA-TLX, and Workload profile methods. Applied Psychology: An International Review, 53(1), 61-86.
5. Wierwille, W. W., & Eggemeier, F. T. (1993). Recommendations for mental workload measurement in a test and evaluation environment. Human Factors, 35, 263-281.
2. Eggemeier, F.T., Wilson, G.F., Kramer, A.F., & Damos, D.L. (1991). General considerations concerning workload assessment in multi-task environments. In D.L. Damos (Ed.), Multiple task performance (pp. 207–216). London: Taylor& Francis.
3. Hart, S.G., & Staveland, L.E. (1988). Development of NASA-TLX (Task Load Index): Results of empirical and theoretical research. In P.A. Hancock & N. Meshkati (Eds.), Human mental workload (pp. 139–183). Amsterdam: North-Holland.
4. Rubio, S., Díaz, E., Martin, J., & Puente, J. M. (2004). Evaluation of subjective mental workload: A comparison of SWAT, NASA-TLX, and Workload profile methods. Applied Psychology: An International Review, 53(1), 61-86.
5. Wierwille, W. W., & Eggemeier, F. T. (1993). Recommendations for mental workload measurement in a test and evaluation environment. Human Factors, 35, 263-281.