Data Collection & Analysis
ObservationObservation is a knowledge elicitation method in which analysts observe workers engaging in tasks of interest in a natural work setting to determine what factors affect worker performance of the task. The primary advantage of observation is the richness of the data collected as the data is gathered about the work as it occurs in a realistic context. However, observation is a time-intensive method, and the amount of data collected is difficult and time-consuming to analyze. This data collection ideally takes place very early in the design cycle and is often the first step in investigating the work of interest. Data gathered via observation can be used to break down work into tasks and subtasks, inform design requirements, inform interview questions, identify sources of errors in system performance, and develop theory about work performance. Given that the richness of the data comes with costs of time and resources, observation should only be employed when highly context-rich data is required to answer the analyst's or designer's research questions. Otherwise, cheaper and faster research methods should be used to answer the research questions more efficiently.
|
What is Observation?
Observation (aka fieldwork or ethnography) is a knowledge elicitation method in which analysts observe workers engaging in tasks of interest in a natural work setting to determine what factors affect worker performance of the task.
Why Use Observation?
Good design requires developers to consider not only the task for which they are making a product, but also the people performing the task and the context and settings in which the people will be using the product. The goal of observation is to identify all potential factors that affect people's performance on the tasks of interest by documenting work practices. Work practices reflect how work is actually done in natural work settings, which can differ radically from how work is believed to be done (Clancey, 2006). Thorough documentation of work practices provides information that should inform design decisions for the product under development.
The advantage of observation as a method is the richness of the data gathered (Cooke, 1999). In contrast to survey research, where the analyst can only receive answers to the questions he or she has asked, observation allows analysts to identify factors that affect performance that may be entirely unexpected. In addition, the analyst can gather data holistically from the total system in which the task of interest is performed. The total system includes the people who perform the tasks, the tools they use and the goals they pursue, the settings and context under which the people complete the tasks, and how all of these system elements interact with one another (Clancey, 2006). This holistic approach allows the analyst to identify performance impacting factors from all possible sources, informing future decisions regarding product design.
A disadvantage of observation is that it is time-intensive, both during data collection and data analysis (Cooke, 1999). Observation itself takes time, and analysts may have to travel to work-sites, which may be costly. During analysis, the content rich data must be sifted to distill out the performance affecting factors. This often requires developing a coding scheme to quantify these factors for further analysis, and this can be a difficult and lengthy process. Further disadvantages include threats to validity of findings from observer biases (Cooke, 1999). Analyzing data from observation requires a large amount of expert judgement from the analysts, and the analyst must be careful not to let preconceived notions about the work practice color his or her conclusions about what factors affect performance. Another issue is the degree of obtrusiveness of the observation. Analysts must be careful not to let his or her presence affect or interfere with normal work practices (Clancey, 2006). Finally, observation of the physical behaviors surrounding task performance may not be sufficient for highly cognitive work.
Given that the richness of the data comes with costs of time and resources (Cooke, 1999), observation should only be selected when highly rich data is required to answer the analyst's or designer's research questions. This may be the case when research questions or product requirements are still being developed or where theory or prior experience is lacking to guide design decisions. If research questions can be answered without the need for detailed, real-world information from the total system, a cheaper and quicker data-collection method (i.e., survey research) may be employed to answer the questions more efficiently.
The advantage of observation as a method is the richness of the data gathered (Cooke, 1999). In contrast to survey research, where the analyst can only receive answers to the questions he or she has asked, observation allows analysts to identify factors that affect performance that may be entirely unexpected. In addition, the analyst can gather data holistically from the total system in which the task of interest is performed. The total system includes the people who perform the tasks, the tools they use and the goals they pursue, the settings and context under which the people complete the tasks, and how all of these system elements interact with one another (Clancey, 2006). This holistic approach allows the analyst to identify performance impacting factors from all possible sources, informing future decisions regarding product design.
A disadvantage of observation is that it is time-intensive, both during data collection and data analysis (Cooke, 1999). Observation itself takes time, and analysts may have to travel to work-sites, which may be costly. During analysis, the content rich data must be sifted to distill out the performance affecting factors. This often requires developing a coding scheme to quantify these factors for further analysis, and this can be a difficult and lengthy process. Further disadvantages include threats to validity of findings from observer biases (Cooke, 1999). Analyzing data from observation requires a large amount of expert judgement from the analysts, and the analyst must be careful not to let preconceived notions about the work practice color his or her conclusions about what factors affect performance. Another issue is the degree of obtrusiveness of the observation. Analysts must be careful not to let his or her presence affect or interfere with normal work practices (Clancey, 2006). Finally, observation of the physical behaviors surrounding task performance may not be sufficient for highly cognitive work.
Given that the richness of the data comes with costs of time and resources (Cooke, 1999), observation should only be selected when highly rich data is required to answer the analyst's or designer's research questions. This may be the case when research questions or product requirements are still being developed or where theory or prior experience is lacking to guide design decisions. If research questions can be answered without the need for detailed, real-world information from the total system, a cheaper and quicker data-collection method (i.e., survey research) may be employed to answer the questions more efficiently.
When Use Observation?
Observations can gather data for formative or summative purposes. Formative purposes involve identifying areas for improvement with a certain design. For instance, analysts may be attempting to identify user requirements, the level of expertise of average users, or common workplace conditions. For formative purposes, observation should occur early in the design cycle (Gillan, 2012). Observations can serve as the first step in eliciting work information, informing later research methods and design decisions (Cooke, 1999). Conducting observations early will ensure that the product is designed from the beginning to account for factors that affect task performance from the total system in which the work is taking place, saving the need for costly redesign efforts later should the product not satisfactorily address users' needs.
Summative purposes, on the other hand, involve determining whether or not a product meets the requirements of the task for which it was designed. For summative purposes, observations should occur later in the design cycle. For instance, analysts can give future users a prototype of the product and conduct an observation to verify that the product is satisfactory in a real-use setting. If the product has been designed in accordance with data from earlier observations, the need for further design changes at this stage should be minimized.
Summative purposes, on the other hand, involve determining whether or not a product meets the requirements of the task for which it was designed. For summative purposes, observations should occur later in the design cycle. For instance, analysts can give future users a prototype of the product and conduct an observation to verify that the product is satisfactory in a real-use setting. If the product has been designed in accordance with data from earlier observations, the need for further design changes at this stage should be minimized.
How to Use Observation?
Observations generally require an analyst to observe others engaging in the task of interest in the natural setting in which the task is completed. While the focus of the observation depends on the analyst's research questions, common things to document are task steps, gestures, communications, emotional displays (i.e., frustration), errors, work hand-offs, documentation practices, and environmental factors amongst others. The specificity of documentation also varies with the research questions. For instance, a designer of surgical tools may document every hand-position of a surgeon in practice, but such specificity of task steps would not be useful to a designer of accounting software.
Observations are often used in tandem with interview methods and reviews of archival data sources (i.e., instruction manuals, job descriptions). Especially in cases where work is more cognitive than physical, analysts can gain additional information by querying workers while they engage in work tasks or with a separate interview. Reviews of archival data can provide additional contextual information. Both interview responses and reviews of archival data can be used to guide observational inquiry.
There are a number of considerations an analyst should account for in how they conduct their observation. First, it is essential that analysts consider the total system when performing their observation (Clancey, 2006). Below are a list of example questions analysts may consider during observation:
These types of questions are important because attending to them should inform design choices. For instance, if a software developer attends to the fact that the eventual users of his software would range from having very little computer experience to very high computer experience, he can build in optional tutorials to get low-experience users up-to-speed without slowing down the experienced users. A designer of a mobile-device who knows that her device will be used in loud manufacturing settings may choose to make notifications vibration-based or text-based rather than auditory-based. Knowing how current work is currently documented and how information is shared within an organization can allow developers to design their technologies to interface well with the tools and systems the organization currently uses. Failing to attend to these issues can lead to design flaws resulting in low customer satisfaction or necessitating costly redesign efforts.
Second, its is imperative that analysts observe a representative sample of the people, uses, and contexts for which they are designing their technology. Observations should be made of the full range of people for whom the technology is being designed as well as across the full-range of settings and contexts. Failing to gain a representative sample can result in the development of a product that only suits the needs of a subset of the technologies eventual users or uses.
While the information above should guide any type of observational inquiry, there are variations in observational methods that analysts may select to suit their research question. Several key variations are addressed below.
Structure
Observational techniques can vary in the degree of structure the analyst imposes upon them (Gillan, 2012). In a completely unstructured observation, the analyst enters the work setting with no preconceived ideas about what elements of the system to observe. He or she observes the work as it naturally occurs, documenting everything of apparent relevance. Over time, the analyst refines his or her inquiry, focusing more on the factors that most drive performance on the tasks of interest.
Analysts may structure their technique using previous knowledge in order to reduce the scope on which they focus during observation. Prior knowledge may be obtained by first conducting interviews with workers about what they feel impacts their performance or by reviewing workplace documents. Analysts may also use their experiences from past projects or theories from academic research to structure their inquiries. In this way, analysts come in to the observation with specific research questions and hypotheses, and they observe workers and their surroundings to find answers that will inform design decisions. In a highly structured observation, analysts may only observe the worker in a prearranged scenario or in a controlled setting (Gillan, 2012).
It is important that analysts pick the degree of structure that best suits the purpose of his or her inquiry. Unstructured observational techniques allow for greater flexibility and for the discovery of unexpected variables that affect work performance, but resources may be invested inefficiently by attending to features of the work scenario that are uninformative. Introducing more structure improves the efficiency of observation by focusing only on the features of the work scenario that are suspected to be relevant to performance, but the data collected may be biased to match the prior expectations of the analyst if the analyst is too inflexible. A balance must be struck such that the analyst gathers sufficient information to answer his or her questions regarding future design without wasting resources in gathering irrelevant information.
Recording
Analysts may choose to take physical notes while observing, but it is advised that they also employ employ video or audio recording devices. The advantage of recording is that analysts can review the work as many times as necessary and share the recording with other analysts (Clancey, 2006). In addition, attempting to take notes while observing may cause the analyst to miss important details. It is also possible for the analyst to record the work taking place without physically attending the workplace to observe. This can reduce the degree to which the analyst interferes with normal work practice but removes the opportunity for the analyst to query workers about the reasons behind their behaviors.
Obtrusiveness
Analysts must consider how much their presence may interfere with the work being performed (Cooke, 1999). In some instances this will pose no problem, and the analyst may be able to ask workers about the work they do, allowing for additional information to be gathered. However, there are instances where the analyst may get in the way (for example, in a crowded operating room). In these cases, analysts should consider video-recording the work and analyzing it later. There are also instances where workers may be uncomfortable being observed. It is best for analysts to keep workers they are observing informed of the purpose for the observation and to allow them to ask questions about it in order to gain buy-in and offset anxiety.
Number of observers
When resources permit, it is advised that multiple observers participate in the observation. Having multiple perspectives reduces threats of observer bias. Video-recording allows for easier sharing of observed materials for review by others.
Use of a subject-matter expert observer
At times, it may be beneficial to have a subject-matter expert, a person from the natural work setting who is familiar with the tasks of interest, observe the work along with the analyst. This is particularly useful if the setting is not one with which the analyst is familiar as the subject-matter expert may have a better idea of what to attend to during the observation. The analysts and subject-matter expert can then compile their observations, both benefiting from the other's perspective (Clancey, 2006).
Temporality
Analysts must also consider how long they plan to conduct the observation. For instance, if work is accomplished differently at different times of the year or on different work shifts, the analysts may need to return for multiple visits to the work-setting (Clancey, 2006). Analysts must balance the costs of additional observations against the needs to answer their design questions, observing long enough to answer the questions without collecting redundant information.
Participation
In some cases, analysts may perform the work tasks themselves to gain information about factors that affect work performance (Clancey, 2006).
Observations are often used in tandem with interview methods and reviews of archival data sources (i.e., instruction manuals, job descriptions). Especially in cases where work is more cognitive than physical, analysts can gain additional information by querying workers while they engage in work tasks or with a separate interview. Reviews of archival data can provide additional contextual information. Both interview responses and reviews of archival data can be used to guide observational inquiry.
There are a number of considerations an analyst should account for in how they conduct their observation. First, it is essential that analysts consider the total system when performing their observation (Clancey, 2006). Below are a list of example questions analysts may consider during observation:
- Who will be using the technology being developed?
- How much experience do they have with the technology or similar technologies?
- Is the technology going to be used by individuals with varying levels of experience?
- What knowledge, skills, and abilities do the people being observed possess?
- How are the people accomplishing tasks of interest?
- What tools are they using? How effective are these tools?
- Do they need to work with others to accomplish the work? If so, how do they communicate?
- What steps do they take in accomplishing the tasks? Which steps are most critical? Which are the most challenging? Which are the most frustrating?
- How is the work documented? Why is it documented in this way? Who accesses this information and for what purposes?
- In what setting are the tasks being completed?
- What facilities are available?
- Are there environmental variables that affect work (i.e., heat, noise, safety hazards)?
- How does the work-setting fit into the larger organization?
- In what context are the tasks being completed?
- Is the work being completed under high time-pressure?
- Are workers fatigued from long hours or lack of sleep?
- Are workers multitasking?
- How does the work done by these workers affect other parts of the organization?
- Is the work completed the same way over time? Are there monthly, seasonal, or annual changes in how the work is performed?
- Are there social or political pressures on workers?
These types of questions are important because attending to them should inform design choices. For instance, if a software developer attends to the fact that the eventual users of his software would range from having very little computer experience to very high computer experience, he can build in optional tutorials to get low-experience users up-to-speed without slowing down the experienced users. A designer of a mobile-device who knows that her device will be used in loud manufacturing settings may choose to make notifications vibration-based or text-based rather than auditory-based. Knowing how current work is currently documented and how information is shared within an organization can allow developers to design their technologies to interface well with the tools and systems the organization currently uses. Failing to attend to these issues can lead to design flaws resulting in low customer satisfaction or necessitating costly redesign efforts.
Second, its is imperative that analysts observe a representative sample of the people, uses, and contexts for which they are designing their technology. Observations should be made of the full range of people for whom the technology is being designed as well as across the full-range of settings and contexts. Failing to gain a representative sample can result in the development of a product that only suits the needs of a subset of the technologies eventual users or uses.
While the information above should guide any type of observational inquiry, there are variations in observational methods that analysts may select to suit their research question. Several key variations are addressed below.
Structure
Observational techniques can vary in the degree of structure the analyst imposes upon them (Gillan, 2012). In a completely unstructured observation, the analyst enters the work setting with no preconceived ideas about what elements of the system to observe. He or she observes the work as it naturally occurs, documenting everything of apparent relevance. Over time, the analyst refines his or her inquiry, focusing more on the factors that most drive performance on the tasks of interest.
Analysts may structure their technique using previous knowledge in order to reduce the scope on which they focus during observation. Prior knowledge may be obtained by first conducting interviews with workers about what they feel impacts their performance or by reviewing workplace documents. Analysts may also use their experiences from past projects or theories from academic research to structure their inquiries. In this way, analysts come in to the observation with specific research questions and hypotheses, and they observe workers and their surroundings to find answers that will inform design decisions. In a highly structured observation, analysts may only observe the worker in a prearranged scenario or in a controlled setting (Gillan, 2012).
It is important that analysts pick the degree of structure that best suits the purpose of his or her inquiry. Unstructured observational techniques allow for greater flexibility and for the discovery of unexpected variables that affect work performance, but resources may be invested inefficiently by attending to features of the work scenario that are uninformative. Introducing more structure improves the efficiency of observation by focusing only on the features of the work scenario that are suspected to be relevant to performance, but the data collected may be biased to match the prior expectations of the analyst if the analyst is too inflexible. A balance must be struck such that the analyst gathers sufficient information to answer his or her questions regarding future design without wasting resources in gathering irrelevant information.
Recording
Analysts may choose to take physical notes while observing, but it is advised that they also employ employ video or audio recording devices. The advantage of recording is that analysts can review the work as many times as necessary and share the recording with other analysts (Clancey, 2006). In addition, attempting to take notes while observing may cause the analyst to miss important details. It is also possible for the analyst to record the work taking place without physically attending the workplace to observe. This can reduce the degree to which the analyst interferes with normal work practice but removes the opportunity for the analyst to query workers about the reasons behind their behaviors.
Obtrusiveness
Analysts must consider how much their presence may interfere with the work being performed (Cooke, 1999). In some instances this will pose no problem, and the analyst may be able to ask workers about the work they do, allowing for additional information to be gathered. However, there are instances where the analyst may get in the way (for example, in a crowded operating room). In these cases, analysts should consider video-recording the work and analyzing it later. There are also instances where workers may be uncomfortable being observed. It is best for analysts to keep workers they are observing informed of the purpose for the observation and to allow them to ask questions about it in order to gain buy-in and offset anxiety.
Number of observers
When resources permit, it is advised that multiple observers participate in the observation. Having multiple perspectives reduces threats of observer bias. Video-recording allows for easier sharing of observed materials for review by others.
Use of a subject-matter expert observer
At times, it may be beneficial to have a subject-matter expert, a person from the natural work setting who is familiar with the tasks of interest, observe the work along with the analyst. This is particularly useful if the setting is not one with which the analyst is familiar as the subject-matter expert may have a better idea of what to attend to during the observation. The analysts and subject-matter expert can then compile their observations, both benefiting from the other's perspective (Clancey, 2006).
Temporality
Analysts must also consider how long they plan to conduct the observation. For instance, if work is accomplished differently at different times of the year or on different work shifts, the analysts may need to return for multiple visits to the work-setting (Clancey, 2006). Analysts must balance the costs of additional observations against the needs to answer their design questions, observing long enough to answer the questions without collecting redundant information.
Participation
In some cases, analysts may perform the work tasks themselves to gain information about factors that affect work performance (Clancey, 2006).
References & Resources
- Clancey, W. J. (2006). Observation of Work Practices in Natural Settings. In K. Ericsson, N. Charness, P. J. Feltovich, R. R. Hoffman (Eds.) , The Cambridge handbook of expertise and expert performance (pp. 127-145). New York, NY US: Cambridge University Press.
- Cooke, N. J. (1999). Knowledge elicitation. In F. T. Durso (Ed.) , Handbook of applied cognition (pp. 479-509). New York, NY US: John Wiley & Sons Ltd.
- Gillan, D. J. (2012). Five questions concerning task analysis. In M. A. Wilson, W. r. Bennett, S. G. Gibson, G. M. Alliger (Eds.) , The handbook of work analysis: Methods, systems, applications and science of work measurement in organizations (pp. 201-213). New York, NY US: Routledge/Taylor & Francis Group.
Further Reading
- Dahlbäck, N., Jönsson, A., & Ahrenberg, L. (1993). Wizard of Oz studies—why and how. Knowledge-based systems, 6(4), 258-266.
- DeJoode, J., Cooke, N. J., & Shope, S.M. (2003). Naturalistic observations of an airport mass casualty exercise. Proceedings of the Human Factors and Ergonomics Society 47th Annual Meeting.
- Dourish, P. (2006). Implications for Design. Proceedings of the ACM CHI Conference on Human Factors in Computing Systems. Montreal, Quebec.
- Roth, E. M., Christian, C. K., Gustafson, M., Sheridan, T. B., Dwyer, K., Gandhi, T. K., ... & Dierks, M. M. (2004). Using field observations as a tool for discovery: analys ing cognitive and collaborative demands in the operating room. Cognition, Technology & Work, 6(3), 148-157.