Presentation
PART 2: From Reactive to Proactive Safety: Joint Activity Monitoring for Infection Prevention
Event Type
Oral Presentations
TimeThursday, June 9th4:00pm - 4:30pm EDT
Location
DescriptionIntroduction:
When given a choice between being proactive and reactive, proactive is almost always preferred. This is as true for safety professionals as it is in any other setting (Provan et al., 2020; Woods et al., 2015). However, there are two daunting obstacles to a proactive safety practice: the lack of resources and the lack of meaningful proactive safety measures. We are developing Joint Activity Monitoring to address both problems and provide a pragmatic solution for organizations to meaningfully engage in proactive safety practices.
Since proactive safety does not focus on adverse events or even near misses, the question of where to focus monitoring efforts is non-trivial. In ultrasafe industries, tracking work that results in non-events increases the scope of potential investigations by up to 10,000 times (Hollnagel, 2014; Hollnagel et al., 2015). Performing proactive safety in the same way as current incident investigations is impossible without substantial scaling of safety processes and resources. Many have proposed that this is a perfect application for artificial intelligence (AI) to assist with monitoring, but results from actual implementations have revealed that (1) phenomena of interest are difficult to track automatically or semi-automatically, and (2) this automation is prone to false alarms, where atypical but benign behaviors are misinterpreted as hazardous (Woods et al., 2015).
Even if a sustainable monitoring capability is implemented, there is still a lack of meaningful, forward-facing proactive safety measures. The vast majority of currently utilized safety measures are backward-facing, focusing on categories and frequencies of unwanted events and the inexorable desire to reduce them (Hollnagel et al., 2015; Woods et al., 2015). Current efforts in proactive safety data collection are largely qualitative, which are often discounted when they conflict with other more traditional performance or safety measures (Woods et al., 2015) and inherently more ambiguous because they are detecting potential problems earlier (Klein et al., 2005).
Joint Activity Monitoring (JAM) addresses both of these potential obstacles to beginning or sustaining a proactive safety program. It extends our recent work on Joint Activity Testing (JAT) where periodic human-on-the-loop simulations are used to test system response (Morey et al., 2020). JAM also extends our work in continuous unobtrusive monitoring of the system to identify troubling trends or targeted vulnerabilities. It does this both through instrumenting the work tools of relevant operators (Murphy et al., 2017) and collecting environmental data. Together, JAM facilitates analyzing the relationships between these data to understand how the system is sustaining through current demands, how that sustaining is trending over time, and how future scenarios might affect it. JAM tracks critically important performance measures over time, paying specific attention to trends that predict future sustaining and decompensation events. Using the trends themselves as data, and orchestrating these data so that multiple trends across multiple data relationships can be seen, JAM mitigates both the risks of a paucity of meaningful measures and the brittleness that the majority of AI-enabled decision-support experiences (Rayo et al., 2020).
We are exploring JAM within the domain of infection prevention. In current practice, the constraints of available tools limit infection preventionists (IPs) to predominantly reactive approaches. Clusters of hospital acquired infections (HAIs) are determined through retrospective investigations which trigger interventions. Furthermore, the evaluation criteria of these IPs is the number of recent infection clusters and outbreaks: both reactive measures. Learning from prior infection clusters and outbreaks may generally inform how IPs can change or improve how diseases are prevented in the hospital, but there is little ability for IPs to do this proactively in the context of an emerging situation. However, we are currently working with IPs to instrument a JAM-enabled decision-support tool designed to facilitate the detection, confirmation, and anticipation of HAI clusters. With this tool, we are developing strategies to measure and store data about how IPs work to better monitor the health of the system and generate foresight about changing risks. In this paper, we discuss the strategies we are employing to implement this near real-time monitoring capability.
Methods:
In transitioning from a periodic testing capability (JAT) to a continuous monitoring capability (JAM), we have followed several steps to ensure the resulting JAM is both complementary to periodic JAT and well-suited for the unique challenges of real-time operations. First, we began with a cognitive task analysis (CTA) involving expert IPs to understand the work environment. From interviews and observations, we built an abstraction network including important goals, processes, components, and relationships in the work, which directly translated to key performance measures of system success. We then cross-referenced CTA findings with multiple macrocognitive functions to operationalize measures of what challenges the system (Patterson et al., 2010). With these performance and challenge measures, we are utilizing a series of JAT’s to discern whether or not our measures are informative cues of system stress and strain. With this subset of measures, we are instrumenting the daily work tools of IPs to unobtrusively and automatically collect performance behaviors by building these collection mechanisms into the code of the tool itself. Additionally, we are working with the organization’s IT department to collect and integrate environmental variables so that measures of challenge can be collected. By automating and integrating these tested performance and challenge measures into existing tools, we expect to reliably provide informative real-time cues of system health that are sustainably low-cost. With such tools, we believe safety professionals will be equipped to establish and sustain a proactive safety program capable of anticipating changing risks, guiding organizational adaptability, and ultimately enhancing patient safety.
References:
Hollnagel, E. (2014). Safety-I and Safety-II. Ashgate Publishing, Ltd.
Hollnagel, E., Wears, R. L., & Braithwaite, J. (2015). From Safety-I to Safety-II: A White Paper. The Resilient Health Care Net: Published simultaneously by the University of Southern Denmark, University of Florida, USA, and Macquarie University, Australia.
Klein, G., Pliske, R., Crandall, B., & Woods, D. D. (2005). Problem detection. Cognition, Technology & Work, 7(1), 14–28. doi: 10.1007/s10111-004-0166-y
Morey, D. A., Marquisee, J. M., Gifford, R. C., Fitzgerald, M. C., & Rayo, M. F. (2020). Predicting Graceful Extensibility of Human-Machine Systems: A New Analysis Method for Evaluating Extensibility Plots to Anticipate Distributed System Performance. Proceedings of the Human Factors and Ergonomics Society Annual Meeting, 64(1), 313–318. doi: 10.1177/1071181320641072
Murphy, T., Balkin, A., Rayo, M., Woods, D. D., & Zelik, D. (2017). Integrated Multi-Method Probes as a Research Method in Cognitive Systems Engineering. Proceedings of the Human Factors and Ergonomics Society Annual Meeting, 61(1), 207–211. doi: 10.1177/1541931213601536
Patterson, E. S., Roth, E. M., & Woods, D. D. (2010). Facets of complexity in situated work. In J. E.
Miller & E. S. Patterson (Eds.). Macrocognition metrics and scenarios: Design and evaluation for real-world teams. Ashgate.
Provan, D. J., Woods, D. D., Dekker, S. W. A., & Rae, A. J. (2020). Safety II professionals: How resilience engineering can transform safety practice. Reliability Engineering & System Safety, 195, 106740. doi: 10.1016/j.ress.2019.106740
Rayo, M. F., Fitzgerald, M. C., Gifford, R. C., Morey, D. A., Reynolds, M. E., D’Annolfo, K., & Jefferies, C. M. (2020). The Need for Machine Fitness Assessment: Enabling Joint Human-Machine Performance in Consumer Health Technologies. Proceedings of the International Symposium of Human Factors and Ergonomics in Healthcare, 9(1), 40–42. doi: 10.1177/2327857920091041
Woods, D. D., Branlat, M., Herrera, I., & Woltjer, R. (2015). Where Is the Organization Looking in Order to Be Proactive about Safety? A Framework for Revealing whether It Is Mostly Looking Back, Also Looking Forward or Simply Looking Away. Journal of Contingencies and Crisis Management, 23(2). doi: 10.1111/1468-5973.12079
When given a choice between being proactive and reactive, proactive is almost always preferred. This is as true for safety professionals as it is in any other setting (Provan et al., 2020; Woods et al., 2015). However, there are two daunting obstacles to a proactive safety practice: the lack of resources and the lack of meaningful proactive safety measures. We are developing Joint Activity Monitoring to address both problems and provide a pragmatic solution for organizations to meaningfully engage in proactive safety practices.
Since proactive safety does not focus on adverse events or even near misses, the question of where to focus monitoring efforts is non-trivial. In ultrasafe industries, tracking work that results in non-events increases the scope of potential investigations by up to 10,000 times (Hollnagel, 2014; Hollnagel et al., 2015). Performing proactive safety in the same way as current incident investigations is impossible without substantial scaling of safety processes and resources. Many have proposed that this is a perfect application for artificial intelligence (AI) to assist with monitoring, but results from actual implementations have revealed that (1) phenomena of interest are difficult to track automatically or semi-automatically, and (2) this automation is prone to false alarms, where atypical but benign behaviors are misinterpreted as hazardous (Woods et al., 2015).
Even if a sustainable monitoring capability is implemented, there is still a lack of meaningful, forward-facing proactive safety measures. The vast majority of currently utilized safety measures are backward-facing, focusing on categories and frequencies of unwanted events and the inexorable desire to reduce them (Hollnagel et al., 2015; Woods et al., 2015). Current efforts in proactive safety data collection are largely qualitative, which are often discounted when they conflict with other more traditional performance or safety measures (Woods et al., 2015) and inherently more ambiguous because they are detecting potential problems earlier (Klein et al., 2005).
Joint Activity Monitoring (JAM) addresses both of these potential obstacles to beginning or sustaining a proactive safety program. It extends our recent work on Joint Activity Testing (JAT) where periodic human-on-the-loop simulations are used to test system response (Morey et al., 2020). JAM also extends our work in continuous unobtrusive monitoring of the system to identify troubling trends or targeted vulnerabilities. It does this both through instrumenting the work tools of relevant operators (Murphy et al., 2017) and collecting environmental data. Together, JAM facilitates analyzing the relationships between these data to understand how the system is sustaining through current demands, how that sustaining is trending over time, and how future scenarios might affect it. JAM tracks critically important performance measures over time, paying specific attention to trends that predict future sustaining and decompensation events. Using the trends themselves as data, and orchestrating these data so that multiple trends across multiple data relationships can be seen, JAM mitigates both the risks of a paucity of meaningful measures and the brittleness that the majority of AI-enabled decision-support experiences (Rayo et al., 2020).
We are exploring JAM within the domain of infection prevention. In current practice, the constraints of available tools limit infection preventionists (IPs) to predominantly reactive approaches. Clusters of hospital acquired infections (HAIs) are determined through retrospective investigations which trigger interventions. Furthermore, the evaluation criteria of these IPs is the number of recent infection clusters and outbreaks: both reactive measures. Learning from prior infection clusters and outbreaks may generally inform how IPs can change or improve how diseases are prevented in the hospital, but there is little ability for IPs to do this proactively in the context of an emerging situation. However, we are currently working with IPs to instrument a JAM-enabled decision-support tool designed to facilitate the detection, confirmation, and anticipation of HAI clusters. With this tool, we are developing strategies to measure and store data about how IPs work to better monitor the health of the system and generate foresight about changing risks. In this paper, we discuss the strategies we are employing to implement this near real-time monitoring capability.
Methods:
In transitioning from a periodic testing capability (JAT) to a continuous monitoring capability (JAM), we have followed several steps to ensure the resulting JAM is both complementary to periodic JAT and well-suited for the unique challenges of real-time operations. First, we began with a cognitive task analysis (CTA) involving expert IPs to understand the work environment. From interviews and observations, we built an abstraction network including important goals, processes, components, and relationships in the work, which directly translated to key performance measures of system success. We then cross-referenced CTA findings with multiple macrocognitive functions to operationalize measures of what challenges the system (Patterson et al., 2010). With these performance and challenge measures, we are utilizing a series of JAT’s to discern whether or not our measures are informative cues of system stress and strain. With this subset of measures, we are instrumenting the daily work tools of IPs to unobtrusively and automatically collect performance behaviors by building these collection mechanisms into the code of the tool itself. Additionally, we are working with the organization’s IT department to collect and integrate environmental variables so that measures of challenge can be collected. By automating and integrating these tested performance and challenge measures into existing tools, we expect to reliably provide informative real-time cues of system health that are sustainably low-cost. With such tools, we believe safety professionals will be equipped to establish and sustain a proactive safety program capable of anticipating changing risks, guiding organizational adaptability, and ultimately enhancing patient safety.
References:
Hollnagel, E. (2014). Safety-I and Safety-II. Ashgate Publishing, Ltd.
Hollnagel, E., Wears, R. L., & Braithwaite, J. (2015). From Safety-I to Safety-II: A White Paper. The Resilient Health Care Net: Published simultaneously by the University of Southern Denmark, University of Florida, USA, and Macquarie University, Australia.
Klein, G., Pliske, R., Crandall, B., & Woods, D. D. (2005). Problem detection. Cognition, Technology & Work, 7(1), 14–28. doi: 10.1007/s10111-004-0166-y
Morey, D. A., Marquisee, J. M., Gifford, R. C., Fitzgerald, M. C., & Rayo, M. F. (2020). Predicting Graceful Extensibility of Human-Machine Systems: A New Analysis Method for Evaluating Extensibility Plots to Anticipate Distributed System Performance. Proceedings of the Human Factors and Ergonomics Society Annual Meeting, 64(1), 313–318. doi: 10.1177/1071181320641072
Murphy, T., Balkin, A., Rayo, M., Woods, D. D., & Zelik, D. (2017). Integrated Multi-Method Probes as a Research Method in Cognitive Systems Engineering. Proceedings of the Human Factors and Ergonomics Society Annual Meeting, 61(1), 207–211. doi: 10.1177/1541931213601536
Patterson, E. S., Roth, E. M., & Woods, D. D. (2010). Facets of complexity in situated work. In J. E.
Miller & E. S. Patterson (Eds.). Macrocognition metrics and scenarios: Design and evaluation for real-world teams. Ashgate.
Provan, D. J., Woods, D. D., Dekker, S. W. A., & Rae, A. J. (2020). Safety II professionals: How resilience engineering can transform safety practice. Reliability Engineering & System Safety, 195, 106740. doi: 10.1016/j.ress.2019.106740
Rayo, M. F., Fitzgerald, M. C., Gifford, R. C., Morey, D. A., Reynolds, M. E., D’Annolfo, K., & Jefferies, C. M. (2020). The Need for Machine Fitness Assessment: Enabling Joint Human-Machine Performance in Consumer Health Technologies. Proceedings of the International Symposium of Human Factors and Ergonomics in Healthcare, 9(1), 40–42. doi: 10.1177/2327857920091041
Woods, D. D., Branlat, M., Herrera, I., & Woltjer, R. (2015). Where Is the Organization Looking in Order to Be Proactive about Safety? A Framework for Revealing whether It Is Mostly Looking Back, Also Looking Forward or Simply Looking Away. Journal of Contingencies and Crisis Management, 23(2). doi: 10.1111/1468-5973.12079
Authors
Assistant Professor
Graduate Research Associate