The Office of the National Coordinator (ONC) for Health IT includes a requirement for Safety-Enhanced Design that involves usability testing as part of Meaningful Use Stage 2 certification criteria. In a recent communication, the ONC solicited feedback about their proposed 2015 certification rule. As a group of usability experts, we responded with recommendations that will be of interest to OJNI readers. For the 2015 rule, the sections focused on safety enhanced design and usability are S. 170.315(g)(3) and S. 170.315(g)(4) of the proposed language available.
Our response is below.
In Stage 3, the ONC should consider asking organizations to map their usability maturation using a model like the HIMSS Usability Maturity Model (HIMSS, 2011). For Stage 2, the ONC will want to clarify what it means to follow an industry standard UCD process, for example ISO 9241-11, ISO 9241-210, ISO 13407, ISO 16982, and ISO/IEC 62366. In addition, the ONC will want to assure certifying bodies hold organizations to their stated User-Centered Design process during certification.
The ONC will want to clarify how the Customized Common Industry Format template (NISTIR 7742) is supposed to be used. Specifically, we recommend that the ONC define operational tasks and scenarios with clinically relevant content in addition to operational measures so testing is consistent across the industry and comparable across sites. The ONC should clarify whether achieving optimal task times is required for pass rates in the current summative testing process and specify the correction factor to use in converting expert times to goal times for summative testing.
The ONC asked for feedback about four specific issues listed below:
1) Whether the scope of ‘‘Safety-Enhanced Design’’ should be expanded to include additional certification criteria:
- Organizations should be strongly encouraged to embrace established User-Centered Design (UCD) processes and procedures. Organizations following UCD processes will produce more usable software than those not following them. As a practical matter, organizations should declare or describe which processes they are using and provide evidence that they are following them. ISO 92411-11 and NIST documents are helpful. Others are available to guide usability testing such as Rubin and Chisnell (2008), Tullis and Albert (2008), Dumas and Redish (1993) and Nielsen (1993, 1994).
- We recommend leaving the current criteria as stated in MU2 with three exceptions: (1) formative testing should be required for moderate/high risk changes and new development; (2) organizations should document how identified usability issues were addressed in subsequent product updates/changes; and
(3) a usability risk management program should be developed and used as part of the testing process. Iterative, formative usability testing is one of the hallmarks of UCD so including formative testing is critical for new designs or for addressing moderate/high risk issues with current designs. The ONC should require documentation of iterative testing to show how major issues were addressed. Organizations should develop and document usability risk management programs. Complex applications (e.g., handoffs/transitions, BCMA, complex orders) and high-risk applications/modules/tasks can be identified and testing applied to mitigate potential patient safety issues in particular.
2) Whether formative usability tests should be explicitly required, or used as substitutes for summative testing;
- This is a false dichotomy. Formative testing is a part of an iterative UCD process although employed at different points in the product development process. Both should be required because neither is a substitute for the other. Formative testing should be targeted to specific, high risk or new development rather than the EHR as a whole or even an entire module and should be done early in the design process (e.g., test using wireframes).
- If organizations can provide evidence that they adhere to an industry standard UCD process and that identified issues are addressed, they should not be required to continue to perform summative testing on the same modules/criteria, especially if no new changes to those areas have been made. The ONC should create language to address this issue.
- A particular concern is that provider organizations may modify applications themselves. Therefore, either independently or by partnering with their vendors, provider organizations should meet the same safety-enhanced design requirements for moderate/high risk areas where the provider organization has customized the EHR, especially for high risk modules, e.g., CPOE, eMAR/BCMA, and/or medications management.
- EHR applications should be integrated into the workflow and "thoughtflow" of end-users. Separating out specific modules such as CDS can be difficult as CDS should be integrated into workflow and be transparent to users. The ONC can clarify that organizations can conduct usability testing for this integration using clinical relevant scenarios versus completing separate CDS testing. Evidence of this integrated testing should be a part of the reporting requirements.
3) Whether there are explicit usability tests that should be required in addition to summative testing;
- Many industry standard methods are available for high-quality organizations to use to achieve high-levels of usability in their applications. The ONC should empower organizations to choose an appropriate industry standard method as long as it is disclosed and organizations adhere to the process. Prescribing testing techniques falls short of being helpful because the type of testing depends upon numerous factors: the risk level of an area, the stage of development, the task type, and the context. Allow organizations to employ usability methods as appropriate to these factors.
4) Whether there should be a minimum number of test subjects explicitly required for usability testing.
- Summative testing with fewer than 15 participants should not be accepted going forward. Summative testing is a benchmark test so more users are required; 15-20 are suggested per user group for which the application is being designed as per NIST IR 7804. If organizations use smaller samples, justification with rationale is needed since this would be outside industry standards and best practice recommendations. This is also consistent with language in the 2011 FDA guidance on validation testing (see Appendix B). If users with distinctly different characteristics [e.g., use responsibilities, age ranges, skills sets or experience levels] will use the device, validation testing activities should include 15 from each major user group. (FDA, 2011).
- It is our strong recommendation that organizations follow industry accepted guidelines on compliance with a number of test subjects for formative testing as well. Several elements are crucial in determining sample size: the type of test, the representative users for that test/type of work, type of task, context, and risk factors (patient safety versus cosmetic). Resources on sample size include the ones above and Falkner (2003) and Virzi (1992).
- Importantly, tested users need to be representative of the intended end-users. Organizations should document how the sample was selected and how they are representative of the end-user population rather than the selection being only a matter of convenience.
- For general guidance, the number of users differs according to the type of usability testing being done. In early design in formative testing, the number of users can be smaller but they still need to represent the groups of intended users. A minimum of six to eight per group is suggested by Nielsen, although Falkner cautions that ten users create less variability in findings. The better guideline is to hold to the stated process, to document how sample size was determined and include a confidence interval.
Dumas J, Redish J. (1993). A Practical Guide to Usability Testing. Chicago, IL: University of Chicago Press.
Falkner L. (2003). Beyond the five user assumption: Benefits of increaed smple sizes. Behavior Research, Methods and Computers, 379-83.
FDA. (2011). Draft Guidance for Industry and Food and Drug Administration Staff - Applying Human Factors and Usability Engineering to Optimize Medical Device Design. Washington, DC.
HIMSS. (February, 2011). Promoting Usability in Health Organizations: Initial Steps and Progress Toward a Healthcare Usability Maturity Model Chicago, IL: Health Information Management Systems Society
Nielsen J. (1993). Usability Engineering. Cambridge, MA: AP Professional.
Nielsen J, Mack R. (1994). Usability Inspection Methods. New York: John Wiley & Sons.
Rubin J, Chisnell D, Spool J. (2008). Handbook of Usability Testing: Howto Plan, Design, and Conduct Effective Tests, 2nd Edition: John Wiley & Sons.
Tullis T, Albert W. (2008). Measuring the User Experience: Collecting, Analyzing, and Presenting Usability
Metrics. Burlington, MA: Morgan Kaufmann.
Virzi. (1992). Refining the test phase of usability evaluation: How many subjects is enough? Human
Factors, 34, 457-69.