J Am Med Inform Assoc 8(2): 163-173

Studying the Human– Computer–Terminology Interface

Affiliation of the authors: Columbia-Presbyterian Medical Center, New York, New York.

Correspondence and reprint requests: James J. Cimino, MD, Department of Medical Informatics, Columbia-Presbyterian Medical Center, 622 West 168th Street, VC-5, New York, NY 10032; e-mail: <ude.aibmuloc@7cjj>.

Received 2000 Aug 18; Accepted 2000 Nov 16.

Abstract

Objective: To explore the use of an observational, cognitive-based approach for differentiating between successful, suboptimal, and failed entry of coded data by clinicians in actual practice, and to detect whether causes for unsuccessful attempts to capture true intended meaning were due to terminology content, terminology representation, or user interface problems.

Design: Observational study with videotaping and subsequent coding of data entry events in an outpatient clinic at New York Presbyterian Hospital.

Participants: Eight attending physicians, 18 resident physicians, and 1 nurse practitioner, using the Medical Entities Dictionary (MED) to record patient problems, medications, and adverse reactions in an outpatient medical record system.

Measurements: Classification of data entry events as successful, suboptimal, or failed, and estimation of cause; recording of system response time and total event time.

Results: Two hundred thirty-eight data entry events were analyzed; 71.0 percent were successful, 6.3 percent suboptimal, and 22.7 percent failed; unsuccessful entries were due to problems with content in 13.0 percent of events, representation problems in 10.1 percent of events, and usability problems in 5.9 percent of events. Response time averaged 0.74 sec, and total event time averaged 40.4 sec. Of an additional 209 tasks related to drug dose and frequency terms, 94 percent were successful, 0.5 percent were suboptimal, and 6 percent failed, for an overall success rate of 82 percent.

Conclusions: Data entry by clinicians using the outpatient system and the MED was generally successful and efficient. The cognitive-based observational approach permitted detection of false-positive (suboptimal) and false-negative (failed due to user interface) data entry.

Abstract

The ability to capture clinical information and represent it by controlled terminology is widely recognized as a necessary aspect of electronic medical record systems.¹2 The ability to represent information (such as observations and assessments) generated by health practitioners is particularly troublesome, because of the richness and complexity of clinical discourse. Depending on the task at hand, natural language processing can achieve the desired level of encoding: human beings record the data in an unconstrained way, and a computer system generates the coded form. Such approaches can succeed in well-defined, relatively narrow domains, such as mammogram interpretation, but are less applicable to large domains, such as history taking and patient problem lists.³

Even when terminologies exist for capturing information in a large domain, the issue of how the information will be transformed from concepts in the clinician's mind to codes in the computer's database remains. A common approach is to allow clinicians to enter their terms in unconstrained text and then use manual or automated means to code them afterward.⁴5 Such an approach is also used to evaluate the degree of domain coverage provided by clinical terminologies.⁶ The obvious disadvantage of this approach is that the clinician is not present to verify that the codes assigned to the text are accurate and represent the best choices available. As a result, the appropriateness of the encoding and the validity of the terminology cannot be guaranteed.⁷ An alternative strategy is to have the clinicians interact directly with the terminology to decide for themselves which terms should be used.⁸14

Typically, terminologies are evaluated in terms of their abilities to represent the concepts at hand, and data entry systems are evaluated in terms of their usability, but the two are rarely examined together. A combined approach offers a critical perspective on how the task is carried out and where it can be improved. Consider, for example, the task of putting a problem on a problem list. If a user wishes to add a problem and fails, the reason might be inadequate completeness (the terminology is not capable of representing the problem adequately), poor usability (the application does not provide adequate access to the terminology), or insufficient representation (some characteristic of the terminology, such as poor organization or inadequate synonymy, interfered with the user's ability to find the proper term). Alternatively, if the user does select a term, the question remains whether the term appropriately captures the intended meaning, and once again, any of these three causes could be to blame. Studying the terminology and the data entry application together offers the possibility of teasing out whether the application has failed (either through lack of data entry or inappropriate data entry) and, if it has, the cause of the failure.

Related work we have conducted over the past six years has focused on issues related to the design of user interfaces, to improve the usability of such systems as computerized patient record systems.¹⁵ We have adopted and modified a number of methodologies from the emerging fields of usability engineering¹⁶ and cognitive science¹⁷ to evaluate systems in terms of both the ease of accessing information and the adequacy of retrieved information. Subjects are typically asked to “think aloud” while interacting with systems to perform representative tasks. In the current work, we are extending this approach to broader issues related to both the design of the user interface and the underlying medical terminology.

Efforts have been made to study the interaction between user interface design and coded data entry. Poon et al.¹⁸ have used timing studies to assess how different user interface features affect data entry speed with a structured-progress note system. They used a variety of methods for presenting lists from which the users selected desired terms. In contrast, Elkin et al.¹⁹ studied the speed and success of users selecting terms by typing words and phrases. Like Poon and colleagues, they used paper-based scenarios from which clinicians were asked to create problem lists. In their scenarios, the desired outcomes (i.e., the specific terms to be entered) were determined in advance. The terminology was known to be complete for the tasks being studied, yet the users entered these terms only 91.1 percent of the time. Because they employed a usability laboratory (which captured detailed video recordings of the user–computer interactions), they were able to determine specific reasons why terms were not entered. This enabled them to differentiate between problems with terminology representation and system usability.

We wanted to examine how clinicians would interact with our controlled terminology, the Medical Entities Dictionary (MED)²⁰21 while using an ambulatory record application, the Decision-supported Outpatient Practice (DOP) system for entering real patient data.²² We chose to study clinicians in the process of actual patient care as they entered a variety of data (regarding problems, allergies, and medications). We employed cognitive-based methods to differentiate between appropriate and suboptimal data capture and to determine the degrees to which problems with completeness, usability, and representation contributed to the unsuccessful data capture.

Acknowledgments

This work was supported in part by an Electronic Medical Record Cooperative Agreement contract with the National Library of Medicine. The authors thank Peter Elkin, Brian Kaihoi and Chris Chute for demonstrating the value of a usability laboratory for terminology evaluation. The authors also thank Randy Barrows for his support in studying the DOP system, the study participants for their cooperation, and Andria Brummitt for editorial assistance.

Acknowledgments

Notes

This work was supported in part by an Electronic Medical Record Cooperative Agreement contract with the National Library of Medicine.

Notes

This work was supported in part by an Electronic Medical Record Cooperative Agreement contract with the National Library of Medicine.

Footnotes

^{DOP was phased out at the end of 1999 and replaced by a Web-based application called WebCIS.}²⁴

Footnotes

References

1. United States General Accounting Office. Automated Medical Records: Leadership Needed to Expedite Standards Development: Report to the Chairman/Committee on Governmental Affairs, U.S. Senate. Washington, DC: USGAO/IMTEC-93-17, Apr 1993.
2. Sittig DFGrand challenges in medical informatics? J Am Med Inform Assoc. 1994;1:412–3. [Google Scholar]
3. Rector ALClinical terminology: why is it so hard? Methods Inf Med. 1999;38:239–52. [[PubMed][Google Scholar]
4. Campbell JR, Givner N, Seelig CB, et alComputerized medical records and clinic function. MD Comput. 1989;6:282–7. [[PubMed][Google Scholar]
5. Brown SH, Miller RA, Camp HN, Giuse DA, Walker HKEmpirical derivation of ane electronic clinically useful problem statement system. Ann Intern Med. 1999;131:117–26. [[PubMed][Google Scholar]
6. Campbell JR, Carpenter P, Sneiderman C, et alPhase II evaluation of clinical coding schemes: completeness, taxonomy, mapping, definitions and clarity. J Am Med Inform Assoc. 1997;4:238–51. [Google Scholar]
7. Jollis JG, Ancukiewicz M, DeLong ER, Pryor DB, Muhlbaier LH, Mark DB. Discordance of databases designed for claims payment versus clinical information systems. Implications for outcomes research. Ann Intern Med. 1993;119:844–50. [[PubMed]
8. van der Lei J, Duisterhout JS, Westerhof HP, et alThe introduction of computer-based patient records in The Netherlands. Ann Intern Med. 1993;119:1036–41. [[PubMed][Google Scholar]
9. Huff SM, Pryor A, Tebbs RDPick from thousands: a collaborative processing model for coded data entry. Proc 17th Annu Symp Comput Appl Med Care. 1993:104–8. [Google Scholar]
10. Scherpbier HJ, Abrams RS, Roth DH, Hail JJA simple approach to physician entry of patient problem list. Proc 18th Annu Symp Comput Appl Med Care. 1994:206–10. [Google Scholar]
11. Barrows RC, Johnson SBA data model that captures clinical reasoning about patient problems. Proc 19th Annu Symp Comput Appl Med Care. 1995:402–5. [Google Scholar]
12. Gundersen ML, Haug PJ, Pryor TA, et alDevelopment and evaluation of a computerized admission diagnoses encoding system. Comput Biomed Res. 1996;29:351–72. [[PubMed][Google Scholar]
13. Campbell JRStrategies for problem list implementation in a complex clinical enterprise. Proc AMIA Annu Symp. 1998:285–9. [Google Scholar]
14. Elkin PL, Bailey KR, Chute CGA randomized controlled trial of automated term composition. Proc AMIA Annu Symp. 1998:765–9. [Google Scholar]
15. Kushniruk AW, Patel V. Cognitive computer-based video analysis. In: Greenes R, et al. (eds). Proc 8th World Conference on Medical Informatics. 1995:1566–9. [[PubMed]
16. Nielsen J. Usability Engineering. New York: Academic Press, 1993.
17. Ericsson KA, Simon HA. Protocol Analysis: Verbal Reports as Data. Cambridge, Mass: MIT Press, 1993.
18. Poon AD, Fagan LM, Shortliffe EHThe PEN–Ivory project: exploring user-interface design for the selection of items from large controlled vocabularies of medicine. J Am Med Inform Assoc. 1996;3:168–83. [Google Scholar]
19. Elkin PL, Mohr DN, Tuttle MS, et alStandardized problem list generation, utilizing the Mayo Canonical Vocabulary embedded within the Unified Medical Language System. Proc AMIA Annu Fall Symp. 1997:500–4. [Google Scholar]
20. Cimino JJ, Hripcsak G, Johnson SB, Clayton PDDesigning an introspective, controlled medical vocabulary. Proc 13th Annu Symp Comput Appl Med Care. 1989:513–8.[Google Scholar]
21. Cimino JJ, Clayton PD, Hripcsak G, Johnson SBKnowledge-based approaches to the maintenance of a large controlled medical terminology. J Am Med Inform Assoc. 1994;1:35–50. [Google Scholar]
22. Barrows RC, Allen BA, Sherman E, Smith KA decision-supported outpatient practice system. Proc AMIA Annu Fall Symp. 1996:792–6. [Google Scholar]
23. Cimino JJ. Terminology tools: state of the art and practical lessons. Presented at: IMIA Working Group 6 Conference; Dec 16–19, 1999; Phoenix, Arizona.
24. Hripcsak G, Cimino JJ, Sengupta SWebCIS: large scale deployment of a Web-based clinical information system. Proc AMIA Annu Symp. 1999:804–8. [Google Scholar]
25. Cimino JJFrom data to knowledge through concept-oriented terminologies: experience with the Medical Entities Dictionary. J Am Med Inform Assoc. 2000;7:288–97. [Google Scholar]
26. Kushniruk AW, Patel VL, Cimino JJUsability testing in medical informatics: cognitive approaches to evaluation of information systems and user interfaces. Proc AMIA Annu Fall Symp. 1997:218–22. [Google Scholar]
27. Kushniruk AW, Patel VL, Barrows RC, Cimino JJCognitive evaluation of the user interface and vocabulary of an outpatient information system. Proc AMIA Annu Fall Symp. 1996:22–6. [Google Scholar]