By Ching-Ching Cheng MS RN, Chien-Lung Chan PhD, Lichin Chen PhD, and Sophie Huey-Ming Guo PhD RN
Citation:Cheng, C.C., Chan, C.L., Chen. L. & Guo, S.H.M. (Fall, 2019). Evaluation of the implementation of a mobile nursing information system. Online Journal of Nursing Informatics (OJNI), 23 (3), Available at http://www.himss.org/ojni
The implementation of a mobile nursing information system (MNIS) has been shown to be beneficial for nurses, taking away the burden of paperwork and enabling them to focus more on patient care. However, the factors that contribute to a successful MNIS implementation have been not been widely studied. This cross-sectional study collected data based on the DeLone & McLean Information System Success Model (1992) from 177 registered nurses in Taiwan to evaluate the implementation of a MNIS. Results showed that user satisfaction, managers’ support (β = .223, P = 0.001), system quality (β = 382, P = 0.001) and service quality (β = .161, P = 0.05) were significant predictor variables. The model consisting of the variables intention to use, managers’ support (β = .245, P < 0.001), years of RN experience (β = .093, P < 0.05), user satisfaction (β = .628, P < 0.001) produced an explained variance of 63.1%. An additional model consisting of system use and user satisfaction was also significant (β = .493, P < 0.001) and contributed to 24.3% of the total variance. Identified factors that affect the use of the MNIS can help hospital administration develop strategies for more successful implementation of MNIS.
The latest advancements in mobile technology and wireless communication have led to the development of nursing information systems (NISs) in some healthcare organizations. Mobile nursing information systems (MNIS) have been implemented in mobile devices such as laptops for years; they are used as part of both an electronic health record and a health information system (Gephart, Carrington, & Finley, 2015). Nurses use MNIS in various work environments to respond quickly in articulating their information needs. The benefits include the reduction of duplicate documentation and automatic data processing (such as automatic visualized vital signs, nursing diagnoses accompanied by disease diagnosis, and ultimately patient care outcomes) (Lee, Sun, Kou, & Yeh, 2017). However, some negative results have been reported by nurses (Gephartet al., 2015). How to maximize the use of MNIS to optimize patient care and satisfy nurses’ increasing information needs is still a critical issue. Leveraging MNIS benefits requires further understanding of the practical issues facing the needs of an ever-changing society and the complexity of health care (Rouleau et al., 2017). This also requires knowledge on whether nursing leaders are able to take an active role in the implementation of MNIS to clinical practice.
The implementation of MNISs still has mixed results and some crucial challenges in some countries. An evaluation study of NISs reported that most nurses acknowledged NIS reduced documentation efforts (Sockolow, Rogers, Bowles, Hand, & George, 2014). But the researchers also identified several problems. First, most MNISs were mounted on nursing carts, which were usually placed in patient rooms or hallways; nurses were not comfortable that they had to stand several minutes during the use of MNIS cart. Second, MNISs are not as flexible as paper-based tools in sharing information during handover between nursing shifts (Sockolow et al., 2014), or in summarizing information on patient admission and discharge (Moghaddasi, Rabiei, Asadi, & Ostvan, 2017; Samadbeik, Shahrokhi, Saremian, Garavand, & Birjandi, 2017; Sockolow et al., 2014). Nurses’ dissatisfaction with NIS is often found because the system does not fully support nursing duties and workflow (Ahmadian, Dorosti, Khajouei, & Gohari, 2017; Moghaddasi et al., 2017; Valizadeh & Bagheriyeh, 2018; Vollmer, Prokosch, Evans, & Kuttler, 2016). These problems of inefficiency and dissatisfaction impact nurses’ willingness to use MNISs and may even hamper patient care.
Influencing factors of end users’ perception are essential to successful system implementation. According to related technology acceptance models and system success theories, several key factors including organizational factors and task factors were shown to increase system use (Hsu & Wu, 2017; LaDage, Prasun, Linton, Kaiser, & Laskowski, 2015; Rashid, Abdulaziz, & Yasin, 2018; Valizadeh & Bagheriyeh, 2018; Zheng, Wang, Doll, Deng, & Williams, 2018). Others are individual willingness to use the technology and a fit between health IT and existing work practices (Vitari & Ologeanu-Taddei, 2018). These factors have not been comprehensively studied in the context of MNIS, which may lead to infrequent discussion in the literature. Moreover, previous studies on evaluation of MNISs is still limited. User acceptance of health information systems are continually studied (Shahzad, Jianqiu, Zia, Shaheen, & Sardar, 2018). User acceptance can be a prelude to or reflection of their satisfaction in using these systems, but acceptance is not representative of user satisfaction. A comprehensive method for gathering user satisfaction feedback is needed.
The Information System Success Model, known as the D&M IS Success Model, originated by DeLone and McLean in 1993, included dependent variables to measure the success and effectiveness of an information system. These variables are “information quality,” “system quality” and “service quality” (DeLone & McLean, 1993). The framework was refined in 2003 and is commonly used in evaluating the success of information systems (Delone & McLean, 2003). There are some missing elements of successfully implementing MNIS. Hsu & Wu (2017) explored the insufficient level of supportiveness from administration in health organizations, the system not being specifically designed for nursing workflows, and the lack of training for system operation. Health care managers or nursing executives are responsible for the successful deployment of MNIS. As a result, the model does not fully interpret the situation or context of MNIS.
Although NIS have been widely studied in the past, there are continual updates of the knowledge about information technologies with potential application in health care. A plethora of studies (but by far not all) point out that there are relationships between external variables of health information systems based upon user acceptance and technologically related models (Kisekka & Giboney, 2018). Lin (2017) applied an integrated model to predict nurses’ satisfaction with NIS by combining individual perception and technological functionality. But this research ignored organizational elements. Moreover, it is still unclear what the impact of administrative support is, whether positive or negative.
In the research context, this study assumed that a greater level of nursing leadership resulted in greater MNIS use. This study added elements into the success model to be more accurate and to address the effect of MNIS, including “managers’ support.” This research aims to explore more critical factors of the MNIS from the perspective of nurses.
The adjusted D&M IS Success Model (Delone & McLean, 2003) was selected as the conceptual model for this study (Figure 1). This research suggests further investigation of the MNIS success model and defines the effectiveness of MNIS in hospitals. “Managers’ support” was added as an input factor, resulting in having these inputs: “managers’ support,” “information quality,” “system quality” and “service quality.” Demographic variables were also added. By applying the D&M IS Success Model to this study, the four inputs may affect the “intention to use” MNIS. The “user satisfaction” may affect the “intention to use” of MNIS. Both “user satisfaction” and “intention to use” may affect the system use of MNIS.
Figure 1: Adjusted D&M IS Success Model
Systems quality measures technical success, referring to the extent to which systems are “user friendly” and can be used without any hassles. “System quality” is significantly related to “intention to use.” Information quality measures semantic success, referring to the system’s output value, as perceived by the respective user of the system. “Information quality” is significantly related to “intention to use.” Service quality measures the combination of service provided rather than the product itself. It is defined as users’ subjective assessment that the service they are receiving during systems testing and deployment is the service they expect. “Service quality” is significantly related with “intention to use.” Use defines whether the behavior is voluntary or mandatory, informed or uninformed, effective or ineffective. One of the major success dimensions of IS success is user satisfaction, which measures the users’ level of satisfaction with a system.
Furthermore, the relationship between the demographic information of staff and the factors in the model were analyzed to investigate whether the characteristics of staff have an effect on their opinions of MNIS.
This research adopted a cross-sectional design based on purposive sampling techniques. The research was conducted at two Taiwanese hospitals, which adopted computerized nursing diagnoses in 1999 and an NIS in 2008. The target population consisted of nursing staff at each hospital. Out of the 10 nursing units that were using MNIS, eight were selected. Staff with the experience of using MNIS over three months were considered as potential candidates. The inclusion criteria were nursing staff who (a) were more than 20 years old, (b) worked more than three months and not part-time, (c) were able to read Chinese, and (d) were willing to participate in research and sign consent forms. Exclusion criteria were (a) an inability to answer questionnaires clearly; (b) working less than 3 months or employed part-time. A total of 225 registered nurses were invited to participate in the study.
The functionality of MNIS
The function of the MNIS included (1) recording nursing information, (2) nursing plan, (3) diagnosis related to nursing decision, (4) patient identification before procedure, (5) patient education, (6) attendance performance, and (7) official duties such as diet orders and adverse event reporting.
The survey included demographic items and the IS Success Model scale. The demographic items included age, education, RN years, and RN level. The levels of registered nurse ranging from 0 to 4th level were assessed by performance appraisal in accordance with procedures agreed to by the Taiwan Nursing Association and the employer. The items of the constructs adapted from DeLone & McLean model consisted of seven dimensions, 43 items, graded by a five-point Likert scale that ranged from strongly agree (5) to strongly disagree (1). The validity and reliability of the tool were validated in previous research. The Cronbach’s alpha was reported to be 0.78 - 0.83 (Ahmadian, et al., 2017).
The researchers introduced the study to each of the eight nursing units, including the purpose of the study, the method, the time required to complete the questionnaire, and how the data would be used after they were collected. A questionnaire survey was conducted at two hospitals in Taiwan that implement the same nursing information systems and devices. The participants filled out the questionnaire and returned it within five days.
The study was conducted after the Institutional Review Board approval (reference number CGH- 98-0415B). Written informed consent forms were obtained from all participants. Participants were given instructions about the study purpose and the questionnaires. They were informed that participation in this study was voluntary, and that they could refuse or withdraw from the study at any point without giving any reason. Moreover, the participants were reassured that their responses would be kept confidential and their identities would not be revealed on research reports and publications of the study.
The researchers used the IBM SPSS software to analyze the statistical results. Multiple regression analysis was used to examine model fit for each construct and its subconstructs (to assess the measurement model).
The 225 nurses who fulfilled the inclusion criteria comprised the sample of the study, of which 216 completed the survey, 39 were excluded due to discontinued use of the MNIS, yielding a final response rate of 78.67% (N = 177). Table 1 details the demographic information of the respondents. The participants were all female, 83 (46.9%) of them were 26-30 years old, the average age was 28.57 (SD = 4.67) years, 96 (54.2%) had nursing experience for less than five years. Most of the participants had an associate (86, 48.6%) or bachelor’s degree (87, 49.2%), 78 (44.1%) were at the N2 level, meaning a registered nurse who has demonstrated competence in basic nursing practice and worked at least 2 years.
Table 1: Participant Demographics
Cronbach’s alpha for total scores was 0.889, and for constructs of MNIS success, the model ranged from 0.872 to 0.966 (Table 2). The descriptive statistics for these constructs indicated that most rated above the median score, managers’ support (mean = 28.76, SD = 5.24), system quality (mean = 21.09, SD = 5.20), information quality (mean = 13.61, SD = 2.96), service quality (mean = 9.18, SD = 2.7=56), intention to use (mean = 13.98, SD = 3.18), and user satisfaction (mean = 26.03, SD = 6.09).
Table 2: Cronbach’s alpha for total scores
Weak to strong correlations were found in the measures of the MNIS success (Table 3). There were positive correlations of quality-related constructs (system, r =.596, p < .01; information, r =.535, p < .01; service, r =.542, p < .01), and managers’ support (r =.712, p < .01) with intention to use. The quality related constructs also have positive correlations (system, r =.694, p < .01; information, r =.529, p < .01; service, r =.574, p < .01), and managers’ support (r =.588, p < .01) with user satisfaction. The correlation between system use and other constructs of MNIS were also weak to moderate.
Table 3: Correlations for all variables
Table 4 demonstrates the association of demographic and work characteristics with the constructs of the MNIS success model. As nurses’ experience were over 10 years, they were significantly more satisfied with the MINIS (F =4.988, P <.01). There were significant differences in “user satisfaction” and “intention to use” among four groups of RN level 0~4.
Table 4: Nursing characteristics and constructs of the MNIS success model
Multiple regression models were built to identify variables independently related to user satisfaction, intention to use and system Use (Table 5). First, the model of predicting for user satisfaction, managers’ support (β = .223, P < 0.001), system quality (β = 382, P < 0.001) and service quality (β = .161, P < 0.05) had a stronger impact on user emotion.
Table 5: Multiple regression results for each variable
The explained variance of user satisfaction model by each predictor is 56.5%. Second, the model of intention to use, managers’ support (β = .245, P < 0.001), years of RN experience (β = .093, P < 0.05), user satisfaction (β = .628, P < 0.001) was analyzed. The explained variance of intention to use model by each predictor is 63.1%. Third, the model of system use, user satisfaction was significant (β = .493, P < 0.001) and contributed to 24.3% of the total variance. The parameters of the MNIS success model were illustrated in Figure 2.
Figure 2: Parameters of the MNIS success model
This study analyzed the factors of MNIS implementation according to the DeLone and McLean theory (1993, 2003) to provide an insight on how to successfully implement a mobile NIS by measuring users’ perceptions. The study found that the domain relationship of RN years and managers’ support for the use of the MNIS were positively related. The most powerful predictor about user satisfaction was system quality. Furthermore, user satisfaction is also the most powerful predictor in both behavioral intention and actual use of the MNIS. Managers’ support had a significant impact on the implementation of MNIS and nurses’ satisfaction.
According to the result, most of the staff continued to use MNIS over three months and showed a positive attitude toward it. This shows that most of the staff were positive about the system. The characteristics of the staff, including age and service experiences, failed to show significant difference toward the factors of the model; this complies with past research that the attitudes of nursing staff would not be affected by personal factors such as age or service experience (Lin, 2017). A well-designed information system is desirable for nurses despite their individual differences (Gephart, et al., 2015).
It appears that the increase of intention to use MNIS leads to higher user satisfaction (Rashid, Abdulaziz, & Yasin, 2018). To increase the intention of using MNIS, an emphasis on manager support and system quality should be included. Previous research confirmed that manager support is crucial in adopting new technology, including providing enough training, clearly introducing the purpose and aim of the adoption, and keeping a close follow-up watch for negative effects of the adoption.
It is worth noticing that information quality does not necessarily affect the intention or the consistency of using MNIS. This may factor on whether MNIS could provide correct, comprehensive and timely information and whether the screen size fulfills the needs of work tasks but does not necessarily affect the staff using MNIS.
Technology adoption affects the work of all the staff in the hospital. A limitation of this study is that it only recruits the opinions of nurses. Also, the data were collected from two medical institutes to ensure that the model does not merely represent a single phenomenon or setting. However, the two institutes were still established under the same foundation thus the generality of the model still needs to be validated. Moreover, this study only sampled eight units across these two hospitals. More data could be retrieved to reveal further insights into new technology adoption. Advanced MNIS design also requires more knowledge that can contribute to improving efficiency and satisfaction solutions in the era of healthcare system reform (Andargoli, Scheepers, Rajendran, & Sohal, 2017).
The results of this study established the variable “managers’ support” linking the use of MNIS, adding to our understanding of nursing management, which is critical in the context of MNIS implementation. Although DeLone and McLean’s theory (1993) provides a useful structured view to assess the successful factors of information systems, more nursing-specific elements should be considered for nurses as real end-users. Thus, a holistic viewpoint can better guide nursing informatics research.
The use of technology has been held as a symbol of a modern organization, and patients, nurses and health care managers can all benefit from the lessons learned during the evaluation of nursing information systems. This research considers the working environment of health organizations and proposed a model to facilitate the feasible evaluation of technology implementation in the future. There has been and will be in the future a need for updating MNIS to meet the changing requirements of the current health care system to serve both patients and nurses. A success model can help healthcare organizations determine specific information system dimensions and characteristics in need of improvement.
Ahmadian, L., Dorosti, N., Khajouei, R., & Gohari, S. H. (2017). Challenges of using hospital information systems by nurses: comparing academic and non-academic hospitals. Electronic Physician, 9(6), 4625. doi: 10.19082/4625
Andargoli, A. E., Scheepers, H., Rajendran, D., & Sohal, A. (2017). Health information systems evaluation frameworks: A systematic review. International Journal of Medical Informatics, 97, 195-209. doi: 10.1016/j.ijmedinf.2016.10.008
DeLone, W. H., & McLean, E. R. (1992). Information systems success: The quest for the dependent variable. Information Systems Research, 3(1), 60-95. doi:10.1287/isre.3.1.60
Delone, W. H., & McLean, E. R. (2003). The DeLone and McLean model of information systems success: A Ten-Year Update. Journal of Management Information Systems, 19(4), 9-30. doi:10.1080/07421222.2003.11045748
Gephart, S., Carrington, J. M., & Finley, B. (2015). A systematic review of nurses' experiences with unintended consequences when using the electronic health record. Nursing Administration Quarterly, 39(4), 345-356. doi: 10.1097/NAQ.0000000000000119
Hsu, H.H., & Wu, Y.H. (2017). Investigation of the effects of a nursing information system by using the technology acceptance model. CIN: Computers, Informatics, Nursing, 35(6), 315. doi:10.1097/CIN.0000000000000313
Kisekka, V., & Giboney, J. S. (2018). The effectiveness of health care information technologies: evaluation of trust, security beliefs, and privacy as determinants of health care outcomes. Journal of Medical Internet Research, 20(4). e122. doi: 10.2196/jmir.9014
LaDage, T., Prasun, M., Linton, M., Kaiser, A., & Laskowski, P. (2015). Nurse anesthetists' perceptions of the electronic anesthesia information management system. Online Journal of Nursing Informatics, 19(3). Available at http://www.himss.org/ojni
Lee, T.Y., Sun, G.T., Kou, L.T., & Yeh, M.L. (2017). The use of information technology to enhance patient safety and nursing efficiency. Technology and Health Care, 25(5), 917-928. doi:10.3233/THC-170848
Lin, H. C. (2017). Nurses' satisfaction with using nursing information systems from technology acceptance model and information systems success model perspectives: a reductionist approach. CIN: Computers, Informatics, Nursing, 35(2), 91-99. doi: 10.1097/CIN.0000000000000293.
Moghaddasi, H., Rabiei, R., Asadi, F., & Ostvan, N. (2017). Evaluation of nursing information systems: Application of usability aspects in the development of systems. Healthcare Informatics Research, 23(2), 101-108. doi: 10.4258/hir.2017.23.2.101
Rashid, A. H., Abdulaziz, S. G., & Yasin, N. B. M. (2018). Technology acceptance models to improve adoption of health information systems. Journal of Advanced Sciences and Engineering Technologies, 1(1), 17-29. doi: 10.32441/jaset.v1i1.66
Rouleau, G., Gagnon, M.P., Côté, J., Payne-Gagnon, J., Hudson, E., & Dubois, C.-A. (2017). Impact of information and communication technologies on nursing care: results of an overview of systematic reviews. Journal of Medical Internet Research, 19(4), e122.
Samadbeik, M., Shahrokhi, N., Saremian, M., Garavand, A., & Birjandi, M. (2017). Information processing in nursing information systems: An evaluation study from a developing country. Iranian Journal of Nursing and Midwifery research, 22(5), 377.
Shahzad, K., Jianqiu, Z., Zia, M. A., Shaheen, A., & Sardar, T. (2018). Essential factors for adopting hospital information system: a case study from Pakistan. International Journal of Computers and Applications, 1-12. doi: 10.1080/1206212X.2018.1504460
Sockolow, P. S., Rogers, M., Bowles, K. H., Hand, K. E., & George, J. (2014). Challenges and facilitators to nurse use of a guideline-based nursing information system: recommendations for nurse executives. Applied Nursing Research, 27(1), 25-32.
Valizadeh, L., & Bagheriyeh, F. (2018). Challenges and barriers of nursing information system: an integrative review. Journal of Urmia Nursing And Midwifery Faculty, 15(12), 874-884.
Vitari, C., & Ologeanu-Taddei, R. (2018). The intention to use an electronic health record and its antecedents among three different categories of clinical staff. BMC Health Services Research, 18(1), 194. doi:10.1186/s12913-018-3022-0
Vollmer, A.M., Prokosch, H.U., Evans, R. S., & Kuttler, K. G. (2016). Evaluation of acceptance of nursing information system in a german and american hospital. In Sermeus, W., Procter, P. M., & Weber, P. (Eds.). Nursing Informatics 2016: EHealth for All: Every Level Collaboration – From Project to Realization. IOS Press.
Zheng, Y., Wang, J., Doll, W., Deng, X., & Williams, M. (2018). The impact of organisational support, technical support, and self-efficacy on faculty perceived benefits of using learning management system. Behaviour & Information Technology, 37(4), 311-319.
Ching-Ching Cheng, M.S., RN, holds a master’s degree from the Department of Information Management, Yuan Ze University, Taiwan, and is a supervisor, Department of Nursing, Chang Gung Memorial Hospital, Linkou, Taiwan, R.O.C.
Chien-Lung Chan, Ph.D., holds doctorate degree in industrial engineering from the University of Wisconsin-Madison and is a professor, Department of Information Management, Yuan Ze University, Taiwan, R.O.C.
Lichin Chen, Ph.D., holds a doctorate degree from the Graduate Institute of Biomedical Electronics and Bioinformatics, National Taiwan University, Taiwan, and is a special technician, Research Center for Information Technology Innovation of Academia Sinica, Taiwan, R.O.C.
Sophie Huey-Ming Guo, Ph.D., RN holds a doctorate degree from the Institute of Business and Management, Chang Gung University, Taiwan, and is an assistant professor, Nursing Department of Mackay Medical College, Taiwan, R.O.C.