Kokol, P. and Blažun Vošner, H. (Summer 2017). Nursing informatics research: A bibliometric analysis of funding patterns. Online Journal of Nursing Informatics (OJNI), 21 (2), Available at http://www.himss.org/ojni
Research funding can have enormous positive impact on nursing informatics research in general and on information and communication technology integration into nursing processes in particular. The objective of the study was to analyze the scope, volume and patterns of nursing informatics research funding. The corpus was harvested from the Web of Science Core Collection using the search string nursing informatic* in information source titles, abstracts and keywords. The corpus was analyzed using descriptive bibliometric and bibliometric mapping. In the study, 1017 information sources were retrieved, and among them 149 (14.7%) were funded. Funded information sources were published by authors affiliated to 40 countries and 284 institutions. Between 2008 and 2013, there was a strong rise in the number of funded information sources, with the number stabilizing after 2013. The most productive country, producing more than two thirds of funded information sources, is the United States of America. The most productive funding agency is the National Institutes of Health. Bibliometric mapping revealed four prolific funded topics; processes and models, web-based emergency care, informatics competency, and skills and interventions. Funded information sources are primarily published in journals contrary to non – funded information sources which are frequently published in conference proceedings. The funded information sources are significantly more cited and more frequently published under open access policy than non-funded sources.
Research funding represents an important factor in conducting research projects (Jowkar, Didegah, & Gazni, 2011). Hence, research institutions compete for research grants and funds on a regular basis (Garcia & Sanz-Menéndez, 2005). A budget given to a research institution enhances its ability to hire eminent researchers, provide access to advanced technology and equipment, extend international collaboration, enable publishing in open access journals, attend top conferences, or hire external organizations to support preparation of competitive project proposals. Consequently, such institutions perform better research and publish more high quality publications. For example, the citation counts of publications from sponsored research are notably higher than that of other research (Zhao, 2010). This finding was confirmed by Gallo et al. (2014), who additionally discovered that the number of citations is not correlated to the amount of funds awarded. Arora and Gambardella (1998) showed that obtaining grants leads to a significant increase in the rate of collaboration and productivity.
The link between research funding and researchers’ performance has also been discovered in other studies (Liefner, 2003; Gulbrandsen & Smeby, 2005; Bornmann, Leydesdorff, & Van den Besselaar, 2010). However, Payne and Siow (2003) showed that the increase in research funding marginally increased the research output, but not necessarily its quality. On the other hand, the percentages of the gross domestic product spent on academic research in different countries are significantly correlated with the percentage of researchers in the population (Leydesdorff & Wagner, 2009). Contrary to the findings above, Randall (2012) claims that nurses must try to obtain funding, however nursing research is not a competition; therefore, the competitive model of funding is counterproductive in nursing research. In the more general health care domain, the benefits of funding also include outcomes improvements and health care cost saving (Wooding, Hanney, Buxton, & Grant, 2005; Leroy, Habicht, Pelto, & Bertozzi, 2007).
Information and communication technology (ICT) plays a significant role in contemporary nursing (Swenty & Titzer, 2014). Integrating ICT and advanced nursing improve nursing practices in all settings. This integration is beginning to yield measurable clinical benefits, such as quality of care delivery and improved health outcomes. However, to satisfy the needs of health professionals and patients, policies including expanded research funding are needed (Krist, et al., 2015) The knowledge about funding bodies, research challenges, “hot topics” and “main players” in the field of nursing informatics (NI) is of vital importance for NI researchers, as well as for their institutions’ managers and decision makers. A useful method to acquire this knowledge is the bibliometric analysis of the research literature production in the NI field.
Bibliometrics measure properties of written materials like the number of articles on a particular topic; dynamics of literature production in a particular period; distribution of articles in source titles; most productive countries, institutions and authors; impact factors of source titles; number of cites per paper; and most cited papers (Garfield, 2006; De Bellis, 2009). A more recent technique used in bibliometrics is bibliometric mapping, which visualizes literature production according to word co-occurrences, co-citations, and similar in form to maps (citation maps, term maps, etc.) (Alfonzo, Sakraida, & Hastings - Tolsma, 2014). A popular bibliometric mapping software is the VOSviewer (Young & Hamer, 2013). It creates so-called term maps or scientific landscapes. According to van Eck & Waltman (2007), bibliometric maps can reveal:
- term relatedness expressed by proximity of terms on the map,
- associations between terms producing term cluster,
- historical characteristic of terms, and
- the importance of a term indicated by the size of the term font.
Data source and corpus
The Web of Science Core Collection (WoS) (Thomson Reuters, United States of America (USA)) contains reliable and complete data about the funded research presented in information sources; hence, this collection was selected as a bibliographical database from which the corpus was formed. The search keyword string used was nursing informatic* and the search was performed in information source titles, abstracts and keywords. Asterisk (*) is a wildcard character that represents zero or more characters in a search string. All types of information sources (papers, articles, reviews, conference papers) written in English in the whole time period covered by WoS, except the year 2015, were included in the corpus. The search was performed on August 19, 2015.
Data extraction and analysis
The publishing year, source title, author's institutional and country affiliation, funding information, WoS category, number of citations, and type of source title (open access or not) for funded information sources were extracted as text files by WoS built-in functions Refine and Analyse. These files were imported to Excel (Microsoft Office, Microsoft Corporation, USA) to derive the most productive countries, institutions and source titles; create a literature production dynamics graph; and calculate basic statistics. The full record in tab-delimited format for information sources reporting funded research (ISRFR) was exported from WoS and imported to VOSViewer for bibliometric mapping analysis. For the analysis, VOSViewer default parameters were used. The impact of funding on the publication and the citation patterns of funded and non-funded information sources were analysed using descriptive statistics and student-t test.
Results and discussion
In the study, 1017 information sources, among them 607 articles, 25 reviews and 375 conference papers, were retrieved. 149 information sources (14.7%) reported funding. The percentage of ISRFRs is a bit smaller than the percentage of ISRFRs in nursing informatics competences literature production (15.0%) as reported by Kokol et al. (2015).
ISRFRs were published by 599 authors affiliated to 40 countries and 284 institutions. The maximal number of information sources per one author was five. The average number of authors per information source was four, and the average number of institutions almost two. This last fact indicates that funded NI research is in general inter-institutional. Most of the funded research is performed in the USA (n = 86), followed by Australia (n=10), Brazil (n=10), United Kingdom (n=8), Taiwan (n=7), and Canada (n=6). The most productive funding agencies are the National Institutes of Health (NIH) (n=18), National Library of Medicine (NLM) (n=12), Fogarty International Center (n=3) and the International Council of Nurses (n= 3). Among non-USA agencies, the Australian Government Department of Health and the Ageing and Italian Ministry of Health are the most productive, funding two information sources each. Table 1 presents a summary of these findings.
The dynamics of funded research literature production are shown in Figure 1. Until 2008, only two information sources reported funded research. However, after 2008, funded research literature production shows a strong positive trend, reaching a steady state in 2013. The beginning of the positive trend might be associated with:
- the publishing of the seminal paper Guideposts to the Future—An Agenda for Nursing Informatics (McCormik, et al., 2007),
- the Tiger initiative, which started in 2004 and began its most intensive period after 2007 (Weaver & Skiba, 2006), and
- the electronic health records (EHR) incentive programs defined in the Health Information Technology Economy and Clinical Health (HITECH)Act provisions of the American Recovery and Reinvestment Act (Blumenthal & Tavenner, 2010).
The most funded WoS categories represent one way to identify hot topics in nursing informatics research, shown in Table 2. While there is no WoS category for nursing informatics, nursing and medical informatics are two of the most-funded categories. The absence of nursing informatics in the WoS categories clearly presents an obstacle in identifying hot topics for nursing informatics research. In addition, WoS categories are too broad to enable hot topics identification on a more detailed research terms level, using WoS services alone.
To overcome these weaknesses, we employed the bibliometric mapping approach to create the term map presented in Figure 2. Based on the induced clusters, four main hot research topics were identified. Furthermore, cluster terms represent hot topics on a finer level. Four main hot topics are:
- Processes and models (green colour): the terms in this cluster indicate the importance of ICT process implementation in nursing practice. Funded NI research on this topic focuses on diabetes and intensive insulin therapy (iit),
- Web-based emergency care (yellow colour): the associations of terms in this cluster shows that “care transition” regarding “emergency department” requires research in “internet” and “communication networks”,
- Informatics competency and skills (red colour): this topic reveals the importance of “education” and “training” to provide “knowledge,” “informatics competency” and “skill” to a “nursing student” to ensure “patient safety,”
- Interventions (blue colour) topic is associating the terms “usual care,” “nursing home” and “mortality”. This topic might indicate the importance of use of ICT and NI in “geriatric nursing.”
All four funded topics are interwoven through four central terms: “communication,” “intervention,” “diabetes” and “diagnosis.” This intertwining once again shows the focus of funded NI research on diabetes. The strong relation between them implied by their proximity on the scientific landscape indicates that they must be holistically researched to be funded.
The chronological evolution of funded NI research topics is shown in Figure 3. During the period of 2008-2010, most agencies started funding research topics associated with education (“nursing student,” “education,” “training skill,” and “knowledge”) and internet (“internet” and “web2ohs”). On the micro level, the transition to the next period (2011-2012) can be seen in term changes like “intensive insulin therapy” (iit) to “intensive insulin therapy computerised decision support system” (iit cdss), from “skills” to “competency” and from “internet” to “access,” “open record act” (ora), “services” and “communication.” The state-of-the-art funded research (2013-2014) is focused on real world facilities like “skilled nursing facility,” “emergency department” and “nursing home” in combination with “video telehealth.”
Impact of funding
Our analysis shows that funding enabled authors to cover publishing fees and publish their research in open journals (9.5%) more frequently than those reporting non-funded research (7.8%). In addition, funded research is primarily published in journals, while non-funded tends to be published in conference proceedings. Both of these facts may indicate that the quality of funded research is better compared to non-funded research, also resulting in significantly higher citation rates for ISRFRs (8.14 ± 34.50 cites to 4.66 ± 6.37 cites for non-ISRFRs; p < 0.01).
Our study showed that bibliometric analysis of the corpus of information sources reporting funded research can reveal interesting and helpful information about funding patterns in NI research. They reveal that nursing informatics research became more intensively funded after 2008; however, the positive trend ended in 2013. The analysis also identified the largest NI research sponsors and most popular funded topics. This knowledge may help NI researchers to increase their chances to get funded. On the other hand, bibliometric analysis may help policy makers to identify underfunded, but important research topics. Furthermore, the analysis showed the importance of funding for research results dissemination – funded research was published in more prestigious source tittles and achieved significantly higher citation rates. Last but not least, the analysis showed which topics should be researched together to improve the possibility of funding.
Alfonzo, P. M., Sakraida, T. J., & Hastings - Tolsma, M. (2014). Bibliometrics: visualizing the impact of nursing research. Online Journal of Nursing Informatics, 18(1), e16.
Arora, A., & Gambardella, A. (1998, April). The impact of NSF support for basic research in economics. Retrieved May 18, 2015, from https://ssrn.com/abstract=163309 or http://dx.doi.org/10.2139/ssrn.163309.
Blumenthal, D., & Tavenner, M. (2010). The "meaningful use" regulation for electronic health records. The New England Journal of Medicine, 363(6), 501-504.
Bornmann, L., Leydesdorff, L., & Van den Besselaar, P. (2010). A meta-evaluation of scientific research proposals: different ways of comparing rejected to awarded applications. Journal of Informetrics, 4(3), 211–220.
De Bellis, N. (2009). Bibliometrics and Citation Analysis. Lanham, Maryland, Toronto, Plymouth, UK: The Scarecrow Press, Inc.
Gallo, S. A., Carpenter, A. S., Irwin, D., McPartland, C. D., Travis, J., Reynders, S., . . . Glisson, S. R. (2014). The validation of peer review through research impact measures and the implications for funding strategies. PLOS One, 9(9), e106474.
Garcia, C. E., & Sanz-Menéndez, L. (2005). Competition for funding as an indicator of research. Scientometrics, 64(3), 271-300.
Garfield, E. (2006). The history and meaning of the journal impact factor. The Journal of American Medical Association, 295(1), 90-93.
Gulbrandsen, M., & Smeby, J. C. (2005). Industry funding and university professors’ research. Research Policy, 34(6), 932-950.
Jowkar, A., Didegah, F., & Gazni, A. (2011). The effect of funding on academic research impact: a case study of Iranian publications. Aslib Proceedings, 63(6), 593-562.
Kokol, P., Blažun Vošner, H., Železnik, D., Vošner, J., & Saranto, K. (2015). Bibliometric patterns of the research literature production on nursing informatics competence. Journal of Nursing Education, 54(10), 565-571.
Krist, A., Green, L., Phillips, R. L., Beasley, J. W., DeVoe, J. E., Klinkman, M. S., . . . Burdick, T. (2015). Health information technology needs help from primary care researchers. Journal of the American Board of Family Medicine, 28(3), 306-310.
Leroy, J. L., Habicht, J. P., Pelto, G., & Bertozzi, S. M. (2007). Current priorities in health research funding and lack of impact on the number of child deaths per year. American Journal of Public Health, 97(2), 219-223.
Leydesdorff, L., & Wagner, C. (2009). Macro-level indicators of the relations between research funding and research output. Journal of Informetrics, 3(4), 353-362.
Liefner, I. (2003). Funding, resource allocation, and performance in higher education systems. Higher Education, 46(4), 469-489.
McCormik, K. A., Delaney, C. J., Brennan, P. F., Effken, J. A., Kendrick, K., Murphy, J., . . . Westra, B. L. (2007). Guideposts to the future—an agenda for nursing informatics. Journal of the American Medical Informatics Association, 14(1), 19-24.
Payne, A. A., & Siow, A. (2003). Does federal research funding increase university research output? The B.E. Journal of Economic Analysis & Policy, 3(1), 1-24.
Randall, D. (2012). The guiding of research funding: corrosive competition or stability, sustainability and serendipity. Journal od Advanced Nursing, 68(12), 2595-2596.
Swenty, C., & Titzer, J. (2014). A sense of urgency: integrating technology and informatics in advance practice nursing education. The Journal for Nurse Practitioners, 10(10), e57-e67.
van Eck, N. J., & Waltman, L. (2007). Bibliometric mapping of the computational intelligence field. International Journal of Uncertainty, Fuzziness and Knowledge-Based Systems, 15(5), 625-645.
Weaver, C. A., & Skiba, D. (2006). ANI connection. TIGER initiative: addressing information. CIN: Computers, Informatics, Nursing, 24(3), 175-176.
Wooding, S., Hanney, S., Buxton, M., & Grant, J. (2005). Payback arising from research funding: Evaluation of the Arthritis Research Campaign. Rheumatology, 44(9), 1145-1156.
Young, F. W., & Hamer, R. M. (2013). Multidimensional Scaling: History, Theory, and Applications. Psychology Press: Florence, Kentucky, USA.
Zhao, D. (2010). Characteristics and impact of grant-funded research: a case study of the library. Scientometrics, 84(2), 293-306.
Professor Peter Kokol has a PhD in computer science. He is the head of the Laboratory for System Design at the Faculty of Electrical Engineering and Computer Science at the University of Maribor. He was the dean of the Faculty of Health Sciences and the director of the independent Centre for Interdisciplinary and Multidisciplinary Studies and Research. He has written over 700 technical and research papers and co-authored some textbooks. He was the president of the IEEE Committee on Computational Medicine. His main skills are in software engineering, health informatics, data mining and complex system theory.
Helena Blažun Vošner, PhD, PhD, is a head of Center for International Cooperation at the University of Maribor, Faculty of Health Sciences, Slovenia. She obtained her first PhD at the University of Maribor in the field of management and organization of online services and applications for elderly people and her second PhD in health care sciences at the University of Eastern Finland. While involved in the project Distant E-learning for Nursing Education (DENED), she was awarded a grant from the Johns Hopkins University School of Nursing. She has cooperated in many international projects closely related to health informatics, including the coordination of the international summer school, “Health living – HEALTH,” for which she was awarded the national award of excellence.