Health Education Research, Vol. 16, No. 6, 747-763,
December 2001
© 2001 Oxford University Press
Online research to guide knowledge management planning
Public Health Informatics Research Laboratory, Department of Public and Community Health, University of Maryland, College Park, MD 20742, USA
 |
Abstract |
|---|
 Top Abstract IntroductionMethodsResultsDiscussionReferences |
|---|
The current paper describes the process and results of an effort to find a way to effectively manage and diffuse prevention knowledge. This study shows the role that today's communication technologies can play in ensuring collaboration and participation in both the design and use of a knowledge management system (KMS) for prevention research, practice and policy. In the context of this study, `prevention research' includes primary through tertiary prevention efforts consistent with general applied public health research in the US. An online Delphi study was used to engage a set of prevention research constituencies in the design of a mechanism to enhance the potential for effective technology transfer. A three-round Delphi was conducted with 58 stakeholders and key informants involved in prevention: government-level policy makers, researchers and front-line practitioners. The study resulted in consensus on 34 functions and 32 output/content elements of a proposed web-based KMS called PreventionEffects.net. The paper also describes the implications of both the processes of development and the benefits of the proposed system for those interested in prevention.
|
Introduction |
|---|
|
TopAbstractIntroductionMethodsResultsDiscussionReferences |
|---|
Effective technology transfer in today's information-rich environment is a challenging task. As the speed with which new information is generated increases, this task becomes even more difficult. The effective application of sound public health science in practice and policy environments grows more complex with time and technology. Further, the diversity in the constituencies involved in public health science and practice makes effective technology transfer an even greater challenge. The purpose of this paper is to describe an effort to find a way to effectively manage and diffuse the science base of public health. Delphi methodology offered a process that could efficiently engage all constituencies in the design of a mechanism to enhance technology transfer. The study shows the role that today's communication technologies can play in ensuring collaboration and participation in both the design and use of a knowledge management system (KMS).
The research process involves five phases. It begins with foundational research; evolves into developmental, efficacy and effectiveness studies; and ends with research on effective diffusion and dissemination strategies (Harlan, 1998
; Holder et al., 1999). These phases allow the initial development of basic understanding of relationships and causal factors, and then the identification and manipulation of intervenable factors in controlled and real-world situations. The final phase of research allows best and promising practices found to be effective in certain contexts to be applied to other settings, communities and cultures.
Research evidence has grown in recent years to demonstrate the efficacy and effectiveness of public health research. For example, in the US, the Centers for Disease Control and Prevention (CDC) has a Health Promotion and Disease Prevention Research Center (PRC) Program with a network of 23 PRCs and three Urban Research Centers (URCs) (Centers for Disease Control and Prevention, 2000). (Another PRC is soon to be added to this network.) The PRC Program has made great strides in conducting excellent community-based disease prevention and health promotion research in a wide variety of public health areas. The research activities (prevention research) found at the PRCs range from primary through tertiary prevention in a wide variety of substantive areas and audiences.
Despite such positive progress in the first four phases of the research process, the success of disseminating prevention research knowledge is less clear. In a 1997 assessment of the PRC program, the Institute of Medicine (IOM) found that the PRCs did not adequately document the translation of their research knowledge into public health programs and policy, a key task of the centers (Institute of Medicine, 1997).
Translating research into practice, sometimes called `technology transfer', is important for improving the efficiency and effectiveness of prevention programs. Some have argued for the need for organizing our research knowledge as a means for effective dissemination. Among the first was Archie Cochrane, a British epidemiologist who suggested the need for effective organization of health care research (Cochrane, 1972). Effective organization involves having programs like the PRC Program find a way to document their translation efforts better as well as improving access others have to prevention research knowledge so that they can effectively apply the knowledge to their own communities and settings. At the current time, research translation would benefit four primary audiences: (1) front-line prevention personnel (referred to here as practitioners), (2) prevention researchers, (3) policy makers likely to influence either the field or practice of prevention and (4) media professionals who may report on prevention research findings or programs.
Spurred on by the IOM findings, the CDC began to explore ways to improve their translation efforts, for the PRCs and URCs, and for others interested in public health disease prevention and health promotion. By starting with the PRC Program, CDC had the benefit of a national research network with nearly 15 years of research products that could help seed an initial system. CDC recognized that others would be able to benefit from such a system and the system would benefit from including a wider range of expertise. They also found that improving the efficiency of prevention programs while controlling costs is a public health priority and using advanced communications technologies provided a primary strategy for accomplishing this important effort. As Green (Green, 1983) has stated:
The World Health Organization has made `technology transfer' and `health education' two of the pillars of its global strategy of `Health for All by the Year 2000'. But its caveat with respect to technology transfer is that the transfer must involve `appropriate technology', meaning technology that can be applied and managed locally to analyze and solve a people's own health problems. The caveat with respect to health education is that it must involve and enable people to take control of the determinants of their own health.
Given these circumstances, CDC felt that a computerized KMS might benefit both the PRC program and the public health system in general. Other related strategies have been tried or are in progress in other countries. Two in particular are important to note here as background for the CDC knowledge management initiative.
- The Cochrane Collaboration. The Cochrane Centre was opened in Oxford in 1992. One year later, spurred on by Cochrane's 1972 call for organization of the randomized control trial literature in the health care field, a group of international researchers initiated the Cochrane Collaboration during an inaugural meeting. As a result of that collaboration, Cochrane Reviews are now published for researchers, practitioners and consumers on a regular basis in the Cochrane Database of Systematic Reviews. These reviews are designed to provide systematic evidence for best practices in health care (Cochrane Collaboration, 2001).
- Health Development Authority (HDA). In the summer of 1999, the National Health Service (NHS) in the UK published a White Paper entitled Saving Lives, Our Healthier Nation (Department of Health, 1999). The paper detailed the health costs of failing to take into account the social, environmental and economic factors influencing the health of all in England, particularly the worst off, and called for greater attention to reducing inequities. In April 2000, the NHS empowered the HDA to address the issues raised in this White Paper. The HDA's mission is 3-fold: to gather evidence of what works, to advise on good practice and to support all those working to improve the public's health. Among the strategies used by the HDA is their Evidence Base (HDA Evidence Base, 2001), an online database designed to deliver a wide range of information to support evidence-based decision making and practice among health professionals. This initiative is one of several in the NHS taking significant steps towards accomplishing the difficult translation of research into practice. Others include the National Electronic Library for Health (NELH), the National Electronic Library for Public Health (NELPH), Health Activity Zones Network (HAZnet) and Our Healthier Nation in Practice (OHNP).
A great deal of work is also going on internationally in the field of knowledge management (Buchan and Hanka, 1977; Hanka, 1977; Heathfield et al., 1998; Keeling and Lambert, 2000; O'Brien and Cambouropoulos, 2000; Wyatt, 2001). Compared to even a few years ago, finding technology capable of knowledge management is not an obstacle. Business and other sectors now employ knowledge management technologies that public health can also use to construct an efficient system that translates prevention research findings into user-friendly outputs, tailored to the needs of the prevention community. However, such a system would need to be designed in such a way that made it relevant, useful and appealing to the broad constituencies of public health practitioners, policy makers, researchers and media professionals. As noted by Green, this strategy is consistent with the tenets of effective technology transfer (Green, 1983).
As with prevention planning efforts, technology development efforts must involve the community from the beginning in the planning process to ensure that the resulting application truly reflects their viewpoints and needs—in its content, appropriate strategies and type of application (Gold and Atkinson, 1999). Therefore, CDC sought the input of key stakeholders from within their organization, the business sector, and the PRCs and URCs. These stakeholders met in an Advisory Committee meeting in June 2000, where they were presented initial ideas about the KMS. CDC envisioned that the KMS would be an electronic system that organized and added value to existing prevention research knowledge. Such a system would be capable of yielding the information across a wide range of health problem areas, including:
- Scope of the health issue—its prevalence, severity, trends over time and impact on quality of life
- Methods of health promotion and disease prevention tactics and strategies, classified by effect desired and by relevant demographic indicators
- Health effects likely to result from the effective application of such tactics and strategies
- Costs of those tactics and strategies and their potential impacts beyond health
- Health and social policy implications
- Research gaps and needs
The CDC also shared possible KMS features that would, given appropriate protocols and quality assurance standards, enable users to augment the system. For example, practitioners could access the system and document their experience in applying specific tactics and strategies they drew from the system's knowledge base. This would provide a much-needed measure of the utility of a given method or strategy. Likewise researchers—whose findings were used to describe specific tactics and strategies and who learned that their work had influenced practice or policy—could enter such documentation into the system. These application markers would provide indicators that could track the extent to which research was indeed being translated into practice or policy, just what the IOM called for in their report on the PRC Program.
The meeting attendees quickly came to the conclusion that a KMS would facilitate knowledge transfer. They also came to the conclusion that others who were not at the meeting could offer valuable insights into the development of the system. The attendees recommended inviting others to help guide the effort, especially those from practitioner and policy-maker audiences who were not well represented at the meeting. Talking to these groups was considered important for developing a relevant system, but it also allowed the effort to meet the goals of participatory research: democratization of knowledge creation and social change (Stoecker and Bonacich, 1992). Democratization of knowledge creation means that people are engaged in the research process and are empowered by sharing in knowledge development rather than being objects to be studied. Social change means working toward longer-term restructuring of social system, in this case the system that led to inadequate research translation. By engaging all groups throughout the process, the KMS development effort will model a new way of collaborating that is crucial to its success and to the success of technology transfer.
Expanding the community participation faced two challenges: a short time frame and the geographic dispersion of the other constituencies the meeting attendees wanted to engage. As a result, a consensus development process called the Delphi Method was planned (Green and Kreuter, 1999). It was chosen because it allowed us to work at a distance with a variety of groups. In addition, differences of opinion among key people could be resolved more easily than in a face-to-face meeting where group conformity, prestige, power and politics can influence ideas and opinions (McKillip, 1987).
A Delphi study involves having a group of experts, opinion leaders and informants complete a series of questionnaires. Usually, the rounds of data collection are focused on a series of processes that build on each other: (1) brainstorming, (2) clarification and refinement, and (3) prioritization based on one or more criteria of interest. Delphi studies have traditionally taken place through the mail, but this project was accomplished using E-mail reminders and online surveys, which reduced the time and cost of a mail effort. It also took advantage of some of the technologies that a KMS would use. Below is a description of the methods used and our findings, followed by a discussion of the implications of these results.
|
Methods |
|---|
|
TopAbstractIntroductionMethodsResultsDiscussionReferences |
|---|
Delphi Method
A Delphi Method was chosen because it allows people who are not in the same room to come to consensus on some issue of importance. The Delphi Method also ensures that all participants have an equal voice in the outcome because they do not experience the interpersonal dynamics that occur in an in-person meeting. This aspect was particularly important in this effort because the respondents ranged from community organizers to research academicians.
The study involved three rounds of online surveys. Participants were invited via E-mail to be part of the study. Each survey was available for 2 weeks. At the beginning of the 2 weeks, participants were sent an E-mail that asked them to visit a password-protected website and to complete an online survey form by a certain date.
Participants
The initial group invited to participate in the Delphi process were attendees of a meeting to discuss the form and function of the proposed KMS, including CDC representatives (n = 10) and members of the PRC Knowledge Management Advisory Group (n = 5). In order to get input from a broad range of people who were potential users of the system, the 15 meeting attendees and two advisory group members who could not attend the meeting were asked to nominate researchers and front-line community health workers to invite to participate in the Delphi process. The nominations were based on willingness to respond and ability to represent some segment of the intended audience for the prevention KMS. A total of 35 people were nominated as well as eight members of the PRC Steering Committee. At least eight nominees were considered front-line community practitioners. However, most of the participants from the PRCs—Steering Committee and nominees alike—considered themselves to play multiple roles: researcher, practitioner, etc. Two of the nominees were also on the PRC Steering Committee; therefore 58 people were invited to participate in the Delphi process.
Online data collection
We prepared survey instruments using html-based `forms'. The survey was deployed on a research-oriented website with links to further information about Delphi studies. Participants used a hyperlink to access the online form. Only those who used the correct username and password were able to view and complete the survey.
Rather than using a `mailto' strategy to return the forms via E-mail, we used a cgi-bulkmail program so users did not have to be at a computer configured for Internet mail. Irrespective of computer or browser configurations, anyone having access to the World Wide Web could submit survey responses. Further, the cgi-based bulkmail program automatically stripped unwanted information from a return and entered the respondent's answers directly into a database on the server-side. This strategy also allowed anonymous survey completion.
Participants were asked to respond to any or all of the three rounds of surveys because the timeframe was limited. In follow-up rounds of Delphi studies, it is customary for participants to see how their responses compare to the responses of other participants. Because participants could respond to any or all of the three rounds, the follow-up questionnaire did not include individual ratings. Instead, participants could view their responses to the previous questionnaire by using a hyperlink with a randomly assigned participant number.
Instrumentation
In June 2000, a planning group was brought together to help identify some of the specific characteristics and functions that would be useful in a prevention research and practice KMS. This group included representatives from CDC and an Advisory Group of representatives from the PRCs and URCs. As the result of a brainstorming session, 32 desired KMS components were identified, of which 15 were categorized as functions (i.e. capabilities the system would have to address the needs of its intended users) and 17 were categorized as output/content elements (i.e. information that the system would contain and ways the system would provide that information to its intended users). These served as the basis of the instrument in the first round of the Delphi study. The instruments for the two subsequent rounds of the Delphi study were based on the responses to the round before.
Round 1 survey form
The first round of the PRC Online Delphi Survey was designed to gather reactions to the initial functions and output/content elements in the KMS. Survey respondents were asked to rate the importance of each of the 15 functions and 17 output/content items on a five-point Likert scale (1 = not important to 5 = extremely important). A default response was checked, NR for `no response', and each respondent was allowed to leave an NR if they chose. Respondents were also asked to make recommendations for additional system functions and output/content components for the KMS. The group was given approximately 2 weeks to respond to the survey.
Round 2 survey form
The first step in creation of the second round instrument was to compile the findings from the first round. Means and standard deviations for each item were computed, and NR responses were treated as missing data. Respondents were asked to re-rate the importance of the originally listed functions and output/content components with knowledge of their original assessments of the items (they could access their Round 1 ratings through a hyperlink from a unique identification number) and the group mean item ratings (next to the items). They were presented with the functions and output/content elements that were newly recommended in Round 1 so they could rate them according to importance to the KMS and they had one more chance to make recommendations for elements to be considered for the KMS. Lastly, the respondents were asked to recommend three prevention research practitioners and three applied prevention research projects that might be contacted to provide information and advice desired by the prevention community that might not be included in published reports. Round 2 of the Delphi study began approximately 2 days after the first round ended and lasted 2 weeks.
Round 3 survey form
As before, the initial step in the creation of the Round 3 instrument was to compile the findings from Round 2. Again, respondents were presented the functions and output/content elements with the individual and group means and standard deviations for each. As in Round 2, each respondent had a user ID with which they could hyperlink to their Round 1 and 2 instrument responses. In Round 3 they were asked to prioritize by `voting' for the 10 elements they considered a priority to include in a first version of the KMS. Respondents were also asked to refer to lists of expert researchers and practitioners and exemplary programs identified in Round 2 (via hyperlinks) and to suggest key questions they would want answered and information they would want gathered (if they were given the chance).
|
Results |
|---|
|
TopAbstractIntroductionMethodsResultsDiscussionReferences |
|---|
Delphi study participants
Of the 58 people invited to participate in the Delphi study, 40 people responded to at least one of the three rounds, with 28 respondents for Round 1, 30 respondents for Round 2 and 29 respondents for Round 3. Table I
shows the response patterns among the 40 participants and the 58 invitees. Although 42.5% of the respondents participated in all three rounds, further analysis showed that Round 1 respondents were likely to complete successive rounds; 82.1% of Round 1 respondents completed at least two rounds and 60.7% completed all three rounds.
|
System functions
Table II
summarizes the system function findings, listed in descending order from the function that received the most priority votes in Round 3 to the function that received the least votes. Table II also provides the importance ratings (means and standard deviations) from Rounds 1 and 2. Functions for which `NA' is listed under Round 1 were proposed by respondents during that round and were presented and rated in Round 2.
|
An examination of the standard deviations among the items rated in both Round 1 and Round 2 shows that the variability in importance ratings decreased in Round 2 in almost all cases (12 out of 15 cases, or 80% of the time). This finding suggests that the consensus process of reviewing individual and group scores influenced reassessment of the functions.
Of the results presented in Table II
, 12 functions were identified by at least 50% of the final round respondents as important to include in the initial version of the KMS. The priority functions are highly but not perfectly correlated with the functions that were rated as most important. This result is understandable because it reflects respondents' distinction between importance to an overall system and desirability in a first iteration.
The most highly rated function—that the system should be user friendly—is telling and a basic assumption of system design. The remaining 11 functions seem to fall into three general categories: (1) organizational structure, (2) user-centered capabilities and (3) interactive capabilities. The organizational structure functions include those that support organizing and coding the knowledge base according to descriptive information, standards of evidence, and a disease and health behavior taxonomy. Functions that show that the system is user-centered include those that confidentially determine user needs, tailor the output to user needs, and provide tools to facilitate browsing and downloading knowledge. The interactive functions facilitate user exploration, input and contribution to the system, which will add value to the knowledge base for the entire community of KMS users.
About a third of the respondents rated two other functions as priorities to the first KMS: (1) providing technical assistance and tutorials, and (2) connecting a user with others having similar interests. The first of these functions could be grouped with the other user-centered functions; the second could be grouped with the interactive capabilities. This finding suggests that the system developer consider these functions as possible additions to the initial KMS.
Several of the functions that received fewer priority votes were related to understanding who the user is and what the user intends to do with the information (determining what the user intends to do with the information through profiling user needs during a session). Because four of the functions that received the fewest votes were recommended during the initial Advisory Committee planning meeting, we further compared the responses of Advisory Committee meeting attendees with those invited to participate in the Delphi study after the meeting. The t-tests for independent groups showed that only two functions differed significantly across groups in terms of importance ratings in Round 2: `Profiling user needs during a session' (P = 0.00) and `Determining what grant opportunities would apply to the user's interests' (P = 0.05). The former was rated significantly more important by the meeting attendees, the later significantly more important by the invitees.
When we compared the proportion of each group that considered each function a priority, we found that several functions that appealed to one group were not priorities to the other group. The following functions were supported by at least 50% of the invitees but did not receive support from the meeting attendees: user browsing, content organized by descriptive information, user downloads and user rating of the value of information. Only one of the functions supported by the Advisory Committee meeting attendees—maintaining a taxonomy to facilitate user navigation of the system—was not supported by at least 50% of the invitees.
System output/content elements
Table III contains a summary of the output/content elements listed in order of vote preferences from the third round, from highest priority to lowest priority. Table III also provides the importance ratings (means and standard deviations) from Rounds 1 and 2. Again, the elements marked `NA' in the Round 1 column were proposed in Round 1 and were not rated until Round 2.
|
As with the functions above, the variability in importance ratings decreased in Round 2 when compared with Round 1 responses on a majority of elements (12 out of 17 cases, or 71%), giving evidence that the consensus process is working. One of the elements—describing the kinds of interventions that exist related to a theme—increased in variability from Round 1 to Round 2 but not significantly.
Of the elements listed in Table III, the first 10 listed were identified as priorities in the first iteration of the KMS by at least 50% of the final round respondents. These 10 output/content elements appeared to fall into three categories of information: (1) information regarding what might reasonably be expected to happen as a result of planned programmatic and research efforts, (2) information on unexpected consequences and barriers to success, and (3) information tailored to community and user needs. The fact that respondents wanted tailored outputs supports the need for user profiling as a core capability of the system, even though they did not rate it among the top functions.
As with the functions, we examined the responses of the two groups (those who participated in the Advisory Committee meeting and those invited to participate in the Delphi following the meeting). The t-tests for independent groups revealed significant differences across groups in four output/content. `Describing what has been tried' (P = 0.04) and `providing information on who else is asking the same questions' (P = 0.05) were rated significantly more important by the Advisory Committee meeting attendees. The invitees rated the other two elements—`describing intervening variables and events that influence success/failure' (P = 0.05) and `providing specific feedback on the context and process variables that underlie successful dissemination of programmatic interventions' (P = 0.02)—more significantly than the meeting attendees. Although the invitees supported including all 10 of these elements, the meeting attendee group did not vote at least 50% of the time for the following ones:
- Providing information on sustainability
- Describing unintended consequences of actions
- Providing the data in forms that are usable within lay community groups
- Describing what does/does not work
- Describing how challenges are dealt with
Prevention practitioners and applied programs
In Round 2, respondents were asked to provide suggestions of prevention practitioners who were in a unique position to add information to the KMS. As part of the Round 3 questionnaire, respondents could see the list of recommended prevention practitioners and suggest questions to ask them to add to the KMS database of knowledge objects to be created. Table IV contains the questions generated in Round 3.
|
Similarly, each respondent was asked in Round 2 to identify exemplary applied programs they would recommend be profiled as part of the knowledge database. Each respondent in Round 3 was able to see the programs recommended and was asked to supply follow-up questions that might be asked of those programs. Table V
contains the follow-up questions suggested by respondents.
|
In both lists of questions, respondents appeared to reiterate their desire for information not commonly covered in published research articles. They wanted help understanding the background issues and unspoken problems that often occur in applied research projects and prevention programs. This feedback was perceived as a crucial guidepost in the overall design of the KMS and pointed to the need to include knowledge beyond what is usually in research and practice databases.
|
Discussion |
|---|
|
TopAbstractIntroductionMethodsResultsDiscussionReferences |
|---|
The work of Archie Cochrane was prophetic in that we are now seeing a confluence of international efforts to organize health-related research for the purpose of improving health outcomes. While a commonality of thought appears to have initiated these efforts, the current effort began with knowledge of both and an interest in the needs and expectations of an expanded audience—one that includes front-line, community practitioners as well as researchers and policy makers. The purpose of the Delphi study was, therefore, to gather data that would be of use in moving forward with establishing the scope, requirements and expectations for a prevention research and practice KMS.
A large group of people were invited to participate in the Delphi study, because it took place over the summer when people were more likely to be on vacations and the Advisory Committee wanted to cast a wide net for nominees. Respondents who participated in the first round were more likely to complete all three rounds of the process. Whether or not technology ability and comfort affected one's ability to participate is difficult to determine because we did not assess invitees on these measures. In general, the Delphi participants were chosen because of their interest in this issue and we were pleased to have the response that we did. Further, only two people required technical assistance during the process. Research is needed to examine the affect of technology ability and comfort on response patterns to online surveys.
The study resulted in the generation and rating of 34 functions and 32 output/content elements. While some of the functions and elements overlapped, clarity and separation came as respondents moved through the rounds. At the end of Round 3, respondents had identified 12 functions that at least 50% felt should be included in the first version of the KMS. These functions were grouped into three broad categories: (1) organizational structure, (2) user-centered capabilities and (3) interactive capabilities that allowed users to add to the system. As with the functions, the importance ratings and final voting produced a set of 10 recommended output/content elements for the first version of the system. These appeared to group into three broad categories of elements: (1) information regarding what might reasonably be expected to happen as a result of planned programmatic and research efforts, (2) information on unexpected consequences and barriers to success, and (3) information tailored to community and user needs. Lastly, respondents posed questions that could be answered by talking to experts and exemplary programs about the real-world challenges and solutions in prevention programs.
System functionality
The mean ratings of system functionality suggested that respondents felt that the functions were all important to the long-term goals of the system. The priority scores revealed that some functions were more needed in the early stages of development. Therefore, we view the mean ratings as recommended priorities for development in a system life cycle rather than absolute judgments for inclusion and exclusion of those functions.
The Delphi study findings on preferred functions provided insight into how to phase the system's development. The magnitude of the development effort, the size of the potential knowledge base and the number of constituencies requires that some form of staging or phasing be done.
According to the findings, the organizational structure of the knowledge base is important for describing, searching and browsing. Therefore, the development effort needs to begin with paying close attention to the taxonomic structure. Using the previous efforts as an example, the potential system could begin with thorough development of specific content areas (similar to the Cochrane Reviews) or with an overarching structure, such as the HDA Evidence Base did when it used the Saving Lives, Our Healthier Nation White Paper for organization purposes. In terms of this latter option, the system could be organized around the priority areas in Healthy People 2010 (Department of Health and Human Services, 2000), a document that details the public health priority areas for the US.
Other early considerations for the development of the system are the type of methodologies and staffing pattern to be used to create and maintain it, given the need for publicly available and up-to-date information. CDC may choose to follow the HDA example and create a centrally managed and funded effort. However, this type of effort requires a large annual budget and staff. Using the international collaborative strategy of the Cochrane Reviews may be more cost-effective but may limit the ability to keep the evidence base current. The PRC Program structure could offer a situation in which a hybrid solution was possible, where different centers were responsible for developing and maintaining different content areas. Public–private partnerships may also offer a structure to enable continued development and maintenance of such a KMS.
An important distinction of the envisioned system compared to other retrieval systems is the inclusion of information with less rigorous evidence. Research other than randomized control trials may indicate important directions for research and practice or patterns of evidence that deserve attention. Businesses have benefited from using this strategy in their knowledge management because it identifies promising practices that will be tomorrow's best practices. Therefore, developers must have a strategy for attaching multiple standards of evidence to each information object. While this requirement will enlarge the database and add complexity to coding procedures, it will expand the options for intervening in the community and take into account the most current knowledge.
The findings on system functionality also showed that the user interface, technical assistance aids, and methods to ensure user privacy and confidentiality are essential elements of design consideration. This means that, once a stable database structure is developed, the design and creation of simple tools and strategies that allow people to use the system should follow. The need for `user friendly' applications is self-evident, yet even simple technology tools are used differentially depending upon their technology literacy. The envisioned system would be developed with attention to user-friendliness without compromising intended functionality.
The findings demonstrated nearly universal agreement on the importance of user feedback to add value to the system and to ensure that the system stays current and meaningful to the end-users. To realize its full potential, the system must establish inviting and efficient protocols that allow users to contribute knowledge to the system. Lastly, user participation would benefit from the appropriate application of both `push' and `pull' technologies. These strategies are commonly used in an Internet environment and complement each other. In the Internet environment there are two sides of a communication loop—the server side (where the web technologies originate) and the client side (at the individual user machine and browser). From the server side, information (text or other media) can be `pushed' to the client (user's browser) based on a user profile or set of known preferences. When a server pushes information to a browser, it keeps the connection open so that it can continue to push additional information. In this way, there is a constant connection between the server and the client and the flow of information can be constant. A good example of this is a `news ticker' or `stock quote ticker' that continually displays headlines or stock quotes on a user's machine.
Pull technology occurs from the client side of the equation, but it originates at the server side. Here, the server software knows the preferences or profile of the user, uses this knowledge to originate a single message to the user and then closes the connection. That message is a group of instructions for the client-side browser to retrieve or pull information from any number of sources on the Internet, including but not limited to those on the original server. An example of this strategy is an E-mail newsletter in which the recipient is able to click on hyperlinked text and go to web pages with the information referred to in the E-mail.
Several issues emerged concerning system functionality in the results. Perhaps the one that is intertwined with most of the intended functionality of the system is the issue of profiling the user. The results clearly show that respondents did not perceive profiling to be one of the early priorities nor the most important element of functionality. One explanation could be that end-users were troubled by the potential threats raised to their privacy and confidentiality. Several horror stories have appeared in today's media outlets about the hazards of allowing personal information to be stored in online databases. When this is coupled with the prospect that a government agency is compiling information about an end-user the concerns might be exacerbated. Another explanation could be that some of the respondents might not have understood the potential benefits of user profiling or the prospects for protection of their privacy and confidentiality. The Delphi questionnaire listed functions such as `anonymous user profiling', `session profiling' and `real-time profiling', which are each difficult concepts in the abstract. Those attending the initial Advisory Committee meeting were presented information on user profiling and discussed it together while the people who did not attend had no such knowledge. For those not participating in this discussion, the absence of clarity on the roles and uses of such profiling information could account for the differences in the importance ratings found between the two groups. Interestingly, some of the priority output/content elements—such as describing how to make programs work in a given user's community—may need to be facilitated through user profiling.
Because of the importance of some form of `profiling' to the customization processes desired in this KMS, an in-depth re-examination of this issue is essential. First, a clear and compelling purpose for profiling must exist, and the end-users should be able to understand its importance. Second, irrespective of the potential value of profiling, any schema adopted to provide this function to the system must be able to protect end-users from inappropriate use of the information. This is an issue that should be re-examined with the Advisory Group and vendor as CDC moves forward with its plans.
System content
Like the elements of system functionality, the mean ratings of the content elements showed that respondents viewed them all favorably and we viewed the priority ratings as recommendations for development order rather than what content should be included in the system. The type of content that emerged and the questions asked revealed that public health practitioners and researchers are looking for greater depth than what is often found in their traditional literature. In addition to information on recommended program practices, the respondents wanted to know about unexpected consequences and barriers and guidance in adapting proven programs to new community settings.
The question posed was `what would you ask these experts' and many of the questions they wanted to ask experts were specific rather than general. However, the crux of many of the questions related to conceptual, procedural and operational issues that only come from those with expertise and field experience. Much of the substance of the answers to these questions would not be found in traditional explicit knowledge sources. Therefore, the database structure needs to accommodate the full range of information on a project, from its inception to institutionalization, as well as information on the why, the how and overcoming barriers to success. Much of this information does not exist in published articles, monographs or reports; rather it is only accessible from the practitioners and researchers themselves. Most knowledge for research and practice only includes the first type of knowledge, which is often called `explicit'. The other knowledge is often called `tacit' and it is usually only available from consultants or experts that come at a high price.
Business-based KMSs include both explicit and tacit knowledge, so the inclusion of such knowledge is possible in a prevention KMS. However, the development of such knowledge presents a challenge in a system designed for public use. In the case of business-related KMSs, the use of the system is integral to the operating practices of the company and therefore mandatory for all employees. Collecting tacit knowledge from human experts on a continuous basis will be challenging. Research is often a competitive process in which experts hold their knowledge close, lest they give away their advantage. Researchers and practitioners also may not share all they know when reporting on their activities because—if things go wrong—they may be perceived negatively, even lose funding. Further, sharing lessons learned takes time and experts are often the busiest people with little time left to devote to such a task.
As part of the development process, the KMS must include a plan for knowledge acquisition. The plan must take into account the demands of collecting knowledge as well as provide for incentives that will overcome the barriers to sharing expertise. For example, a benefit of sharing expert advice may be `micro-publishing' in which contributors get credit for the work uploaded and shared on the system. Another benefit may be feedback from the field.
Because the success of the system depends on the contribution of users to the system (their knowledge as well as their opinions of the knowledge they access and use), the development effort should continue the participatory research process begun with the Advisory Committee and the Delphi study. Such a process will build awareness and comfort with the system as well as and the willingness to contribute. Involving the intended users in development will assure that the system fulfills the wants and needs of its users as well as its potential. For example, CDC could engage large organizations and professional associations during the development and field test. As these employees and memberships learn about and use the system, they will improve their proficiency with it and be able to tell other colleagues about it.
The Delphi Method, by its very nature, is selective about the type of respondents who are invited to participate. The findings here are limited by two issues related to the respondents. First, the initial respondent pool was dominated by participants of a meeting at which there was some consensus regarding the importance of such a system. Thus inviting them to participate in a consensus development process (i.e. The Delphi Process) subsequent to this meeting must be seen as building on work already done. This strategy is not unusual for Delphi studies because they often begin by identifying experts who have comparable past experiences. Second, we expanded the participant list with other stakeholders and representatives from other PRCs and URCs. While this expansion was consistent with the principle of participation that is well known in public health, participation of many diverse groups reduces the ability to achieve convergence of opinion in the Delphi process. In this case we compromised between participation and homogeneity by inviting experts who were deemed by the initial group to have a vested interest in the knowledge management system.
In conclusion, this study verified that the collaborative process guided by the Delphi Method worked in two ways. It both enabled the discovery of important design elements to a prevention KMS and it ensured the buy-in of constituencies regarded as crucial to the success of this effort. The study also demonstrated that technology facilitated an efficient and acceptable way to use consensus-building strategies without the constraints imposed by geographic distance. These findings provided incentive and substance for CDC to move forward in its KMS-based technology transfer efforts. With a working title of PreventionEffects.net, CDC is pursuing support to move forward with the development.
|
Acknowledgments |
|---|
We gratefully acknowledge the participation and contributions of the PRC Advisory Committee and the Delphi study participants. We also thank the Centers for Disease Control and Prevention for their interest and enthusiasm in pursuing this effort, and, in particular, wish to thank three individuals who played essential roles in the conduct of this study: Bobby Milstein and Marshall Kreuter (Centers for Disease Control and Prevention), and Mark Dignan (University of Alabama Prevention Research Center).
|
References |
|---|
|
TopAbstractIntroductionMethodsResultsDiscussionReferences |
|---|
Buchan, I. E. and Hanka, R. (1997) Coherent exchange of healthcare knowledge in open systems. In Pappas, C., Maglaveras, N. and Scherrer, J.-R. (eds), Studies in Health Technology and Informatics 43: Medical Informatics Europe `97. IOS Press, Amsterdam, pp. 821–824.
Centers for Disease Control and Prevention (2000) Prevention Research Centers: Investing in the Nation's Health. Centers for Disease Control and Prevention, Atlanta, GA.
Cochrane, A. L. (1972) Effectiveness and Efficiency. Random Reflections on Health Services. Nuffield Provincial Hospitals Trust, London. (Reprinted in 1989 in association with the BMJ.)
Cochrane Collaboration (2001) The Cochrane Collaboration: Preparing, Maintaining and Promoting the Accessibility of Systematic Reviews of the Effect of Health Care Interventions [Brochure]. UK Cochrane Centre, Oxford.
Department of Health (1999) Saving Lives: Our Healthier Nation. The Stationery Office, London. Available: http://www.ohn.gov.uk/
Department of Health and Human Services (2000) Healthy People 2010, 2nd edn. With Understanding and Improving Health and Objectives for Improving Health. Government Printing Office, Washington, DC, 2 vols.
Gold, R. S. and Atkinson, N. L. (1999) Technological applications of PRECEDE–PROCEED. In Green L. W. and Kreuter, M. W. (eds), Health Promotion Planning: An Educational and Ecological Approach, 3rd edn. Mayfield, Mountain View, CA, pp. 470–501.
Green, L. W. (1983) Foreward. In Gold, R. S. (ed.), Microcomputer Applications in Health Education. Brown, Dubuque, IA, pp. xv–xvi.
Green L. W. and Kreuter, M. W. (1999) Health Promotion Planning: An Educational and Ecological Approach, 3rd edn. Mayfield, Mountain View, CA.
Hanka, R. (1997) Information overload and the need for `just-in-time' knowledge. In Proceedings of the 4th Asia–Pacific Medical Informatics Conference 97, Hong Kong.
HDA Evidence Base (2001) Putting Public Health Evidence into Practice. Health Development Agency, London. Available: http://www.hda-online.org.uk/evidence/eb2000/corehtml/intro.htm. Accessed: 19 June 2001.
Heathfield, H., Pitty, D. and Hanka, R. (1998) Evaluating information technology in health care: barriers and challenges. British Medical Journal, 316, 1959–1961.
Harlan, W. R. (1998) Prevention research at the National Institutes of Health. American Journal of Preventive Medicine, 14, 302–307.[Medline]
Holder, H., Flay, B, Howard, J., Boyd, G. and Voas, R. (1999) Phases of alcohol problem prevention research. Alcohol Clinical Experimental Research, 23, 183–194.[ISI][Medline]
Institute of Medicine (1997) Linking Research and Public Health Practice: A Review of CDC's Program of Centers for Research and Demonstration of Health Promotion and Disease Prevention. National Academy Press, Washington, DC.
Keeling, C. and S. Lambert (2000) Knowledge management in the NHS: positioning the healthcare librarian at the knowledge intersections. Health Library Review, 17, 136–143.[Medline]
McKillip, J. (1987) Need Analysis: Tools for the Human Services and Education. Sage, Newbury Park, CA.
O'Brien, C. and Cambouropoulos, P. (2000) Combating information overload: a six-month pilot evaluation of a knowledge management system in general practice. British Journal of General Practice, 50, 489–490.
Stoecker, R. and Bonacich, E. (1992) Why participatory research? The American Sociologist, 23, 5–14.
Wyatt, J. C. (2001) Management of explicit and tacit knowledge. Journal of the Royal Society of Medicine, 94, 6–9.[ISI][Medline]
Received on February 13, 2001; accepted on July 20, 2001
CiteULike 
Connotea 
Del.icio.us What's this?
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||