From inclusion to independence – Training consumers to review research

Abstract

Health and medical research invariably impacts on the lives of everyday people. Organisations in the developed world are increasingly involving the public in health research projects, and research governance structures and processes. The form the involvement takes varies, as does the level of involvement, from individuals, to groups, to the wider community. Lay community members can be trained to independently review health and medical research, and wider societal involvement in funding decisions, can be effectively fostered. The theoretical foundation, design and development of a task based consumer-training program, including a number of enabling factors to support the success of such training are presented. This work is likely to be of value to those planning to train consumers in technical or complex areas.

Background

Purposeful public involvement in health and medical research has been an emerging agenda item in research governance processes in many economically developed nations for over a decade. Its importance arises, in part, out of an egalitarian desire to improve public access to and understanding of science [1], which was a key recommendation in a 1999 strategic review of Australian research and development [2]. Reported benefits of consumer involvement include raising the awareness of scientists to the realities faced by community members with particular health problems, a greater acceptance and uptake of research findings in the community, and more efficient use of research resources [3, 4].

Peer review of research funding applications involves an assessment of scientific merit. Each scientific reviewer is assumed to have the expertise to judge the feasibility and suitability of proposed research methods and analysis, and the capacity and track record of the research team. Whilst consumers are not involved in the assessment of the scientific value of research applications, they are included on peer review panels by an increasing number of organisations around the world to provide lay input concerning the public significance of the research [5–10]. Involving consumers in research review requires developing realistic expectations of what they can and cannot contribute to an existing review process. Clear expectations will, in turn, increase the likelihood of appropriate opportunities being identified for effective and meaningful consumer input and ensures that the right balance between expert and lay assessment of research is struck.

The Cancer Council NSW [11] (CCNSW) is a large publicly funded charity that supports a significant proportion of Australia's cancer research. Cancer Voices NSW [12] (CVN) is the independent, peak advocacy organisation for all people affected by cancer in New South Wales, Australia. Striking a balance between funding research that has significant scientific merit and research that responds to the priorities of the community that finances it, has been a long-standing goal of the CCNSW and CVN. An analysis of the CCNSW research funding structures and procedures identified an avenue for the independent assessment of research grants by consumers. A recently developed national framework for community participation in Australian health and medical research guided the broad strategy components [13]. To ensure impartiality in the assessment process, the CCNSW developed a consumer research appraisal tool that is underpinned by the findings of purpose designed qualitative research [14]. Each criterion within the appraisal tool reflects prevailing community values relevant to making health research funding decisions. These include the level of positive impact on human lives, whether the research can be applied in the real world, the extent to which it will be available to all who could benefit by it, how soon the research was likely to be available in clinical practice, and whether consumers were involved [14]. The availability of this tool provides the opportunity for consumers to independently and objectively judge research applications, subsequent to scientific review, for the CCNSW.

As the specialist nature of research review is significantly different from most other activities of consumer representatives, detailed, quality training is essential. Increasing public interest in health and medical research, along with policy directives for improving public participation in research, have created a global opportunity and market for consumer education programs and has led to a recent growth in the number and type of such training courses [15–18]. However, there have been no information or published papers detailing the training of consumers to independently judge cancer research funding proposals and further research to develop different training methods and education for consumers has been recommended [19].

The purpose of this paper is to report on our experience in developing a task-based training program to support an independent consumer review of research grant applications. This paper summarises the theoretical framework for the development of this training, the structure of the course, the lived experiences of the participants, and a range of important enabling factors to effectively support consumers in research review. While this work is likely to help those who wish to train consumers in an independent research review process, it may also be valuable to others who intend to train consumers in technical or complex areas where there are few, if any, existing resources to inform effective development and successful implementation.

The Training

Design

The training program was first conceptualised and implemented as a pilot initiative with an expectation that it would proceed to a more systemic ongoing program. Whilst we could find no established instructional design principles or standardised development approaches specifically for training consumers to review research, the complex yet practical nature of research review lent itself to a task based instructional approach. According to the 'situated cognition' movement, which takes from Vygotskys' activity theory [20–22] and more recently from developmental work in educational psychology [23], learning is maximised if the context for learning resembles the actual situation in which the instruction will be used. Task based learning is a model that organises learning around an activity [24–26]. Central to the process is that the task, for which training is developed, is applied in the real life context. The key principles of the model are that instruction should reflect the complexity of the thinking and work that learners are expected to be able to do in the real world circumstance, and that learners must engage in active practice to establish genuine connections to the real situation [27].

To ensure the training involved adequate information and relevant practice of the required task, we first identified all the training elements in a storyboard format. This provided a reference point by which the design process could follow. Three aspects of learning (awareness, knowledge and skill) and three formats of delivery (oral information, interaction and written resources) were subsequently identified. The development of a matrix, combining the learning and delivery areas, provided a simple means of representing and organising the content and linkages of the training elements (Table 1).

Table 1 Consumer Research Review Training Matrix

In designing the training it was important to make sure that participants were not taken out of the process. As the role of reviewer is a social activity, it was felt that participants would benefit greatly from activities that led to group interaction and active participation in the training process itself. Critical thinking skills that challenged consumers to continuously seek new ways of understanding the information before them and to recognise the complexities within research funding submissions were considered especially important. The design also focused on developing the necessary interpersonal requirements such as working as part of a team, the ability to negotiate and understand group dynamics and conciliate to assist the resolution of complex issues. The ultimate requirements of the training concern the development of appropriate cognitive and practical skills so each consumer review panel member becomes competent to conduct an assessment of research using predetermined consumer review criteria.

Training applicants were purposefully selected after a general recruitment process via newspaper advertisements and consumer health organisation newsletters. Inclusion in the training was based on a number of aspects, including whether they had an interest in cancer research, a willingness to become familiar with research terminology, disciplines and concepts, experience in consumer centred cancer groups and organisations, and the ability to keep up to date with current consumer issues via consumer networks and associations. Screening and interviewing potential training participants minimised the effect of real-world factors such as differing levels of experience among potential members and facilitated their commitment to, and expectations of the training. It showed potential participants that the CCNSW took both the training and their time seriously. Screening also gave the opportunity to match personal characteristics and skills with the needs of the review task such as the ability to consider the views of others. Those subsequently trained were a mix of cancer survivors and members of the general public affected by cancer through the experience of family members and/or friends.

Cooperation and collaboration among experts was also crucial in the development of the training program. An advisory committee, which included informed consumer representatives, was established to advise the development of necessary training materials and methods.

The design of the training program included the development of detailed written guidelines (Additional file 1) to support the review process and explain the review criteria in detail. Examples of research summaries in lay language were made available in the guidelines to further support the process.

When dealing with relatively complex information it is important that learners are able to access detailed information in a number of forms. A comprehensive manual was provided to each participant with information covered in the training and additional resources that were felt could assist the learning process, such as a step-by-step written guide of the consumer review process (Additional file 2).

Training programs are in many cases not developed and managed as a product that is able to be re-used and adapted by others. A broad approach to facilitate the replicability of the training program was adopted. Table 2 provides the outline of the final training program.

Table 2 Outline of the Cancer Council NSW Consumer Review of Research Training Program

Delivery

To cover all the necessary information and skills required by research reviewers, the training mixed didactic presentations on each cancer research discipline, with group and practical sessions in actual research review using previously funded grant applications and suitable lay summaries. Complex information was considered best taught as discrete elements so a lay overview of each research discipline was designed and presented separately by prominent experts in each of the fields. Written lay summaries of each research discipline were developed by CCNSW staff and edited and approved by each relevant expert (Additional files 3 and 4).

The positive value of involving specialist researchers to describe their disciplines without compelling trainees toward any particular type of research over another was a recurring finding throughout all elements of the evaluation of the training. Other important contributors to increasing participant knowledge and skill was learning from each other in group discussions, and from staff who routinely answered questions, clarified meaning and provided any additional information that was required.

It was recognised that an atmosphere of respect, trust, useful dialogue and effective practice was necessary if we were to successfully train consumers to review research applications in two days. In the practice review session participants were supported to apply the review guidelines and assist us to work out the finer details of the review process. Use of actual research proposals allowed us to efficiently obtain a snapshot of review skills and participants' baseline confidence and communication techniques. This fostered a productive discussion during the feedback sessions, which focused on how to more effectively undertake the review process.

Participant input was reviewed regularly throughout the two-day program. The ongoing critical reflection of the training led to several modifications such as allowing more time for several of the sessions, improvements to the type and sequencing of information available in the training manuals and the need for greater interaction with speakers. Participants were given the opportunity to describe what they had gained from the practical experience:

"Increased my confidence tremendously."

"I was surprised that I could 'get it'."

"Doing the review made the information so much more meaningful."

Application

The victory lap cannot realistically be taken until the training actually produces the results that were intended. The ability to move from training to practice is dependent on the relevance of the training to the particular task. The reliable measure used to determine the effectiveness of the research review training was an assessment of the process and outcomes of consumer review. The assessment of the first round of consumer review of research found that all participants, while having a range of experiences and abilities, were able to strictly apply the consumer review criteria to research proposals which, as the training intended, significantly minimised subjective input. It also found, that the application of the criteria limited unnecessary dialogue which made the review process efficient and professional. Most importantly, the participants ranked higher in priority order those proposals that best met the requirements of the consumer review criteria.

In making final funding decisions, an oversight committee allocates equal weighting to applications prioritised secondarily through the consumer review process and those prioritised by scientific merit review alone. Significant discordance between consumer and scientific review is resolved in open discussion by a purpose-convened group that includes both trained consumers and researchers. The success of the CCNSW consumer review of research, which is now an enduring research governance process, has led to its keen adoption by other organisations [28].

It is possible to train consumers in research review and achieve the desired effect. A summary of important enabling factors to effectively support consumers in research review is provided in Table 3. However, since this reform is a challenging concept and a major departure from the old strategies, proper orientation of all stakeholders on the new approach is critical to its effective application. Researchers submitting proposals for funding by the CCNSW and those involved in the final selection process were made aware of the changes and implications of the consumer review process through written protocols and guidelines, and formal discussions including updates and refinements in the process. It is essential that appropriate organisational information be constructed in parallel to training so the newly developed reform can become truly embedded.

Table 3 Enabling factors to support the independent consumer review of research

Conclusion

Consumer training in research review underpins an important change in CCNSW research governance [29]. The new research review process encompassing consumer review, subsequent to scientific merit review, is designed to offer the highest promise of fulfilling community needs. It provides an approach to independently assess the social value of research through the use of empirically identified criteria considered important by community members. The first year of the consumer review process led to the funding of three out of twelve research funding applications that would not have been previously considered, but which better served the needs of the community.

We endeavoured to create a training program that would meet the needs of the consumer reviewers and would be sustainable in the long-term through a dedicated, yearly training program that is offered to existing trainees and new recruits.

The CCNSW and Cancer Voices NSW have achieved their goal to ensure that a trained group of consumers is available to participate in grant review. Consumers can reliably and effectively undertake a challenging and complex requirement if they are given training that is aligned to their ability and specific to the task. The involvement of expert scientists is critical to the continued value of this program and to make sure participants benefit from the wealth of experience that these individuals have to offer. As national and international policy directives continue to build on the momentum that presently drives public participation in research, careful attention must be paid to both the organisation and quality of essential training.