Skip to main content

Table 1 Evaluation design of the innovation platform and key findings

From: Methodological pluralism for better evaluations of complex interventions: lessons from evaluating an innovation platform in Australia

Objectives of the evaluation Methodology—evaluation approaches Rationale for the chosen methodology and methods Implementation, data collection and analysis Brief overview of evaluation findings
Objective 1: To refine the formation, functioning and outcomes of the innovation platform by supporting continuous reflection, rapid learning and adaptation Evaluation approach 1: Developmental evaluation [3] •To inform the formation, functioning and outcomes of the collaboration, we focused on the use of data to inform ongoing decision-making, reflection and adaptation—e.g. operations of the innovation platform, work programmes and future directions
•Developmental evaluation embraces innovation, complexity and systems thinking. Innovation platforms are complex systems and have continuous reflection and adaptation as design elements
•Administrative project records provided ongoing intelligence on the innovation platform development and context, e.g. minutes from the management committee and biannual stakeholder meetings; publications, posters and conference papers; attendance lists and evaluations of biannual meetings, masterclasses and research capacity-strengthening teleconferences; research project applications; and student projects
•Major contributions were the CRE-IQI Year 2 and Year 4 Reviews
•Interviews as part of Year 4 Review (n = 28)
•Further round of interviews to explore emergent themes from Year 4 Review (n = 36)
•Analysis and feedback, iterative and ongoing
•Developmental evaluation was well suited to innovation platforms, where there is a developmental purpose, innovation niche and complexity
•Adjustments made to the operation of the innovation platform based on the evaluative feedback included adding an Indigenous researcher to the innovation platform leadership team; targeting resources for policy and parliamentary submissions; implementing collaborative processes to identify and refine research priorities; and discussing principles of the innovation platform at the start of regular six-monthly meetings
•We used opportunistic and planned iterative cycles of reflection and analysis to understand how, and how well, the innovation platform was functioning and meeting its goals and how it could be adapted in rapid time to function more effectively
Objective 2: To identify the mechanisms and contextual factors that enable innovation platforms to have a positive impact in Indigenous PHC systems Evaluation approach 2: Principles-focused evaluation [19] •Emerged from feedback from innovation platform members, as part of the developmental evaluation, on the importance of the principles in guiding our work. From this, we agreed to a principles-focused evaluation to address objective 2
•To examine in depth how the principles we developed to underpin the governance of the collaboration were implemented and their expected outcomes
Data collection
•Purposive sampling to capture a diversity of views from interviews with innovation platform members (n = 35) by two researchers
•Reflective summary generated after each interview
•Document review of administrative project documents, results from developmental evaluation, publications, etc.
Data analysis
•Data inductively coded and categorized into strategies, outcomes and conditions
•Member-checking processes included presenting early findings at innovation platform meetings; comparing, contrasting and seeking consensus of findings between c0-authors; and triangulation with findings from document reviews
•The principles were viewed as an integrated whole, with overlap but mutually reinforcing, that would enable us to navigate complexity and conflict
•Implementation of the principles occurred through five strategies: honouring the principles; being dynamic and adaptable; sharing and dispersing leadership; collaborating purposefully; and adopting a culture of mutual learning
•Outcomes included increased Indigenous leadership and participation; the ability to attract principled and values-driven researchers and stakeholders; and the development of trusting and respectful relationships
•The conditions that facilitated the implementation of the principles were collaborating over time; increasing the number of Indigenous researchers; and taking an “innovation platform” approach
•Given that the focus of innovation platforms is on empowering local actors to solve problems collaboratively, and as lessons on their use accumulate, the findings from our study suggest that there is scope to be more explicit about the principles governing them and to embed, constantly monitor and reflect on their role within innovation platforms
Evaluation approach 3: Framework analysis [21] •To gain an understanding of the elements that enabled the innovation platform
•Emerged when it became evident that the evaluation working group needed to find an appropriate way to answer critical questions about the attributes of an innovation platform
Data collection
•Purposive sampling to capture a diversity of views from innovation platform members (n = 35) by two researchers
•Document review of publications and reports from the innovation platform
Data analysis
•Framework analysis using a taxonomy as framework. [41] Data were deductively coded to the four primary elements—innovation, communication, time and social system. Though primarily a deductive qualitative approach, we remained “nimble to emerging attributes”, and this application enabled us to identify emergent attributes not encompassed within the taxonomy
•Findings compared and contrasted, and a consensus process from authors undertaken through multiple reviews, triangulation of findings and discussions
•The long history of working together enabled trusting relationships, a collective identity and a foundation for new people to join
•Time was identified as a crucial element
•Innovation was stimulated by bringing people together to learn, share ideas and solve problems, with Indigenous participation and leadership at the core of the research agenda-setting
•The innovation platform outputs exceeded 92 peer-reviewed publications; 81 conference presentations; 27 research and technical reports; 26 newsletters, 16 masterclasses; 31 research capacity-strengthening webinars; 24 students (PhD, masters and undergraduate)
•The innovation platform had 18 research projects with investigators from 27 different organizations and $31,998,410 in collaborative research grants
•There was an ongoing need to focus on increasing the engagement of and leadership by Indigenous and health service stakeholders
Objective 3: To assess the development of, and change in, innovation platform collaborators over time Evaluation approach 4: Network analysis [20] •To understand why and how the collaboration has grown and changed over time
•Coauthorship approach emerged from the developmental evaluation and discussions with members who wished to explore the growth and change in membership, in particular, its Indigenous representation
Coauthorship network analysis
Data collection
•Peer-reviewed journal articles and books published by authors from the network
Data analysis
•Co-author networks across four phases of the network (2002–04; 2005–09; 2010–14; 2015–19) were constructed based on author affiliations
•Social network analysis methods
•Descriptive characteristics included organization types, Indigenous representation, gender, student authorship and thematic research trends
Social network analysis
Data collection
•Surveys of those defined as active members of the innovation platform at two time points (2017 and 2019)
Data analysis
•Social network analysis methods
Coauthorship network analysis
•Publications accelerated when the collaboration changed to an “innovation platform”, which coincided with a broader thematic focus and an increased number and diversity of participating organizations
•This expansion occurred largely due to the cumulative effect of building trust and relationships over time, including the development of a comprehensive data set on CQI in Indigenous PHC for use by all stakeholders
•Network analyses indicated a core/periphery structure of organizations connected to each other, rather than a network structured around a single central organization
•Increased productivity was associated with increased authorship diversity and a decentralized network, suggesting these may be important factors in enhancing research impact and advancing the knowledge and practice of CQI in PHC
•Despite improvements over time, further work is needed to address inequities in both female and Indigenous authorship
Social network analysis [24, 26]
•There was more sharing of knowledge and collaborating between those who had prior knowledge of each other. However, 48% also reported sharing and 37% collaborating with people of whom they had no prior knowledge. This shows both a broadening of relationships and a sharing of knowledge not only with existing partners but also new ones
•In addition, 36% of sharing occurred outside immediate collaborative partnerships, indicating good network support
•Further reporting is expected on how well the innovation platform worked as a collaborative network over time
Objective 4: To generate new knowledge on and approaches to evaluating innovation platforms The application of the multiple evaluation approaches listed above speak to this objective As listed above As presented in this manuscript As presented in this manuscript