Our impact evaluation framework
Our impact evaluation activities provide credible evidence of the effects of our research and innovation activities on the economy, environment and society.
Our research activities and their impacts are diverse in nature and occur across many sectors of the economy. While some impacts are primarily economic and capable of being evaluated in monetary terms, many others – especially those relating to environmental or social effects – may have to be evaluated qualitatively. Ultimately though, each impact must be assessed within the context of a common framework if a comprehensive understanding of our impact and return on investment is to be developed.
Our Impact Evaluation Guide articulates such a common framework, and its consistent and rigorous use across our organisation ensures that outcomes from each evaluation are comparable – across business units and across time.
The guide describes the minimum requirements for all our impact evaluations, regardless of the purpose of the evaluation or the ‘unit of evaluation’ (which could be an individual project, subject area, business unit or the whole enterprise). It guides researchers, staff and engaged external support to address key relevant questions in a logically consistent manner and to select the appropriate resources and methods in the evaluation of our research.
We have just released an updated version of our Guide - View the full Impact Evaluation Guide PDF (4 MB).
The major changes made were:
- Identification of the role of evaluation in both accountability and ensuring that R&D actions are aligned to enhancing Australia's competitiveness and economic and social well-being.
- A shift in focus towards the use an appropriate mix of methodologies (not just CBA) to ensure that the methods used to evaluate impact are suited for the particular impacts under examination. Evaluation methods beyond CBA identified include statistical approaches (e.g. regression discontinuity, difference in differences); cost-effectiveness analysis; scientometric approaches; social network analyses; and qualitative analyses. In particular, the guide helps manage potential methodological ambiguity by directly identifying that topics must be addressed qualitatively if they cannot be addressed quantitatively.
- Reinforcing of the point that impacts should be measured relative to a baseline and/or a counterfactual under which the project/program was not supported and prevailing trends continued.
- An increased focus on enhancing comparability of case studies by being explicit about how cash flows are analysed, among other aspects. While it can be hard to standardise benefits categories or the monetisation of benefits, how cash flows are analysed can and should absolutely be standardised.
- An enhanced list of source materials.
- A call for CSIRO Business Units to be mindful of the need for collecting and curating (at least) the most basic information requirements that could support future impact evaluations. This includes any market or technical publications describing the rationale for action and investment; any analysis results describing or detailing the advantages, disadvantages, costs, or benefits of the solution; and basic data about uptake and usage by different stakeholders (e.g., who, when, where, why, how many, etc.).
- Identification of the multi-faceted nature of the potential impacts from R&D, and the need to consider all potential attributable impacts when conducting an evaluation, while still being careful to avoid ‘double-counting’.
- The need for clarity around ‘private’ & ‘public’ benefits, and transfer payments when conducting an evaluation.
- The requirement for the consideration of cultural sensitivities when conducting evaluation (e.g. exercising caution when considering the monetisation of impacts relating to indigenous culture and heritage, and the need to discuss the appropriateness of this with relevant stakeholders).
- Enhanced guidance on estimating costs; externalities; spill-over and economic flow-on effects; inflation adjustment; and discounting.
The International School on Research Impact Assessment
The International School on Research Impact Assessment (ISRIA) was founded to foster the science of research impact assessment across all fields of sciences, and to support sustainability of research systems. The Fourth Edition of the International School on Research Impact Assessment was hosted by CSIRO from September 19 – 23, 2016 in Melbourne, Australia.
The School aims to enhance participants' skills in research impact assessment, and the fourth edition brought together international experts in the field of Research Impact Assessment for a five-day, hands-on event.
In addition to the curriculum delivered through the School, CSIRO also hosted a Gala Dinner with the theme ‘Building an innovation led Australia’, which was held on Monday, September 19 at the Melbourne Town Hall. The Gala Dinner featured a live, webcast panel discussion, bringing together a broad and influential set of stakeholders to facilitate constructive discussion on national innovation issues and challenges.