Since 2010, we have developed and implemented an organisation-wide impact framework to consistently plan, monitor and evaluate the impact of our research.
This page is available as a printable brochure: How CSIRO ensures it delivers impact.
CSIRO is one of the largest and most diverse research agencies in the world. Since its establishment in 1926, CSIRO has been charged with conducting research to address the scientific problems facing industry and the nation. Importantly, CSIRO must also “... encourage or facilitate the application or utilization of the results of such research”.1 In other words, CSIRO is a mission directed organisation and, as a result, has an abiding interest in better understanding the impact it effects.
While CSIRO has always planned its research and evaluated its impact, it is only recently that CSIRO has taken a consistent, organisation-wide approach. Since 2010, CSIRO has developed and implemented an organisation-wide impact framework to consistently plan, monitor and evaluate the impact of its research. This approach allows for a more comprehensive and complete view of the impact being delivered and supports the management of impact of a large portfolio of research.
The main drivers behind CSIRO’s increased interest in measuring research impact are to improve its:
- advocacy an increased capacity to articulate future and delivered impact
- accountability the ability to provide defensible, robust evidence of impact
- analysis the opportunity to better understand and maximise research impact, and
- allocation to enable more informed decision making.
Impact and evidence based planning, monitoring and evaluation is an intrinsic element of the CSIRO’s Planning and Performance Framework and Management Cycle. Impact management relies on a two-way engagement with all actors along the impact pathway to ensure the research is relevant, realistic and that risks are identified and mitigated.
CSIRO’s Impact Framework
CSIRO’s approach to impact management assumes that in order to understand the value of research, it must be possible to track the process by which research translates into benefits in the real world. Planning, monitoring and evaluating impact is based on a theory of change model called program logic. CSIRO’s Impact Framework is based on the hypothesis that the process of creating impact begins with deploying inputs, to conduct research activities and produce outputs, which themselves are translated through short to medium term outcomes into long term impact.
The input to impact model (see Figure 1) involves a systematic grouping of information types. For the sake of simplicity the model depicts a simple linear process to aid discourse, but it is understood that science is serendipitous and agile in execution, with multiple feedback loops and engagement at all stages. Therefore the framework is operationalised in the same manner.
Figure 1: CSIRO’s Impact Management Framework
Diagram shows the stages of engagement:
- Inputs, Activities, Outputs, Outcomes and Impact.
- Each of these stages interacts with an overarching feedback loop.
- Planned work occurs during the Inputs, Activities and Outputs stages.
- Intended results occur during the Outputs, Outcomes and Impact stages.
- What we control is part of the Inputs, Activities and outputs stages.
- What we influence directly is part of the Outcomes stage.
- What we influence indirectly is part of the Impact stage.
- Planning Impact Activity occurs through all stages.
- Monitoring Impact Activity occurs through the Inputs, Activities, Outputs and into the Outcomes stages.
- Evaluation of Impact Activity occurs through all stages.
Impact planning and management hierarchy
CSIRO’s Flagships are the core vehicle for research impact delivery. They integrate all activities, from capability development, through to science delivery, and client and commercial engagement. Flagship Goals specify the nature of science to be undertaken within the context of broader national challenges and research priorities. A Flagship refines how it will achieve its goal through the articulation of the broad areas of impact it is planning to deliver.
Within CSIRO the articulation of a Flagship’s future intended impact, characterised using the program logic and described in Impact Statements, is undertaken at the Program level. Impact Statements (which summarise several impact pathways developed at a lower level closer to the research) are aggregated up to describe achievement of Program and Flagship goals. Programs typically have between 3-5 impact statements. Programs are responsible for developing and managing the impact pathways for the portfolio of projects they manage (see Figure 2). This approach provides a clear link between projects and Impact Statements, and between Impacts Statements, Impact Areas and Flagship goals.
Impact Statements are stored in a central database capable of producing simple reports and graphics. The information in the Impact Module allows for the creation of CSIRO’s Impact Map (see Figure 3), a communications tool which, for the first time in CSIRO’s history, describes on one page the major impacts being progressed and delivered by the organisation.
Impact Statements also rely on a performance monitoring concept known as ‘time to goal’ which aggregates project milestones from the subordinate research projects into a single metric – ‘time to output’, ‘time to outcome’ and ‘time to impact’. These ‘time to goal’ measures are important Program and Flagship KPIs in the new CSIRO performance reporting framework under development.
To complement its planning hierarchy, CSIRO has developed an Impact Evaluation Guide that provides a series of impact evaluation principles, along with a standardised cost-benefit approach, to be used for both ex-ante (forecasting) and ex-post (after the fact) impact evaluations. Use of the Impact Evaluation Guide ensures the comparability of impact evaluation findings and reports, regardless of where in CSIRO the evaluation is being conducted, the field of science or the area of impact.
Figure 2: CSIRO’s Impact planning and management hierarchy
Diagram shows a flow chart from Flagship Goal (aggregate primary outcome domains, scale and timeframe) to Impact Areas to Programs to Impact statements to Impact Pathway Planning to Projects.
Figure 3: CSIRO’s Impact Map
CSIRO's National Research Flagships are taking on the biggest challenges and opportunities in manufacturing, minerals, energy, digital services, water, agriculture, food and nutrition, oceans and atmosphere, and biosecurity. This map represents the key impacts CSIRO has delivered and intends to deliver. Some impacts will not happen, new discoveries will make others possible, and Australia will ask CSIRO to respond to new challenges. CSIRO does not deliver impact alone - we work with more than 2000 Industry, Government and Research partners each year to create a better future for Australia and humanity.
Impact categories: Defence; Plant Production and Primary Products; Animal Production and Primary Products; Mineral Resources; Energy; Manufacturing; Construction; Transport; Information and Communication Services; Commercial Services and Tourism; Economic Framework; Health; Law, Politics and Community Services; Environment; Expanding Knowledge.
Supporting impact management practice, CSIRO is also designing and delivering an enterprise wide and enduring cultural program to support long term improvement in our outcome focus, customer service and impact delivery. Impact management is to be understood, valued, prioritised and rewarded as “the way CSIRO does business”.
CSIRO’s impact culture program includes considerations of the ‘people’ aspects of how science is managed for impact, including:
- Strong leadership and accountability requirements relevant to role.
- Capability building (including training and learning & development courses).
- A focus on evidence of outcomes in performance conversations at all levels – not just inputs or compliance.
- Linking organisational efforts to corporate outcomes through setting specific, measurable, achievable and time-bound outcomes and ensures teams and individuals are accountable for, and own, them.
- Accountability for actions and behaviours supporting impact is reflected explicitly in staff APAs (each according to function).
- Staff induction, review, recruitment, promotions & rewards and other HR processes are being reviewed and amended as necessary to reinforce/build desired impact behaviours.
- Individual training in CSIRO Impact Framework is available to all Flagship, Science and Research Directors.
The benefits of CSIRO’s approach to impact
- Systematic, one-CSIRO approach to impact planning, monitoring and evaluation
- Capability building to enhance and accelerate the delivery of impact from research
- Better informed resource allocation
- Streamlined internal reporting and communication
Flagship Leadership teams
- Better informed decision making
- Improved transparency
- Improved understanding of the portfolio
- Systematic, strategic management framework
- Improved line of sight from the research bench to the intended impact
- Increased awareness of CSIRO’s strategic direction for delivering and demonstrating impact
- Opportunities to develop skills and competencies around impact
- Improved understanding of CSIRO’s impact
- Greater understanding of the results of investment into research and development
- Improved transparency
Part II, Section 9 (1)(b) of the Science and Industry Research Act 1949 (SIR Act), http://www.comlaw.gov.au/Details/C2011C00647