Ensuring we deliver impact

Since 2010, we have developed and implemented an organisation-wide impact framework to consistently plan, monitor and evaluate the impact of our research.

This page is available as a printable brochure: How CSIRO ensures it delivers impact. [pdf · 1mb]

CSIRO is one of the largest and most diverse research agencies in the world. Since its establishment in 1926, CSIRO has been charged with conducting research to address the scientific problems facing industry and the nation. Importantly, CSIRO must also “... encourage or facilitate the application or utilization of the results of such research”.1  In other words, CSIRO is a mission directed organisation and, as a result, has an abiding interest in better understanding the impact it effects.

While CSIRO has always planned its research and evaluated its impact, it is only recently that CSIRO has taken a consistent, organisation-wide approach. Since 2010, CSIRO has developed and implemented an organisation-wide impact framework to consistently plan, monitor and evaluate the impact of its research. This approach allows for a more comprehensive and complete view of the impact being delivered and supports the management of impact of a large portfolio of research.

The main drivers behind CSIRO’s increased interest in measuring research impact are to improve its:

  • advocacy an increased capacity to articulate future and delivered impact
  • accountability the ability to provide defensible, robust evidence of impact
  • analysis the opportunity to better understand and maximise research impact, and
  • allocation to enable more informed decision making.

Impact and evidence based planning, monitoring and evaluation is an intrinsic element of the CSIRO’s Planning and Performance Framework and Management Cycle. Impact management relies on a two-way engagement with all actors along the impact pathway to ensure the research is relevant, realistic and that risks are identified and mitigated.

CSIRO’s Impact Framework

CSIRO’s approach to impact management assumes that in order to understand the value of research, it must be possible to track the process by which research translates into benefits in the real world. Planning, monitoring and evaluating impact is based on a theory of change model called program logic. CSIRO’s Impact Framework is based on the hypothesis that the process of creating impact begins with deploying inputs, to conduct research activities and produce outputs, which themselves are translated through short to medium term outcomes into long term impact.

The input to impact model (see Figure 1) involves a systematic grouping of information types. For the sake of simplicity the model depicts a simple linear process to aid discourse, but it is understood that science is serendipitous and agile in execution, with multiple feedback loops and engagement at all stages. Therefore the framework is operationalised in the same manner.

Figure 1: CSIRO’s Impact Management Framework

CSIRO's Impact Framework

Show text description

Impact planning and management hierarchy

CSIRO’s Flagships are the core vehicle for research impact delivery. They integrate all activities, from capability development, through to science delivery, and client and commercial engagement. Flagship Goals specify the nature of science to be undertaken within the context of broader national challenges and research priorities. A Flagship refines how it will achieve its goal through the articulation of the broad areas of impact it is planning to deliver.

Within CSIRO the articulation of a Flagship’s future intended impact, characterised using the program logic and described in Impact Statements, is undertaken at the Program level. Impact Statements (which summarise several impact pathways developed at a lower level closer to the research) are aggregated up to describe achievement of Program and Flagship goals. Programs typically have between 3-5 impact statements. Programs are responsible for developing and managing the impact pathways for the portfolio of projects they manage (see Figure 2). This approach provides a clear link between projects and Impact Statements, and between Impacts Statements, Impact Areas and Flagship goals.

Impact Statements are stored in a central database capable of producing simple reports and graphics. The information in the Impact Module allows for the creation of CSIRO’s Impact Map (see Figure 3), a communications tool which, for the first time in CSIRO’s history, describes on one page the major impacts being progressed and delivered by the organisation.

Impact Statements also rely on a performance monitoring concept known as ‘time to goal’ which aggregates project milestones from the subordinate research projects into a single metric – ‘time to output’, ‘time to outcome’ and ‘time to impact’. These ‘time to goal’ measures are important Program and Flagship KPIs in the new CSIRO performance reporting framework under development.

Impact evaluation

To complement its planning hierarchy, CSIRO has developed an Impact Evaluation Guide that provides a series of impact evaluation principles, along with a standardised cost-benefit approach, to be used for both ex-ante (forecasting) and ex-post (after the fact) impact evaluations. Use of the Impact Evaluation Guide ensures the comparability of impact evaluation findings and reports, regardless of where in CSIRO the evaluation is being conducted, the field of science or the area of impact.

Figure 2: CSIRO’s Impact planning and management hierarchy

CSIRO’S impact planning and management hierarchy

Show text description

Figure 3: CSIRO’s Impact Map

CSIRO's Impact Map

Show text description

Impact culture

Supporting impact management practice, CSIRO is also designing and delivering an enterprise wide and enduring cultural program to support long term improvement in our outcome focus, customer service and impact delivery. Impact management is to be understood, valued, prioritised and rewarded as “the way CSIRO does business”.

CSIRO’s impact culture program includes considerations of the ‘people’ aspects of how science is managed for impact, including:

  • Strong leadership and accountability requirements relevant to role.
  • Capability building (including training and learning & development courses).
  • A focus on evidence of outcomes in performance conversations at all levels – not just inputs or compliance.
  • Linking organisational efforts to corporate outcomes through setting specific, measurable, achievable and time-bound outcomes and ensures teams and individuals are accountable for, and own, them.
  • Accountability for actions and behaviours supporting impact is reflected explicitly in staff APAs (each according to function).
  • Staff induction, review, recruitment, promotions & rewards and other HR processes are being reviewed and amended as necessary to reinforce/build desired impact behaviours.
  • Individual training in CSIRO Impact Framework is available to all Flagship, Science and Research Directors.

The benefits of CSIRO’s approach to impact

CSIRO

  • Systematic, one-CSIRO approach to impact planning, monitoring and evaluation
  • Capability building to enhance and accelerate the delivery of impact from research
  • Better informed resource allocation
  • Streamlined internal reporting and communication

Flagship Leadership teams

  • Better informed decision making
  • Improved transparency
  • Improved understanding of the portfolio
  • Systematic, strategic management framework

Researchers

  • Improved line of sight from the research bench to the intended impact
  • Increased awareness of CSIRO’s strategic direction for delivering and demonstrating impact
  • Opportunities to develop skills and competencies around impact

External stakeholders

  • Improved understanding of CSIRO’s impact
  • Greater understanding of the results of investment into research and development 
  • Improved transparency
  1. Part II, Section 9 (1)(b) of the Science and Industry Research Act 1949 (SIR Act), http://www.comlaw.gov.au/Details/C2011C00647

Enquiries

Have an enquiry about this page?

Contact us