alluvium_logo_new EVALUATION REPORT: Assessment of the Atlas of Living Australia’s Impact and Value October 2016 Document history Revision: Revision no. 2 Author/s David Barratt Jian Wang Jim Binney Checked David Winfield Approved David Barratt Distribution: Revision no. 2 Issue date 11 October 2016 Issued to CSIRO Description: Final report Citation: Alluvium (2016) Assessment of the Atlas of Living Australia’s impact and value. Report produced for the CSIRO, 50pp. Acknowledgements: We would like to acknowledge and thank the following people for their input in this review: John La Salle Stephanie von Gavel Rebecca Pirzl Michael Hope David Winfield Miles Nicholls Peter Brenton Ref: Final Report V2 Executive Summary Introduction To date there has been limited attempt to measure the impact of public research data infrastructure. In August/September 2016, an Impact Evaluation of the Atlas of Living Australia (ALA) was undertaken. This project represents the first attempt to obtain an independent assessment of the ALA’s economic, social and environmental impact and value. This evaluation was intended to be preliminary, reporting some initial findings and making recommendations for appropriate next steps in impact evaluation. Evaluation design The impact evaluation utilised a number of data collection methods, including an analysis of web metrics related to the use of the ALA, conducting an online ALA impact survey involving key stakeholders and user groups, in-depth follow-up interviews to the survey, and the documentation of pertinent case studies. Initially five areas were selected for evaluation: 1. Uptake and usage. To what extent has there been uptake and usage of the data delivered through ALA and the tools, services and infrastructure of the ALA? 2. Cultural change. To what extent has ALA influenced cultural change in the behaviours relating to open sharing of data and integration and interoperability between platforms and services and data. 3. New products and services. Has ALA amplified and improved the creation, delivery and competitive advantage of new products and services and enterprises by others? 4. Efficiency gains. Has the ALA resulted in productivity and efficiency gains in information management and delivery of services? 5. Applications and derivatives. To what extent has ALA helped in the application of knowledge and activities in key sectors including through the creation of “information derivatives”? Key Findings The evaluation indicated that ALA has led to a range of delivered and potential impacts, including increased open sharing of data and standards, production of reports, papers and publications, significant efficiency gains for biodiversity data management and on-ground intervention and actions relating to biodiversity. Of the impacts identified, economic impacts are estimated in monetary terms, as discussed in the section below. Given the constraints to data availability for environmental and social impacts, these impacts are noted, but not assessed. A cost–benefit analysis was performed and found that for every $1 invested in the ALA there is at least $3.5 economic value created. The ALA provides very good value for money, even when conservative assumptions are used. -The annual value delivered by ALA is at least $26.9 million * -The Return on Investment ratio is 3.5:1* It is also worth noting that in this analysis, we take a conservative approach that only includes benefits to Commonwealth expenditure on biodiversity and national parks and did not include any State spend (which in total was 4 times the amount of Commonwealth spend). In addition, economic benefits only include direct user benefits (such as time and resources savings), and did not include the wider/indirect benefits, arising from the additional use of the data facilitated by the ALA (i.e. the value of the data re-use). One could therefore imagine that an overall benefit–cost ratio for the ALA could be significantly greater than 3.5. Recommendations 1. Continue support for the ALA impact study. The present study could profitably be followed by further studies to provide a more comprehensive picture of the ALA impact and value. 2. Track changes over time. Value and perceptions of value change over time. A series of metrics should be determined that could be captured on an ongoing basis to monitor ALA performance in key areas. 3. Conduct more granular analysis. A set of further case studies should be determined for examination to add to the impact evaluation. There is significant scope to examine the value of specific collections, or the economic value of service to specific groups. 4. Further develop the methods. There would be benefits from further developing, refining and exploring applications of the methods used in this study (i.e. impact survey, monetisation techniques). 5. Study the wide value and impact of NCRIS. The framework developed in this study could be applied across the NCRIS and other publicly funded research infrastructure seeking to measure where open data infrastructure benefits or makes broader contributions to society beyond those contributions to academic knowledge. Contents 11 Introduction 1.1 Background to the Atlas of Living Australia 1 1.2 Purpose and audience of this report 2 1.3 Clarification of Value and Impact 2 2 Evaluation Design and Methodology 4 2.1 Theory of Change 4 2.2 Evaluation Questions 4 2.3 Data Collection 5 2.4 Outcomes and Impact Indicators 6 3 Results informing the Evaluation of ALA Impact 10 3.1 Uptake, Usage and Adoption 10 3.2 Influence on Cultural Change 20 3.3 New Products and Services 29 3.4 Productivity and Efficiency 34 3.5 Applications and Derivatives 36 3.6 Counterfactual 39 4 Economic Quantification of ALA Impact 41 4.1 Costs 41 4.2 Benefits 42 4.3 Results – The Threshold Test 43 4.4 Sensitivity analysis 45 5 Conclusions 46 5.1 Uptake, Usage and Adoption 46 5.2 Influence on Cultural Change 46 5.3 New Products and Services 47 5.4 Productivity and Efficiency 47 5.5 Applications and Derivatives 47 5.6 Economic Quantification of Impact 48 6 Limitations and Future Directions 49 7 References 51 Figures Figure 2-1. Impact pathway for the ALA project 4 Figure 3-1. Number of records stored in the ALA from January 2012 to July 2016 10 Figure 3-2. Number of records downloaded from the ALA between July 2013 and July 2016 11 Figure 3-3. Google Scholar results for “Atlas of Living Australia” from 2008 to 2015 (number of references to the ALA online) 11 Figure 3-4. Number of people using the ALA and number of sessions of use from 2009 to 2015 12 Figure 3-5. Geographical distribution of respondents to the online ALA Impact Survey 12 Figure 3-6. Distribution of respondents to the online ALA Impact Survey with respect to the sector they work in 13 Figure 3-7. Spatial scale that respondents indicated was the main focus of their use of data, information and tools from the ALA 13 Figure 3-8. Responses to the question “Where do you currently go to discover and access information/data for your purposes/project?” from the 2016 ALA Impact Survey 16 Figure 3-9. Responses to the question “Which kinds of data and information from ALA do you work with?” from the 2016 ALA Impact Survey 17 Figure 3-10. Responses to the question “Which kinds of tools, services and infrastructure from ALA do you work with?” from the 2016 ALA Impact Survey 17 Figure 3-11. Level of agreement with statements about the ALA from the 2015 ALA User Survey 18 Figure 4-1. Efficiency gains required to justify ALA 44 Tables Table 2-1. Outcome indicators 6 Table 2-2. Impact Indicators 7 Table 3-1. Descriptions of uses of ALA data, information and tools provided by ALA Impact Survey participants 14 Table 3-2. Survey responses to statements related to question of the extent to which the ALA has influenced or changed behaviours, practices or culture in relation to open sharing of data, standards, use of open services and partnerships 20 Table 3-3. Survey responses to statements related to the question of whether the ALA has amplified and improved the creation, delivery and competitive advantage of new products and services and enterprises 30 Table 3-4. Estimated % productivity and efficiency gains 35 Table 3-5. Survey responses to statements related to the question of the extent to which the ALA has helped in the application of knowledge and activities in key sectors including through the creation of “information derivatives” 36 Table 3-6. Where did you previously (prior to ALA) discover and access information/data for your purposes/project? 39 Table 3-7. Where did you previously (prior to ALA) discover and access information/data for your purposes/project? 40 Table 4-1. Analysis of survey responses – economic questions 42 Table 4-2. ALA funding arrangements ($ million) 41 Table 4-3. Results of sensitivity analysis 45 Abbreviations AABR Australian Association of Bush Regenerators AEKOS Advanced Ecological Knowledge and Observation System ABS Australian Bureau of Statistics ALA Atlas of Living Australia Alluvium Alluvium Consulting Australia Pty Ltd API Application Programming Interface AVH Australia's Virtual Herbarium BCR Benefit-Cost Ratio BIOCLIM Bioclimatic variables BOM Bureau of Meteorology CBA Cost-Benefit Analysis CHIP Catchment Health Indicator Program CRIS Collaborative Research Infrastructure Scheme DA Development Application EIF The Education Investment Fund ERIN The Environmental Resources Information Network FCIG Faunal Collections Informatics Group GBIF Global Biodiversity Information Facility GIS Geographic Information System HISCOM Herbarium Informatics Systems Committee ISO The International Organization for Standardization KEQ Key Evaluation Question MERIT Monitoring Evaluation Reporting and Improvement Tool NCCARF The National Climate Change Adaptation Research Facility NCRIS The National Collaborative Research Infrastructure Strategy NECTAR National eResearch Collaboration Tools and Resources NGO Non-governmental organisation NRM Natural resource management NPV Net Present Value OECD Organisation for Economic Co-operation and Development OGC Open Geospatial Consortium OTU Operational Taxonomic Units OZCAM Online Zoological Collections of Australian Museums ROI Return on Investment STEM Science, technology, engineering and mathematics TERN Terrestrial Ecosystem Research Network 1 Introduction 1.1 Background to the Atlas of Living Australia Effective biodiversity research and management rely on comprehensive information about the species or ecosystems of interest. Without this information, it is very difficult to obtain reliable results or make sound decisions. A major barrier to Australia's biodiversity research and management efforts has been the fragmentation and inaccessibility of biodiversity related data. Data and information on Australian species has been, and still is, generated and housed in museums, herbaria, collections, universities, research organisations, and government departments and agencies. Obtaining an integrated suite of records and data sets from these groups takes considerable time and effort, and often results in incomplete information. To overcome these issues, Australia's biodiversity information needs to be aggregated, connected and made easily discoverable and available by the users and contributors of this data. The Atlas of Living Australia (ALA), is an e-infrastructure that is funded by the Australian Government via its National Collaborative Research Infrastructure Strategy (NCRIS). It is a collaborative partnership of organisations that have stewardship of biological data and expertise in biodiversity informatics, including museums, biological collections, community groups, research organisations, government (state and federal), and natural resource managers. CSIRO acts as the lead agent in supporting delivery to the NCRIS program. The ALA was established in 2006, and the ALA portal went live in 2010. It delivers a centralised web-based infrastructure to capture, aggregate, manage, discover and analyse all classes of biodiversity data and associated information, through a suite of tools and spatial layers for use by research, industry, government and the community. Its vision is to lead the digital transformation in sharing biodiversity knowledge thereby supporting and enabling high quality research and innovation outcomes to address national and global challenges. It supports a host of activities by its stakeholders from research, biodiversity discovery and documentation, environmental monitoring and reporting, conservation planning, biosecurity activities, education and citizen science, together with enterprises and organisations leveraging off the open infrastructure to create and enhance their own services and products. Since 2010, the ALA team has worked to aggregate Australia's biodiversity information and to make it available online via the ALA website. The ALA is founded on the principle of open access – collect data once, make it freely accessible and discoverable, use it many times. This is particularly important in the context of public data produced, collected, held and funded by government as well as in a global biodiversity informatics framework. The ALA currently holds over 60 million records of more than 111,000 different species from across Australia, and over 470 spatial layers, with its adoption and utilisation illustrated by over 8 billion records having been downloaded for use to date. This vast repository of information makes the ALA the most comprehensive and accessible data set on Australia's biodiversity ever produced. With additional records and information being added to the ALA on a regular basis, the ALA is constantly growing. The ALA also features a wide range of powerful, open-source mapping and analysis tools, which allow users to explore and analyse information in new ways. Over 470 spatial layers let users explore the relationships between species distribution and factors such as rainfall, temperature, soil moisture, political or regional boundaries, fire and vegetation. The objectives of the ALA are: 1. To act as the enabling open infrastructure platform to mobilise, share and integrate biodiversity information and tools across diverse communities or stakeholders, including identifying gaps in that information. 2. To help address and inform national (and global) challenges and priorities around sustainable and resilient ecosystems in a changing and anthropogenic environment. 1.2 Purpose and audience of this report To assist with decision making, efforts to untangle the value of public research data and infrastructure require a robust process and framework for impact evaluation. To date there has been limited attempt to measure the impact of public research data and infrastructure. This project represents the first attempt to obtain an independent assessment of the ALA’s economic, social and environmental impact and value relative to the investment it has received from the Australian Government over the same period. This project attempts to provide a framework that may be generalised for understanding diverse elements of research data and infrastructure impacts. By drawing on results from a recent survey analysis of the ALA, the suitability of the proposed framework is assessed. Further methodological improvements for assessing ALA’s socio-economic impacts are also recommended. The scope of the evaluation includes: 1. Undertaking an assessment of the key impact areas of the ALA such as influence on cultural change, new products and services, productivity and efficiency gains and applications and derivatives. The assessment has been undertaken using the CSIRO’s Impact Evaluation Guide as a basis for evaluation. 2. Providing an initial and contemporary estimate of the benefit-cost ratio for investment in ALA and contextualising this in the organisation’s overall value. This task included: . Proposing and utilising a suitable methodology to estimate the benefit-cost ratio for investment in ALA by drawing on results from interviews and case studies, and . Providing a summary of the role and value of ALA, which details the qualitative value of ALA beyond that estimated above. This information is provided for accountability, communication and continual improvement purposes. Audiences for this report are primarily NCRIS, together with ALA partners and stakeholders, CSIRO as lead partner, and the ALA project team. To facilitate this, a targeted evaluation design and methodology was developed and employed, which is outlined in Section 2 of this report. 1.3 Clarification of Value and Impact According to the OECD1 and Donker et al. (2016), economic assets are a store of value from which economic benefits may be derived by holding or using them. This is as true for information as it is for assets like transport infrastructure, agricultural lands and hospitals. However, there are some key differentiators (Moody and Walsh, 1999): . Information is infinitely shareable, reusable and repurposable. Information can be shared by multiple business areas and used to generate the same value for each party as if they had exclusive use. . The value of information increases with use. Unlike most assets, the more information is used, the more valuable it becomes. . Information is perishable. Like most assets, the value of information tends to depreciate over time. The rate of depreciation depends on the type of information. . The value of information increases with accuracy. In general, the more accurate information is, the more useful and therefore valuable it is. . The value of information increases when combined with other information. 1 "Economic assets -OECD.Stat." 2005. 29 Jun. 2015 . More is not necessarily better. There are human limits to processing information beyond which decision-making performance decreases. . Information is not depletable. In fact, as information is used, derivatives are created and added to the information asset base. The economic value of information often comes from saving resources or costs elsewhere in the system. The concept of deprival value is of great help in the psychology of identifying areas of value. Decision makers often find it hard to answer questions like ‘what is this information worth to your organisation?’ However, they find it much easier to answer the question ‘what would the organisation lose and what would the consequences be if it were deprived of the information?’ Thinking in terms of deprival value helps enormously to focus on the economic value of a resource and how it would have to be replaced. Measures of the value of information may include some or all of the following (Higson and Waltho, 2010): . Data security – we care for what we value and will make an effort to reduce data loss. . Data quality and timeliness – as the quality and/or timeliness of data increases, so does its value, because user trust and usage increases and better decisions can be made. . Data users – the number of data users across a wider ecosystem is also a proxy of information value creating a ‘value multiplier’ effect. . Data uses – the number of applications to which the information is used is also a sign of information value. . User satisfaction – is a key indicator of information value, especially as it assists them in solving their business problems. 2 Evaluation Design and Methodology 2.1 Theory of Change The ALA is enabling research possibilities, improving access to information about biodiversity, and enhancing the way environmental management occurs in Australia. By aggregating biodiversity information and making it available online, the ALA is assisting scientists, planners, managers and others to create a more detailed picture of Australia's biodiversity, its threats and likely future state. It allows users to focus on discovering answers to their questions, rather than searching for and managing data. From an economic perspective, this superior information accessibility and reduced cost of analysis should pay significant efficiency dividends in the form of enhanced cost effectiveness of biodiversity management. This Impact Evaluation report is based on the theory of change. This is a description of how inputs are used to deliver activities, which in turn result in outcomes and impacts (changes) for each stakeholder. The theory of change tells the story of how stakeholders interact with the ALA and their perception and belief of changes as a result. This Impact Evaluation report uses the evaluation framework outlined in the CSIRO Impact Evaluation Guide. The results of applying that framework to ALA are summarised in Figure 2-1. Economic impact . FUNDING: . Open infrastructure for . Uptake of tools, services • New product/ services NCRIS/CRIS/EIF funding; development. biodiversity data. and data by users using ALA. CSIRO co-invest, (including researchers, . Data ingestion, . Open source code. • Improved productivity and resources and support; industry, government aggregation, efficiency in information . Integrated open data. other partners/ and the community. management and delivery integration, curation . Tools, services, virtual collaborators co-invest of services. and management. . Adoption of tools and labs. • Economic growth through and in-kind. services by other . Tools for data discovery, improved productivity. . Shared community . DATA: Contributions of platforms. visualisation and spaces. data from partners and . Changes in practice, Environmental impact analysis. . Dynamic links / web collaborators. • Address national and products and services. . Stakeholder and global challenges. services / APIs. . Existing infrastructure, community engagement . Networks and open source code and Social impact and liaison. data standards. collaborations • Influence on cultural . Communication and (nationally and changes on the open . Existing networks and outreach. sharing of data, use of internationally). collaborators (e.g. . Global interactions. information standards, . Best-practice relationships with peak integration between . Partnership and management of bodies). platforms and services collaboration building. biodiversity . Existing and improved community . Business development information supply & connectivity. culture/attitudes (e.g. for external. value chains. Australia already revenue/opportunity . Communication prepared to embrace generation. material. data sharing). Figure 2-1. Impact pathway for the ALA project 2.2 Evaluation Questions This report was guided by a series of specific evaluation questions, which were developed to facilitate the initial building of a knowledge base around the impact and value of the ALA and how ALA might better understand, shape and enable them into the future. The following key evaluation questions (KEQs) were derived for each of the impact areas to guide data collection, which help assess the impact and value of the ALA: 1. Uptake, Usage and Adoption KEQ: To what extent has there been uptake and usage of a) the data delivered through ALA and b) the tools, services and infrastructure of the ALA by its four key stakeholder groups (research, industry, government, community) and other platforms and countries? 2. Influence on Cultural Change KEQ: To what extent has ALA influenced cultural change in the following behaviours? a. Open sharing of data b. Improved use of information standards in data collection and information management c. Use of open services/web services/APIs – linkages and integration and interoperability between platforms and services and data d. Improved community connectivity and partnerships 3. New Products and Services KEQ: Has ALA amplified and improved the creation, delivery and competitive advantage of new products and services and enterprises by others? 4. Productivity and Efficiency KEQ: Has the ALA resulted in productivity and efficiency gains in information management and delivery of services? 5. Applications and Derivatives KEQ: To what extent has ALA helped in the application of knowledge and activities in key sectors including through the creation of “information derivatives”? Programs of works will often yield multiple diverse impacts, and there are often challenges in combining them into a singular impact figure. It is realistic to accept that, even for programs with many clear monetary costs and benefits, there will be other non-monetary costs and benefits that can be included in an impact evaluation. These non-monetary impacts include, but are not limited to, an effect on, change or benefit or cost to the activity, attitude, awareness, behaviour, capacity, opportunity, performance, policy, practice, process or understanding of an audience, beneficiary, community, constituency, organisation or individuals in any geographic location whether locally, regionally, nationally or internationally. Impact also includes the reduction, avoidance or prevention of harm, risk, cost or other negative effects. For example, KEQ 2 and KEQ 5 have been designed to capture these non-monetary impacts. The overarching principle in the impact evaluation framework used for this report is that all relevant impacts are to be measured using a Cost-Benefit framework. There are a number of economic impacts that could occur as a result of the ALA, including new products or services (e.g. the identification of biodiversity offsets), or increased government efficiency (e.g. the selection and management of the protected areas estate). Given the scope of information available through the survey/interviews and resourcing constraints for this introductory evaluation project, we focussed on the potential economic benefits in the form of government efficiencies in managing biodiversity information. Through economic modelling, we attempted to identify whether the benefits attributable to ALA products and services exceed their costs of provision. If they do, there is a business case for the ALA to continue to provide those products and services. 2.3 Data Collection To elicit the appropriate data and information for the identified key impact areas, the evaluation utilised a number of data collection methods, including i) an analysis of web metrics related to the use of the ALA, ii) conducting an online ALA impact survey involving a selection of key stakeholders and user groups, iii) in-depth follow-up interviews to the survey, and iv) the documentation of pertinent case studies. Online Surveys An online survey (2016 Impact Survey) was developed and applied as a key data collection tool for the evaluation. Selected representatives from research, industry, community and government agencies (total number = 178) were invited by email to complete this survey. This survey covered the KEQs and sought to gain a detailed insight into the impact of the ALA. Data from this survey (total respondents = 45) was analysed and cross-referenced, as appropriate. The details of the specific survey questions used to collect data are provided in Appendix A. The evaluation also used responses from the ALA’s online user survey conducted in 2015 (total respondents = 833). This survey asked where ALA users come from, how the ALA contributes to their work and life, and which features they use, like most and would improve. Individual Interviews As part of the online survey, respondents were asked if they would be willing to participate in a follow-up interview. There were 10 people subsequently interviewed to explore in more depth their responses to questions in the online survey and to investigate the likely precision of estimates provided on the economic benefit of the ALA to them and their organisation. In the survey, participants were asked to state their degree of agreement with a number of statements about the ALA and its use and value, using a 5-point scale from ‘strongly agree’ to ‘strongly disagree’. In follow-up interviews, where participants either strongly agreed or strongly disagreed with one of the statements provided, these responses were explored further to understand why they responded that way and whether examples could be provided or described in support of the response. Web metrics Several web metrics were used as a means to gain insights into the uptake and usage of the data, tools, services and infrastructure delivered by the ALA. These web metrics included the number of occurrence records in the ALA, the number of occurrence records downloaded from the ALA, mentions of “Atlas of Living Australia” in Google Scholar, and ALA usage statistics from Google Analytics. These metrics are used because they are easy to generate, and (although they may not necessarily point to impact) they are indicative of broad trends of usage and outcomes. For example, results in Google Scholar do not necessarily point to only publications (or the impact of those publications), but they will show a response curve indicative of increased growth and usage. Case studies Case studies were used to provide insights into the role of ALA in opening up research possibilities, improving knowledge of our biodiversity, and changing the way environmental management occurs in Australia. They also provide more depth for the understanding of problems that ALA has addressed and its significance to different stakeholders and users. 2.4 Outcomes and Impact Indicators The key outcome indicators developed for the evaluation are summarised in Table 2-1. Table 2-1. Outcome indicators Data 1. The uptake and use of ALA . Total number of records downloaded data. . Google Scholar results for "Atlas of Living Australia" Tools, services 2. The uptake and use of ALA and tools, services and infrastructure infrastructure. . Response to 2016 Impact Survey question – Where do you currently go to discover and access information/data for your purposes/project? . Google Scholar results for "Atlas of Living Australia" . ALA usage based on Google Analytics . Response to 2016 Impact Survey question – Which kinds of tools, services and infrastructure from ALA do you work with? ALA has led a range of delivered and potential outcomes and impacts. Using a triple bottom line impact classification approach, the Table 2-2 below summarises the nature of the existing and potential outcomes and potential impacts. Table 2-2. Impact Indicators INFLUENCE ON CULTURAL CHANGE 1. The extent to which the ALA has changed the understanding, use and acceptance of the creative commons licencing framework. 2. The extent to which the embedding of standards in ALA systems has improved the fitness-for-purpose, consistency and accessibility of all online biological and ecological data. 3. The extent to which there has been an improvement in the amount of “trusted” or quality data online as a consequence of the establishment of the ALA. 4. The extent to which the ALA has influenced others to ensure their own services/platforms are available as open infrastructure/architecture (e.g. software downloadable on GitHub; APIs all available for services/products of others). 5. The extent to which the linking and integration of different data types through the ALA (e.g. through spatial portal) has added value to information and services. 6. The extent to which the ALA has enabled “communities” or groups to connect, collaborate and self-organise in groups of interest to improve their own activities, avoid re-inventing the wheel, . 2015 User Survey responses . 2016 Impact Survey responses . Phone interviews . Case studies reduce duplication, provide critical mass, and attract new customers/clients/partners. PRODUCTIVITY & EFFICIENCY 1. The extent to which use of the ALA has improved data management efficiency in relation to the time and resources spent in biodiversity data access. 2. The extent to which use of the ALA has improved data management efficiency in relation to the time and resources spent in biodiversity data transformation. 3. The extent to which use of the ALA has improved data management efficiency in relation to the time and resources spent in biodiversity data delivery. 4. The extent to which use of the ALA has improved expenditure on meeting Australia’s biodiversity commitments. 5. The extent to which use of the ALA has improved biodiversity investments. . 2016 Impact Survey responses . Phone interviews . Case studies . Literature review APPLICATIONS & DERIVATIVES regulatory compliance by industry and government responses with respect to environmental legislation. government decision-making regarding investment of public funds and meeting its international biodiversity treaty obligations. application of research to key national and global challenges. community to undertake on-ground interventions with respect to biodiversity (e.g. restoration, community participation, surveys). 3 Results Informing the Evaluation of ALA Impact The following results are a combination of quantitative (survey) and qualitative (interviews/narratives) results that informed the evaluation of ALA impact, against the five key evaluation question areas of: . Uptake, usage and adoption (Section 3.1) . Influence on cultural change (Section 3.2) . New products and services (Section 3.3) . Productivity and efficiency (Section 3.4) . Applications and derivatives (Section 3.5) 3.1 Uptake, Usage and Adoption Web metrics The total number of records held in the ALA has risen from around 20 million in January 2012 to over 60 million in July 2016 (Figure 3-1). Total number of records in ALA 10000000 0 Jan-12 Jul-12 Jan-13 Jul-13 Jan-14 Jul-14 Jan-15 Jul-15 Jan-16 Jul-16 record count 6 month increase Figure 3-1. Number of records stored in the ALA from January 2012 to July 2016 The total number of records downloaded from the ALA as at July 2016 totalled almost 7.3 billion (as at October 2016 now 8.5 billion) (Figure 3-2). Total number of records downloaded (millions) 8000 Jul-13 Jan-14 Jul-14 Jan-15 Jul-15 Jan-16 Jul-16 Total per 6 months Figure 3-2. Number of records downloaded from the ALA between July 2013 and July 2016 Google Scholar results for the “Atlas of Living Australia” show that references to the ALA have been increasing steadily each year and now number 1050 (Figure 3-3). Google Scholar results for "Atlas of Living Australia" 2008 2009 2010 2011 2012 2013 2014 2015 per year total Figure 3-3. Google Scholar results for “Atlas of Living Australia” from 2008 to 2015 (number of references to the ALA online) The number of people using the ALA and the number of sessions of use has also been increasing since 2009, with the number of sessions of use now totalling almost 1.6 million (Figure 3-4). ALA usage -based on Google Analytics 1800000 2009 2010 2011 2012 2013 2014 2015 users sessions Figure 3-4. Number of people using the ALA and number of sessions of use from 2009 to 2015 2016 ALA Impact Survey The 2016 ALA Impact Survey was distributed to 178 users, data providers and stakeholders. There were 45 responses from across all States and Territories (Figure 3-5). Of respondents, 47% were from research institutions (or research backgrounds), 24% were from government, 24% were from industry and 9% were from community groups (Figure 3-6). The spatial scale that respondents indicated was the main focus of their use of data, information and tools from the ALA is shown in Figure 3-7. National scale was more common than local or international, though local scale was the primary focus of use of ALA data and tools for more than 40% of respondents. As part of the ALA Impact Survey 2016, survey participants were asked to give a short description of an instance where they used data, information and tools from ALA. The responses (paraphrased) are shown in Table 3-1. Table 3-1. Descriptions of uses of ALA data, information and tools provided by ALA Impact Survey participants I used occurrence records to determine a taxon, its distribution and relation to environmental records 1. for biosecurity research purposes. I explored occurrence data for the crustacean group Copepoda. I wanted to find out information about 2. the distribution of copepods and the families, genera and species that have been recorded from Australia. The Costa Rican Biodiversity Information Facility is using ALA tools to update its data portal 4. (occurrences) functionality available at www.crbio.cr. I regularly use ALA GIS mapping tool to plot OTU distributions against various environmental 5. surrogates. I use occurrence data to help me learn more about species distributions and also to see where there 6. are few (or no) records for species in their range. Particularly for invertebrates, I use location records and photos to help with identification. 7. I searched for data on the geographic distribution of a species, and the source of the data. 8. Use dashboard data for organisation reporting. Downloaded and explored occurrence records in relation to spatial layers and downloaded information 9. statistics on data usage in the collectory. 10. I manage a program within a Federal government department I downloaded and explored (and excluded) occurrence records and used these data to write a report for BushBlitz. I used associated environmental data to explore the habitats of species when conducting taxonomic research. I viewed occurrence records to determine geographic distributions of taxa for taxonomic revision, and 11. to assess which collection records from contributing herbaria had incorrect identifications (required cross-referencing with other data sources due to issues with duplicate detection in ALA). I have been involved in testing development sites (ALA and AVH), in particular testing names indexing, faceting, and profiles (in the development of an eFlora). 12. I download ALA data for reports. 13. I investigated the occurrence of a specific species through the online tools. I downloaded lizard occurrence records, checked and corrected the data, combined it with our group's own records. Then I used it to model distributions of species, combined these with phylogenetic data on 14. the relationships between species, and published a scientific paper on conservation in the Northern Territory. From what I learnt from ALA regarding species collection records I have written reports for 15. environmental agencies, local authorities and other agencies. 16. I downloaded and explored occurrence records of species. I wrote a report. 17. I only look at species distribution maps. Last night discussed with colleague distribution of Olive Python so looked at ALA records and found no 18. Olive Python in our location then what alternative Morelia species occur here. I produced a map of a species, using filtered data, in the spatial portal and exported this for use in 19. presentations such as conferences and for reports prepared by government agencies. We routinely use the ALA API to search and add species occurrences to our web application to enable 20. distribution modelling and trait modelling. 21. Downloaded occurrence data when developing conservation plans. 22. I use the ALA tools largely to understand how they work in order to contribute data better. 23. Spatial Portal – plotted summary statistics for a latitudinal gradient for Australia. I maintain and use data related to water quality, macroinvertebrates and riparian conditions. I 24. regularly download data for small-scale projects, and annual report writing regarding catchment health. I downloaded occurrence records and phylogenetic trees for teaching purposes for a third year 25. biodiversity class first semester 2016. I use spatial mapping of a species to understand its distribution, and to compare between species: and might include an image of a map in a report. I don't do much analysis, because: [1] I am interested 26. primarily in wetlands, rivers and floodplains and the contextual info from ALA is very terrestrial-oriented; [2] the number of layers is an endless list with no 'a priori' guidance. 27. I download occurrence records of a plant genus, and analysed patterns of distribution. I utilise occurrence data to create spatial distributions and species lists as part of the fauna impact 28. assessment process for clients. 29. Used the spatial portal to select datasets, stratified by time, geographical and/or environmental data. I explored and downloaded the data of occurrence records for a particular search area. This was used 30. for the desktop assessment within an ecological assessment report to an energy company. I downloaded records for particular group of species and undertook continental analysis of endemism 31. and richness. 32. Pulling out threatened species records around a DA site. 33. Assessment of impact of proposal. I downloaded occurrence records of vertebrates and weeds across a large area in which I'm working. 34. This desktop work contributed to a fauna survey program and a weed management plan. I used occurrence records and environmental data to help identify potential areas for seed collections 35. for restoration plantings under climate change. I wrote a paper based on the work. We use an ALA portal to capture biodiversity data by a group of community members, which assists in 36. the management of natural resources in our region. Responses to the survey question, “Where do you currently go to discover and access information or data for your purposes/project?” from the 2016 ALA Impact Survey are shown in Figure 3-8. Almost three quarters of respondents said they primarily use the ALA to discover and access data or information for their needs. Responses to the question “Which kinds of data and information from ALA do you work with?” from the 2016 ALA Impact Survey are shown in Figure 3-9. Occurrence records were the most commonly accessed type of data, being used by 89% of respondents to the survey. Species descriptions or profiles (including taxonomy/names), environmental data, spatial layers and organisational/collection/data set information were also used by over 50% of survey respondents. Figure 3-9. Responses to the question “Which kinds of data and information from ALA do you work with?” from the 2016 ALA Impact Survey Responses to the question “Which kinds of tools, services and infrastructure from ALA do you work with?” from the 2016 ALA Impact Survey are shown in Figure 3-10. The spatial and mapping tools were the most commonly used services, but analysis tools, including visualisation, data discovery tools, data capture, collection and upload tools and species profiles were also popular. ALA User Survey 2015 The ALA undertook a general user survey in 2015, and the results of that survey indicated that more than half of the ALA users who completed the 2015 User Survey reported that they had gained ‘information for improved personal understanding of Australia's biodiversity’ by visiting the website. Just over 20% of users reported that they had gained ‘information for improved decision making’ and ‘scientific research results’. Fourteen per cent of users reported that their use of the ALA had contributed to a publication. One user listed Building Nature’s Safety Net 2014 as an example. Users were asked to state their degree of agreement with the 11 statements shown in Figure 3-11. A 5-point scale from ‘strongly agree’ to ‘strongly disagree’ was offered in the survey. However, for ease of comprehension, Figure 3-11 compares the proportion of users who agreed (either strongly agreed or agreed on the original scale) with those who responded neutrally or disagreed (strongly disagree or disagreed on the original scale). Figure 3-11 indicates that, overall, ALA users believe the website is valuable and understandable. Most users did not feel that the ALA required major improvement. People most commonly visit the ALA to look up information about a species or find out what lives in an area. ALA users are more likely to download data than upload data and the ALA is used internationally. A vast majority of users agree that the ALA is a valuable research (91%) and education (89%) tool. Overwhelmingly, ALA users find the website valuable and understandable. A substantial proportion of website users are ALA industry stakeholders. Industry stakeholders are more likely to visit the ALA to download data and use the mapping and analysis tools, compared with other users. They are less likely to visit the website to identify species and find out what lives in an area. The ALA is also engaging with the education sector, with the website being used by students as a teaching resource, resulting in educational outcomes. Vignette#1 – Using the ALA to help develop biodiverse plantings suitable for changing climatic conditions Acacia dealbata Link. Image by Richardson, R.G. & F.J. A key challenge for revegetation practitioners is to select plant material that will be suitable for both current and future climatic conditions. There have been an increasing number of revegetation projects established in Australia over the last 20 years. For example, the Australian Government is investing large sums of money through a variety of programs to assist communities, NGOs and government organisations to revegetate, rehabilitate and restore landscapes to store carbon, enhance biodiversity and build environmental resilience, including support for planning and implementation of strategies under possible climate change scenarios. Two papers, published in Ecological Management and Restoration in 2012, describe methods relevant to selecting plant material for changing climatic conditions. The first paper describes species distribution models and growth simulation models that could assist the selection process. The second paper describes how the ALA platform can assist revegetation project leaders to select species and provenances likely to be suitable for changing environmental conditions at particular sites and suggests improvements to the platform to allow even more reliable selections to be made. Source: Dr Trevor Booth, CSIRO. 3.2 Influence on Cultural Change ALA Impact Survey 2016 The 2016 ALA Impact Survey revealed a strong perception by respondents of the value of ALA influencing behaviours, practices or culture in relation to the open sharing of data and standards, and the use of open services and partnerships. The results of the survey related to influence on cultural change are shown in Table 3-2. Some highlights are described below: 84 per cent of respondents agreed or strongly agreed that the linking and integration of different data types through the ALA (e.g. through spatial portal) had added value to the information. . 66 per cent of respondents agreed or strongly agreed that the embedding of standards in ALA systems had helped improve the fitness for purpose, consistency and accessibility of online data. . 64 per cent of respondents agreed or strongly agreed that the ALA had enabled communities or groups to connect, collaborate and self-organise in groups of interest to improve their own activities, avoid re-inventing the wheel, reduce duplication, provide critical mass, and attract new customers or clients. . 60 per cent of respondents agreed or strongly agreed that their use of the ALA had influenced their capacity for advocacy or promotion of the ALA to internal and external people and organisations. . 58 per cent of respondents agreed or strongly agreed that there is an improvement in the amount of “trusted” or quality data online as a consequence of the establishment of the ALA. Table 3-2. Survey responses to statements related to question about the extent to which the ALA has influenced or changed behaviours, practices or culture in relation to open sharing of data, standards, use of open services and partnerships Neither Don't Strongly disagree Strongly Statement Disagree Agree know Disagree nor Agree / N/A agree My use of the ALA has changed my 0% 5% 42% 24% 21% 8% understanding and acceptance of open (n= 0) (n= 2) (n= 16) (n= 9) (n=8) (n=3)access using creative commons. My use of the ALA has influenced my 0% 8% 42% 21% 8% 21% decision to provide data free of change – (n= 0) (n= 3) (n=16) (n= 8) (n=3) (n= 8)previously fees for download and access. The embedding of standards in ALA 0% 3% 21% 42% 24% 11% systems has helped improve the fitness (n= 0) (n= 1) (n= 8) (n=16) (n= 9) (n= 4) for purpose, consistency and accessibility. There is an improvement in the amount 0% 8% 29% 29% 29% 5% of “trusted” or quality data online as a consequence of the establishment of the (n= 0) (n= 3) (n= 11) (n= 11) (n= 11) (n= 2) ALA. The linking and integration of different data types through the ALA (e.g. through 0% 3% 11% 37% 47% 3% spatial portal) has added value to information. (n= 0) (n= 1) (n= 4) (n= 14) (n=18) (n= 1) The ALA has influenced me to ensure my services/platforms are available as open infrastructure. 0% (n= 0) 3% (n= 1) 45% (n=17) 26% (n= 10) 8% (n= 3) 18% (n=7) The ALA has enabled communities or groups to connect, collaborate and self­organise in groups of interest to improve 0% 5% 29% 32% 32% 3% their own activities, avoid re-inventing (n= 0) (n= 2) (n= 11) (n= 12) (n= 12) (n= 1) the wheel, reduce duplication, provide critical mass, and attract new customers / clients. My use of the ALA has influenced my “advocacy/promotion” to internal and 0% 8% 24% 26% 34% 8% external people & organisations about (n= 0) (n= 3) (n= 9) (n= 10) (n= 13) (n= 3) any of the issues above (e.g. open access, standards, quality). Should we include the “neither agree nor disagree” option? Across the survey results, introducing a middle option attracted 11% and 45% of respondents. It does not appear related to observed respondent characteristics (industry, education). Many literature (e.g. Liezt 2010, Sturgis et al 2010) shows it makes sense to include a neutral category. It is because a neutral middle point lowers random error variance. An interesting caveat to this is highlighted by Sturgis et al (2010): “Neither agree nor disagree” can either be a ‘hidden don’t know’ (i.e., the respondent has no opinion) or it can mean a neutral opinion (i.e. the respondent is somewhere between agreeing and disagreeing). Sturgis et al 2010 indicate that there is no solution beyond a follow-up question for every “neither agree nor disagree”, but for robustness it may be worth re-coding neutral responses as “don’t know” and seeing how much it affects results. Comments made by respondents with regard to the question of the extent to which the ALA has influenced or changed behaviours, practices or culture in relation to open sharing of data, standards, use of open services and partnerships included: “The ALA has massively improved the efficiency with which I and collaborators can do our research”. -Dr Dan Rosauer, Australian National University Interviews Interviewees provided the following more detailed information on, and examples of, the influence of the ALA on cultural change. Understanding and acceptance of open access using creative commons Interviewees indicated that being involved with the development and maintenance of the ALA had provided great exposure to the systems and processes that have been implemented by the ALA. They commented that using the ALA helped them better understand the importance of open access in getting data used more effectively in research and public policy forums, and the importance of metadata in controlling access to, and use of, sensitive data. Linking and integration of different data types through the ALA Interviewees commented that the ALA had significantly reduced the time and effort required to access and combine data from different institutions around the country; they previously had to write to the separate institutions and would receive data on CDs and in a variety of different formats. The ALA has meant that sometimes months of work are reduced to just hours, and that this enormous improvement in productivity is being repeated for researchers and students all over Australia. Mention was also made of the ability that now exists through the ALA of linking specimens to samples, and multi-tiered linking – between specimens and collections – that is only provided through the ALA. Elsewhere in the world, they are now using ALA infrastructure to build this type of functionality; for example, the Atlas of Living Scotland. An interviewee provided an example of the use of the ALA to identify and predict the distribution of tree species, including where species are at the limit of their adaptability. This enables researchers to predict whether proposed plantations will survive new climate regimes under global warming scenarios. Some examples include the Carbon Sequestration Schemes in NSW which require tree survival for 100 years. In another example, a group monitoring hundreds of sites across a catchment, including the collection of vegetation, abundance, diversity and water quality data, uses the ALA to upload and manage these different but related data types in one place, using the same upload and download processes. Without the ALA this would be very difficult and, prior to establishment of the ALA, the process involved in creating reports manually involved significant double handling, errors and different formats, with no simple way to connect the different types of data. Enabling communities or groups to connect, collaborate and self-organise in groups of interest to improve their own activities, avoid re-inventing the wheel, reduce duplication, provide critical mass, and attract new customers or clients Interviewees commented that community groups use the ALA to manage the data they collect via community BioBlitzes, monitoring programs (e.g.Waterwatch) and other initiatives and that the process is very effective and a massive improvement in capacity relative to what occurred before the existence of the ALA. The information gathered and uploaded becomes much more visible to others undertaking similar activities. For example, the community group Australian Association of Bush Regenerators (AABR) is using the ALA to identify climate adaptation data for ecological restoration projects (Booth et al. 2012). In another example, a council in Western Australia is using the ALA to map species in the local area (Source: anonymous, Department of Environment and Resources). The ALA is seen to be very valuable in supporting and enabling citizen science. The ALA supports the connection of different groups which enhances the productivity and value of the investigations and reduces the impost previously applied on Government environmental officers in assisting community groups with data and data-based services. Influence on advocacy and promotion to internal and external people and organisations about open access, and standards around data and data quality The ALA infrastructure applies international data standards as much as possible and wherever appropriate. These include Darwin Core (DwC), Darwin Core Archive (DwC-A), Dublin Core, the International Organization for Standardization (ISO), Open Geospatial Consortium (OGC), and others. The ALA is also an active participant in the further development, maintenance and implementation of relevant international standards. Interviewees noted that establishment of the ALA has meant that all of the data they collect is uploaded to the ALA and therefore is required to pass all the necessary standards in order to be publicly shared. Requiring researchers and students to upload data to the ALA has meant promoting and advocating ALA data sharing and standards, principles and policies. The ALA has facilitated much greater visibility of biodiversity data and therefore has been a key agent in driving much greater knowledge and use of data standards and the promotion of requirements and standards associated with data collection, management and sharing. Interviewees indicated they had advocated the use ALA data standards for university collections and the placement of collections on the ALA. However, interviewees also stated that there was a need to improve standard processes to indicate data quality to contributors and users, and to consult further with the collections community in that regard; in particular the FCIG (Faunal Collections Informatics Group) and HISCOM (Herbarium Informatics Systems Committee). Vignette#2 – ALA supporting major improvements in data capture, investment, reporting and processes in natural resource management MERIT home page https://merit.ala.org.au In 2013 the ALA partnered with the Commonwealth Dept. of Environment & Energy (DoEE) to develop the Monitoring Evaluation Reporting and Improvement Tool (MERIT) to support the implementation and management of natural resource management (NRM) projects which are funded by the Australian Government. MERIT is designed to collect and store planning, monitoring and reporting data associated with natural resource management grants projects. The system allows individual projects to record and document their projects in both standardised structured and unstructured data formats and aggregates data across multiple projects to tell a whole-of-program story about the impact of natural resource management investments. Aggregated information can be accessed by individuals, natural resource managers and DoEE staff for a wide range of planning, management, policy and administrative purposes. This project has been outstandingly successful in: . Standardising the data collection and reporting processes; . Improving the manner in which projects are planned, monitored, implemented and reported on; . Identifying gaps and inefficiencies in DoEE processes and providing accurate and timely empirical data which highlights specific areas where improvements can be made. MERIT enables grant recipients to not only report on outputs from projects, but also to report and demonstrate outcomes, and to communicate their project activities and achievements more effectively to their stakeholders, partners and participants. Interviewees in this evaluation described how the MERIT system enables continuous access to around 2,800 projects and allows DoEE staff to more effectively monitor and assist grant recipients to achieve successful project outcomes than was previously possible under the previous management and reporting structure. Data from MERIT has also contributed to new research and peer-reviewed papers by the Department and CSIRO, which would not have been possible beforehand. Data collected and aggregated through MERIT not only allows the Department to run the funding programs that it is responsible for more effectively, but it also delivers a very rich and important set of information to the Commonwealth Government around program performance, policy achievements and progress. For example, MERIT provides accurate data on progress towards many of the actions in the threatened species programs under the Environment Protection and Biodiversity Conservation Act 1999, particularly those funded by the Australian Government. Improving the fit-for-purpose, consistency and accessibility of data The implementation of the ALA infrastructure improved the fit-for-purpose, consistency and accessibility of data in the Queensland Museum – all data was improved to ‘Darwin Core’ standard. The Darwin Core is body of standards. It includes a glossary of terms (in other contexts these might be called properties, elements, fields, columns, attributes, or concepts) intended to facilitate the sharing of information about biological diversity by providing reference definitions, examples, and commentaries. There were previously 25 separate databases in the Queensland Museum. Driven by the ALA infrastructure, the museum invested $1m in consolidating the 25 databases into one system and applying consistent standards. The museum now delivers data directly to and via the ALA and the cost of delivering data to subscribers is much lower. As a result many other museums have followed suit. For the Queensland Museum, ALA information standards mean that case­by-case investigation of data standards by each institution is no longer required; this is done now as a group and facilitated. The information standards ALA applies help to prevent the misuse of analyses and make it easier for data providers to make their data consistent so that users can be confident of their assumptions and can understand what data is fit-for-purpose and what is not. Improvement in the amount of “trusted” or quality data online Interviewees believed the ALA had resulted in an improvement in the quality of data online and the amount of trusted data, but recognised that even in the ALA there is still a significant amount of low quality data online. However, fundamentally it was recognised that it is more important that the data is accessible and that the gaps in data quality can now be identified.Previously these gaps could not be identified in any systematic way. Another benefit of open data access was the improved ability for error checking and reporting of errors back to the data custodians. Interviewees indicated that prior to the ALA, researchers had to go to the original data custodians to collate data for their research (especially species distribution analysis) and many organisations or individuals that held such data were not always keen to share it. The ALA has provided an excellent example of the idiom “build it and they will come”.Now many data collection organisations want to have their data in the ALA, and to track usage statistics in order to demonstrate how, and how often, their data is being used. Users also indicated that with the sheer amount of data now in the ALA, there is greater confidence in the quality of the data because outliers are more evident. In addition,the ALA identifies suspicious outliers through its quality checks. The ALA provides a means through its data management process, for researchers to check fitness for purpose of the data they are using. Vignette#3 – Data exploration using the Spatial Portal Tool The spatial portal tool in the ALA is particularly well regarded. A number of interviewees in this evaluation highlighted using the portal for a significant amount of data mining and exploration. They commonly used the portal to undertake statistical analyses – correlation and regression analyses for example – to test hypotheses relating to species distributions and environmental niches. Many interviewees identified enormous value in the capacity of the ALA to assist in exploring and describing associations and relationships between species occurrence data and environmental (particularly climate) data. Interviewees valued this both in terms of the number and sophistication of the relationships able to be explored, and the speed and ease with which it can be done. There are many ways to put an area on the map in the ALA Spatial Portal. One of them is creating an “environmentally defined” area – in this case all areas in Australia with an elevation over 1500m are shown in red on the map. Running an area report on this area will then give you records for all species that occur in these alpine/subalpine zones, including endemic species, invasive species, threatened species, and migratory species. The images in the spatial portal (uploaded photographs) were also deemed helpful. One ALA Spatial Portal function is the “scatterplot”. This permits mapping all distributional records in environmental space – in this case the small box on the left shows records mapped according to temperature by rainfall. The dots in the lower right of that box are the “high temperature / low rainfall” records.Putting a boundary box around those records will mean they displaywith red circles on the main map. Another very useful aspect of the ALA identified by interviewees was the ability to zoom in on distributions and see whether a provenance is a large, genetically diverse population or just one isolated tree. Explore your area. This ALA tool shows what has been recorded near any given location. In this case, 7075 species have been recorded within 5 km of the ALA headquarters in Canberra. These records can be downloaded and sorted – for example – to show only orchids or only endangered species in this area. The ALA Spatial Portal allows a variety of mapping and analysis functions. This example shows a map for all records for endangered species in Kakadu National Park. Vignette#4 – State Data Management Working Group Since 2014, the ALA has organised and hosted regular face-to-face meetings of representatives from the state, territory and federal government environment departments with the ALA, TERN Ecoinformatics and additional attendees. These workshops discuss the management of biodiversity data across Australia with a focus on government departments. The ALA also maintains a Basecamp community site for this group to continue discussions online. The workshops have inspired the States and Territories to realise the value of such a community and they now hold quarterly community-of-practice teleconferences to discuss common issues and lessons learnt from other jurisdictions. This has included topics such as data management practices, standards adoption, data collection tools and policies and procedures. The ALA has allowed efficient sharing of data amongst the jurisdictions. The ALA as a standardised, central access point and has facilitated the transfer of full resolution data between jurisdictions providing more data to support environmental management decision making. “The ALA state-based data provider workshops have been instrumental in creating a new network of colleagues within the biodiversity informatics space. This network has benefitted ALA in gaining a much better understanding of State-based use cases, particularly those associated with conservation planning and environmental impact assessment. It has benefitted theStates by gaining access to higher resolution data from the ALA repository. Concurrently, State-based colleagues have used these workshops to spawn ongoing national conversations that have resulted in a better understanding of how sister organisations work, and what best practice biodiversity informatics and capability maturity might look like.” – Western Australia Department of Parks and Wildlife. “The State Data Management meetings in 2014 and 2015 hosted by the ALA came at a very important time for the N.T. We were in the process of developing a new oracle database system for the storage and management of biodiversity data for the Northern Territory. The meetings resulted in the incorporation of standards into our database, such as Darwin Core, that facilitated data exchange between the ALA and the NT. The meetings also provided us with a greater understanding of how the ALA works and the services it provides. This has contributed to our adoption of ALA web service for some of our internal data management system as well as the Field Data system (wildwatch.nt.gov.au) and we currently have applications in development using the ALA BioCollect system. These systems will likely streamline our data collection and management systems in the future as well as result in significant cost savings. Finally the meetings provided a valuable opportunity to meet biodiversity data managers from other jurisdictions and to discuss common issues and experiences.” – Northern Territory Department of Environment and Natural Resources. 3.3 New Products and Services Examples As open infrastructure, the ALA currently supports or contributes over 100 portals, hubs, apps and virtual labs – this can include hubs sitting on top of the ALA infrastructure, new instances of the ALA infrastructure or apps or websites that use the ALA’s open APIs for external parties to independently access ALA data and some data processing services. Examples of these types of products and services include the NeCTAR supported Biodiversity and Climate Change Virtual Lab – BCCVL for use by the science community; the ZoaTrack lab for visualising tracking data to specific hubs (Australian Virtual Herbarium, Murray-Darling Basin Authority Data Portal, FishMap, Atlas of Prehistoric Australia, Australian Plant Pest Database, and others) to community science or citizen science projects (e.g. Many and various koala counts, platypus, bird and other taxonomically focussed projects, numerous projects undertaken by local and state governments, NGOs, community groups, NRM groups, environment and education centres, universities, and many others, ) and even a couple of commercial apps (NatureMap and QuestaGame). Additionally, the new BioCollect tool is enabling even greater community and professional access to sophisticated environmental data collection capabilities for field data capture and management. This is enabling standards-based data capture to facilitate aggregation and sharing, without compromising local and specific recording requirements. The ‘project finder’ tools in BioCollect also provide a comprehensive standards-compliant project registry for both hosted and non-hosted data recording projects, and the citizen science version of this is linked to other global project catalogues, principally based in the USA, providing the most comprehensive and sophisticated access point to discover and connect with citizen science projects globally. This project finder currently lists over 420 projects which Australians can participate in (approx. 100 of which are supported directly by ALA for data capture). It also lists and provides access to over 520 more projects globally through interoperability with other project finders. The ALA also hosts DigiVol, a volunteer transcription portal for such activities as digitising specimen label information, transcribing field notebooks and tagging/identifying camera trap photos. DigiVol has reached over 400,000 transcription tasks achieved by volunteers. The ALA infrastructure has been used by numerous countries for their national biodiversity portals, and importantly, this work has been undertaken in collaboration with the Global Biodiversity Information Facility (GBIF). The following are some examples (with development of other portals underway for Germany, Portugal, Sweden, UK and Canada): . GBIF France – http://portail.gbif.fr/ . GBIF Spain – http://datos.gbif.es/ . CRBio Costa Rica – www.crbio.cr . ICMBIO Brazil – http://www.icmbio.gov.br/portal/portaldabiodiversidade . NBN Scotland – http://www.als.scot/ These examples of leveraging open infrastructure without having to duplicate or re-create represent enormous value that the ALA is providing not only to a national community but also globally. ALA Impact Survey 2016 Survey respondents revealed that the ALA has definitely amplified and improved the creation, delivery and competitive advantage of new products and services and enterprises,results are shown in Table 3-2. Some highlights are described below: 76 per cent of respondents agreed or strongly agreed that the ALA had assisted them in the production of reports, papers and publications. . 61 per cent of respondents agreed or strongly agreed that their use of the ALA had helped support or inform government (local, state, national) policy or programs. . 47 per cent of respondents agreed or strongly agreed that they had used the ALA as a tool in building and improving skills in science, technology, engineering and mathematics (STEM). Table 3-3. Survey responses to statements related to the question of whether the ALA has amplified and improved the creation, delivery and competitive advantage of new products and services and enterprises. Neither Don't Strongly Strongly Statement Disagree disagree Agree know Disagree Agree nor agree / N/A The ALA has assisted me in the 3% 8% 5% 39%37% 8% production of reports, papers and (n= 1) (n= 3) (n= 2) (n= 15) (n= 14) (n= 3)publications. I use the ALA as a tool in building and 3% 13% 24% 29% 18% 13% improving skills in science, technology, (n=11) (n= 5) (n=9) (n=11) (n=7) (n= 5)engineering and mathematics (STEM). The ALA has helped increase interest in, 3% 8% 39% 21% 18% 11% participation in, and awareness of the (n= 1) (n= 3) (n=15) (n=8) (n= 7) (n=4)STEM sector. I have used the information/data/tools/services of ALA 5% 24% 16% 16% 21% 18% as part of a value proposition for a new (n= 2) (n=9) (n= 6) (n=6) (n= 8) (n=7) business or service or approach (e.g. app, methodology, platform). My use of the ALA has helped support 8% 5% 11% 29% 32% 16% or inform government (local, state, (n= 3) (n= 2) (n= 4) (n= 11) (n= 12) (n= 6)national) policy or programs. A comment made by a respondent illustrated a limitation of the ALA for the respondent’s environmental impact assessment work – specifically, the restricted access to full resolution priority and threatened species occurrence data. For another respondent, the ALA API enabled them to search and add species occurrences to their web application to enable distribution modelling and trait modelling. Interviews Interviewees provided the following more detailed information on, and examples of, the influence of the ALA on the development of products and services. Assistance in the production of reports, papers and publications Some examples of significant reports produced using ALA data, tools and services include: . The conservation of lizards in the Northern Territory. . The future of eucalypt trees in Australia. . Papers confirming information about specimens and species identification. . NCCARF reports, which are often underpinned by ALA data, e.g. report on biodiversity refuges Reside et al (2013). Interviewees mentioned the significant assistance the ALA provides in producing maps. The ALA also helped with the establishment of research projects by identifying areas and designs for sampling, which can often contribute to a paper. The ALA can provide excellent statistics on the location and abundance of species. It was suggested that many people are using ALA records to do distributional analyses. Without the ALA, these studies would not have had the appropriate number of records for statistical interpretation. Access to ALA data and tools has assisted the Upper Murrumbidgee WaterWatch Program, and others like it, to produce an important “state of the environment” condition report for rivers and streams – namely their annual Catchment Health Indicator Program (CHIP) report. The ALA tools enable them to capture field data from hundreds of individuals and groups in a consistent, efficient and standard way. It allows them to visualise and make that data accessible to their contributors, and also to download the data from one source (rather than multiple sources) for analysis in other off-line tools. Such analysis would be extremely difficult, time consuming and costly without the services provided by the ALA. Use of the data, tools or services of ALA as part of a value proposition for a new business or service or approach Interviewees pointed out that people can and do use the ALA to identify issues that may occur in relation to them undertaking a new enterprise, for example environmental impact assessments for a mining venture or other enterprise such as a fish farm. From Queensland Museum’s perspective, the value of the ALA in this regard is about extrapolating from the specific to the global.Using the pharmaceutical development process as a theoretidcal example, if a species is demonstrably rare and poorly distributed, then there may be little or no point in pursuing pharmaceutical development. Conversely, if the information in the ALA shows the species is widespread and common, it may provide a very good opportunity for pharmaceutical development. Supporting and informing government (local, state, national) policy or programs The ALA has also helped increase understanding of the level government investment in actions, programs and outcomes and their alignment with government policies. Interviewees provided the following insights: . The use of data from the ALA to model species distribution to develop surveys for representative national reserve systems (i.e. identifying data and knowledge gaps to help target new survey areas for cost and time efficiency) is a critical component of many programs, including BushBlitz. It can also be used to evaluate the potential biodiversity value of land. . The ALA helps identify the biodiversity values of remnant vegetation in and around urban areas, which assists in making decisions as to whether remnants can be rehabilitated. . CSIRO has produced many reports on the impact of climate change on biodiversity to DoEE and the former Department of Climate Change and Energy Efficiency using the ALA. . Reports produced with the assistance of the ALA are beginning to influence policy and funding decisions. For example, WaterWatch data collected and exposed in the ALA was able to show very poor water quality conditions for Cooma Creek, which in turn led to an investment of more than $100,000 in a program to restore the creek. . In the ACT, WaterWatch data captured by the community and delivered through the ALA is having impact on the future development of monitoring programs for catchment health and investment in rehabilitation programs. An even greater influence was expected over the next 3-5 years. . Use of the ALA can contribute to documenting how the Australian Government is delivering on international policies. For example, the ALA provides details of any projects that are working on specific species that are being considered by the threatened species scientific committee for listing or delisting. . For the ALA to have greater value to state governments in their responsibilities of checking for compliance with environmental regulations or for development planning, there would need to be greater transparency of ALA infrastructure development (e.g. through governance or technical committees). . At a state jurisdictional level it was highlighted by a respondent that biodiversity information management challenges across jurisdictions are similar. So, a central aggregation information system such as the ALA makes economic and practical sense, provided there is strategic alignment with the states/territory governments in terms of needs and objectives for dealing with biodiversity data-related problems. . This respondent also suggested the relationship between ALA and AEKOS is unclear and consequently there is uncertainty about the future of these products and how they relate to one another. Building and improving skills in STEM One interviewee spent nearly a year travelling and talking to university students and agencies, showing how they can use the ALA spatial portal to assist with exploratory data analyses, ecological and spatial analyses. The same interviewee continues to assist people to build their skills in the use of the ALA to carry out analyses. The spatial portal provides tools for analysing where associations in distribution occur and for downloading data for modelling. In this regard, it was felt that the ALA provides an excellent environment for primary exploratory data analyses. Universities could set up a project for first year students using the ALA and let students explore the data as they choose. Vignette#4 – Native forests and climate change: Lessons from eucalypts When looking at climate change, researchers tend to look at the natural distribution of a species, then apply climate change data to assess the proportion of species that may be affected. Using the ALA data and tools however, it is possible to look at Australian species, which have been planted around the world and where they have been found suitable. For example, several Eucalyptus species grow overseas under warmer conditions than they can in Australia. In studying climate change, there is a need to be able to understand their adaptability globally. Using the ALA to analyse the bioclimatology of native eucalypt forests, Booth et al. (2015) reviewed studies relevant to potential climate change impacts on natural stands of eucalypts, with a view to identifying not only specific lessons for the management of native forests in Australia but also some general lessons relevant to native forests anywhere. More than 800 species of Eucalyptus are found naturally across Australia, as well as species such as E. deglupta and E. urophylla in countries north of Australia. Eucalypts provide a particularly interesting opportunity to examine the likely impacts of climate change, as many species have been widely evaluated in trials within and outside Australia, often under conditions that are warmer and sometimes drier than those found within their natural distributions. Results from these trials indicate the intrinsic ability of particular eucalypt species and provenances to tolerate conditions that are somewhat different from those experienced within their natural distributions. Eucalypts have particularly poor dispersal capabilities, so natural stands will be generally unable to track changing climatic conditions. Therefore, in the period to the end of the present century a key issue for each eucalypt species under climate change is whether its intrinsic adaptability will be sufficient to allow it to survive where it is currently located. Their ability to survive will be affected not only by climatic, but also atmospheric changes, which will affect important processes such as photosynthesis and water exchange. Again, eucalypts provide a useful group for climate change studies as their commercial significance has led to various enhanced carbon dioxide experiments being carried out, as well as detailed genomic studies. (Source: Dr Trevor Booth CSIRO) 3.4 Productivity and Efficiency ALA Impact Survey 2016 The survey and interviews attempted to elicit whether there were any percentage productivity or efficiency gains associated with use of the ALA in terms of a respondents biodiversity data management workflows or along the information supply chain – with the key components of that supply chain being : . Access – data discovery (where is it, does it exist), licencing or legal arrangements, data access (transfer, extract, download (manual/machine), load data. . Transformation – understanding the data, cleaning/formatting the data, making data fit for purpose, linking data with other data. . Delivery – using the data, delivery of a product, undertaking analysis. In addition, respondents were also asked to estimate their organisation’s annual expenditure on meeting Australia’s biodiversity targets and investment in biodiversity that was better targeted or more efficient due to their use of the ALA. Most importantly, respondents were specifically asked to estimate the relative efficiency gains (primarily reduced input costs) they achieve by using the ALA. Of most interest to this review were the estimates of relative efficiency gains. Responses are discussed below and used as an initial attempt to estimate the potential economic value of the benefits outlined in Section 5 of this report. The analysis is, to a degree, quite limited. It should be noted that many respondents were unable to answer all of the economic questions because often they simply did not know their organisation’s levels of relevant expenditure, or were unable to provide a quantitative range for efficiency gains.2 Estimated productivity and efficiency gains Key findings from the survey are shown in the table below, where: . The first column shows the question and specific use of the ALA. . The second column shows the weighted average response across all respondent groups for each question. Note: The weighted averages were calculated where respondents were able to answer questions relating to relative efficiency gains and expenditure levels. . The four columns to the right of the table show the average responses and the range of responses (in brackets) for each of the individual response group. . Only responses that have estimated efficiency gains were analysed. This reduced the sample size used in the analysis. The small sample size does not allow for weighted averages to be calculated across user types (researchers, industry, government, community). The key points to note from the analysis are: . Generally, estimated efficiency gains had a weighted average around 12-13% across the whole sample. This relates to data access, transformation and delivery. This also relates to expenditure. . Researchers generally estimated higher relative efficiency gains than other respondents. This is likely to be due to a greater technical understanding of the complexities and costs of developing the data themselves. . Government respondents generally estimated lower efficiency gains than other respondents, particularly with respect to expenditure and investments. While the use of third-party researchers for data analysis may explain the data-related results, the lower efficiency gains identified for expenditure and investment may indicate that government expenditure and investment decisions are influenced by broader considerations. 2 This is not unusual for data and research projects as they are rarely subject to formalised business cases due to underlying uncertainty in benefit streams. Table 3-4. Estimated % productivity and efficiency gains Averages for respondent groups (range in brackets) Question and ALA-related activity To what extent do you think your use of the ALA has improved your organisation’s data management efficiency in relation to the time and resources spent in biodiversity data access? To what extent do you think your use of the ALA has improved your organisation’s data management efficiency in relation to the time and resources spent in biodiversity data transformation? To what extent do you think your use of the ALA has improved your organisation’s data management efficiency in relation to the time and resources spent in biodiversity data delivery? To what extent do you think your use of the ALA has improved your expenditure on meeting Australia’s biodiversity commitments? To what extent do you think your use of the ALA has improved your biodiversity investments? Weighted 3 average 12.7% 12.7% 12.3% 12.5% 13.5% Researchers Government 20.8% 11.9% (0-100%) (0-25%) 16.1% 10.9% (0-50%) (0-25%) 13.3% 7.8% (0-22.5%) (0-22.5%) 11.9% 2.2% (0-22.5%) (0-12.5%) 15.5% 1.7% (7.5-22.5%) (0-7.5%) Industry 21.1% (2.5­95%) 9.2% (0-50%) 14.1% (2.5­50%) 8.1% (2.5­22.5%) 6.0% (2.5­12.5%) Community No quantitative responses No quantitative responses No quantitative responses No quantitative responses 22.5% Overall, the ALA Impact Survey data indicate significant relative efficiency gains from the use of the ALA across the broad spectrum of typical uses, and across the diversity of different user types. 3 The weightings for this calculation are based on expenditure indicated by the respondent. This figure is based on responses where both an expenditure figure and an efficiency gain were estimated by the respondent. Draft Evaluation Report: Assessment of the Atlas of Living Australia’s Impact and Value 35 3.5 Applications and Derivatives ALA Impact Survey 2016 The survey asked a number of questions related to how the use of the ALA may have helped in the application of knowledge and activities in key sectors, including through the creation of “information derivatives” – results are shown in Table 3-5. The survey found that ALA has helped significantly in the application of knowledge and activities in key sectors, including that: 65 per cent of respondents agreed or strongly agreed that the ALA had helped the community and organisations to undertake on-ground interventions and actions with respect to biodiversity. . 53 per cent of respondents agreed or strongly agreed that the ALA had helped in the application of research to address key national and global challenges, and . 41 per cent of respondents agreed or strongly agreed that the ALA had helped the Government meet its international biodiversity and development treaty obligations. Table 3-5. Survey responses to statements related to the question of the extent to which the ALA has helped in the application of knowledge and activities in key sectors including through the creation of “information derivatives” Neither Don't Strongly Strongly Statement Disagree disagree Agree know / Disagree Agree nor agree N/A The ALA has helped in the 0% 0% 24% 21% 32% 24%application of research to address key national and global (n= 0) (n= 0) (n= 8) (n= 7) (n= 11) (n= 8) challenges. The ALA has helped improve 6% 6% 32% 24% 6% 26%regulatory compliance by Industry and Government with respect to (n= 2) (n= 2) (n= 11) (n= 8) (n= 2) (n= 9) environmental legislation. The ALA has helped improve 3% 9% 18% 24% 15% 32%Government decision-making regarding investment of public (n= 1) (n= 3) (n= 6) (n= 8) (n=5) (n= 11) funds. The ALA has helped the 0% 6% 21% 35% 6% 32%Government meet its international biodiversity and (n= 0) (n= 2) (n= 7) (n= 12) (n=2) (n= 11) development treaty obligations. The ALA has helped the community and organisations to 0% 0% 21% 47% 18% 15% undertake on-ground (n= 0) (n= 0) (n= 7) (n= 16) (n= 6) (n=5) interventions and actions with respect to biodiversity. Interviews Interviewees provided the following more detailed information on, and examples of, the influence of the ALA on the development of applications and derivatives. Helping the community and organisations to undertake on-ground interventions and actions with respect to biodiversity Interviewees indicated the ALA had greatly assisted citizen science groups and projects in particular. For example, community groups undertake biodiversity surveys or intense activities such as a BioBlitz and upload data to the ALA. This empowers communities as the groups are able to tackle local habitat damage issues using data in the ALA. Community groups use the ALA to demonstrate local biodiversity values to local government and to advocate and promote areas for conservation. They are also able to use the ALA to identify regional issues that are impacting locally, such as runoff to the Great Barrier Reef. In addition, maintaining the value of the ALA (in terms of its data quality) provides motivation for ongoing funding and maintenance of systems run by local and regional organisations. The tools provided through the ALA are very useful for local and community groups. The application of research to address key national and global challenges Some examples provided by interviewees of the use of the ALA in underpinning research to address key national and global challenges include: . Habitat and species priorities for conservation. . Predicting the ranges of invasive species. . Predicting the influence of climate change on biodiversity and pest species. . Improvements in data and data management standards and tools, including software. . Input to pure research on first principles around ecological understanding. Specific examples cited included the movement of problematic ant species up altitudinal gradients as the climate warms, and investigation of the distribution of various native species in Tasmania with a view to what might happen with global warming. Of particular value is the removal of the cost of access to data which helps inform the research. Improved decision-making regarding investment of public funds Interviewees believed that, in general, governments are using ALA data with respect to investing in land acquisitions for reserve systems and in biodiversity services programs. Interviewees felt government now had access to information which enabled them to better understand what actions on the ground (e.g. surveys) are being undertaken and, from this, undertake research s to determine whether outcomes are being achieved. The MERIT case study provides a specific example (although not the ALA website) where government is better able to capture and provide data to underpin government investment with respect to outputs from various funding programs, as well to be able to analyse program implementation and performance. A respondent indicated that for the ALA to have greater influence on state government planning, compliance and regulation activities, and therefore industry regulatory compliance with environmental legislation, they (the ALA) would need to host more systematic and structured survey data, not just observational data. This would necessitate greater engagement with the objectives and direction of ALA infrastructure development, and also improve the clarity between AEKOS and ALA data sharing. Meeting international biodiversity and development treaty obligations Interviewees stated that the scale of the ALA and its data helps in meeting international reporting obligations and that currently no other system can do this. Vignette#5 – Estimating potential range and hence climatic adaptability in selected tree species Interviewees in this evaluation described how the ALA helps to identify gaps in species survey coverage and where research needs to be done to contribute to the global evolutionary commentary. It has already contributed to identifying past refuges and speciation and can contribute to further investigations of what might happen under climate change. Estimating climatic conditions within the potential range of different species is important, as it can assist to evaluate their ability to tolerate climate change. Booth (2016) analysed potential range using data from the ALA and BIOCLIM analysis in relation to three climatic variables: a growth index, the mean minimum temperature of the coldest period (week) and a moisture index. Three eucalypt species were analysed to demonstrate some of the strengths and weaknesses of the method. These included a well-known commercially important species (Eucalyptus globulus), a lesser-known species (E. botryoides) and a rare species (E. kruseana). To provide a simple assessment of climatic adaptability the highest values of mean annual temperature were determined from within the potential ranges of the three species. It was concluded that, if they are available, analysing conditions at eucalypt plantings outside their natural distributions may be most useful for determining how species may cope with changing climates. However, if such data is not available, for example for lesser-known or rare species, then the analysis of the potential range may provide some tentative indication of species likely to display climatic adaptability (Source: Dr Trevor Booth, CSIRO). 3.6 Counterfactual The value of considering a counterfactual situation in impact analysis rests on the need to avoid overestimating the beneficial impact attributable to an activity – for example a successful research project. The appropriate baseline is an estimate of what would have happened in the absence of this activity. For example, some of the impact may be achieved through “structural” changes or trends, or it could be that the activity displaces another, which would have achieved some or most of the observed impact. The questions therefore are: . What scale of changes in the impact area might have taken place without ALA? . Are there any substitutes that could have led to similar outcomes/impacts available to society in the absence of this work? In relation to the first question, there is strong evidence that in the absence of the ALA, obtaining biodiversity data and information would have involved considerably more time and effort and resulted in incomplete information. This is because Australia’s biodiversity information management efforts had been fragmented and data difficult to access. A major barrier to Australia's biodiversity research and management efforts has been the fragmentation and inaccessibility of biodiversity related data. Data and information on Australian species has been, and still is, generated and housed in museums, herbaria, collections, universities, research organisations, and government departments and agencies. Obtaining an integrated suite of records and data sets from these groups involved considerable time and effort, and often resulted in incomplete information. To overcome these issues, Australia's biodiversity information needed to be aggregated, federated, connected and made easily discoverable and available by the users and contributors of this data. In relation to the second question, as discussed above, ALA infrastructure has enabled the aggregation of Australia’s biodiversity data and information and made it easily discoverable and available by the users and contributors of this data – a task difficult to emulate by other programs that have not had the significant investment that the ALA has had. ALA users from the impact survey and interviews were clear about improvements in outcomes and impacts that were uniquely attributable to ALA. Comments from many ALA users highlighted the value offered by the ongoing (stable and significant NCRIS funding), integrated, flexible, and user-focused model, which has not been duplicated by other relevant programs such as the Environmental Resources Information Network (ERIN) and AEKOS. In short, the counterfactual for removal of what is perceived to be an efficient program delivering significant benefits in an area of national and global need would be significant loss of national and global benefits. ALA Impact Survey 2016 A majority of ALA Impact Survey respondents reported that prior to the ALA, they went to individual data providers or their own data sources in order to discover and assess biodiversity data and information. Table 3-6. Where did you previously (prior to ALA) discover and access information/data for your purposes/project? Where did you previously (prior to ALA) discover and access information/data for your purposes/project? Response Answer Options Response Count Percent Individual data providers 63.9% 23 Own data sources (existing) 61.1% 22 Own data sources (new – had to go and collect specifically) 36.1% 13 Other data aggregation / repository service 19.4% 7 answered question 36 Approximately 74% of respondents now use ALA as their primary source for biodiversity data and information. Table 3-7. Where do you currently go to discover and access information/data for your purposes/project? Where do you currently go to discover and access information/data for your purposes/project? Response Answer Options Percent Response Count Primarily the ALA 74.3% 26 ALA only and nowhere else 0.0% 0 Individual data providers 31.4% 11 Own data sources (existing) 37.1% 13 Own data sources (new –go and collect specifically) 31.4% 11 Other Data aggregation / repository service 17.1% 6 answered question 35 Interviews Participants in interviews undertaken as part of this evaluation from commonwealth agencies indicated that previously without the ALA they had to request data from state and territory agencies individually and in some cases pay for that data. Similarly there was no system for automatic updates to the data. ALA has also now mobilised for them previously difficult to access data such as from herbaria or Birdlife Australia data. Other interviewees stated that prior to the ALA, the full potential of data was not fully realised and that it was often of lesser quality and consistency. The ALA has allowed streamlining and standardisation of information management workflows, and has improved user confidence in the data. Prior to the ALA, data collation, transformation, storage and analysis was done on paper or in Excel or an Access database. It was their opinion that if the ALA did not exist, agencies would need a great deal more resources (people and infrastructure) to do the same job. Interviewees from WaterWatch noted that prior to the ALA, data was managed in Excel within each catchment group, in a non-consistent manner across catchments and types of analysis – with the focus of groups more on community education and engagement. With the implementation of the ALA and the use of its tools, the focus has now shifted to being data driven and data focused. Attribution ALA was the primary source of information, tools and services that underpinned the productivity or efficiency gains associated with many respondent’s biodiversity data management workflows and along the information supply chain. It was appropriate to attribute benefits among the results from Table 3-7 on a response percentage basis. Consequently, in this analysis, it is assumed that roughly 75 per cent of benefits arising from the use of the ALA can be attributed to ALA. 4 Economic Quantification of ALA Impact Given project timing and data constraints, only the major and most obvious benefit streams have been evaluated. Therefore, it should be noted that this approach is relatively conservative and has likely resulted in an underestimate of actual benefits. Costs have been based on financial analysis of the ALA budget. Two separate cost bases have been used in the analysis. The first one represents the proposed cash budget of the ALA. Secondly, as a sensitivity test, we have also run the analysis using the full cost of the ALA including any likely future in-kind inputs. Using the benefit and cost estimates, we have then used our benefit-cost analysis model to calculate the: . Estimated net benefit from ALA products and services (benefits–costs). . Benefit-cost ratio (BCR) (benefits/costs). . Threshold for the change in benefits required before benefits exceed costs. This is a relatively standard approach to measurement of the benefits and costs of research and development. To estimate economic benefits we estimated the expenditure or value base for key users of the ALA (i.e. what is the typical annual budget for the key agents using the ALA). This was obtained through the 2016 ALA Impact Survey. We also estimated the relative benefits (% efficiency gain / % cost reduction) attributable to the use of the ALA using a weighted average from the survey responses (including responses where no efficiency gain was identified). Finally, we estimated the benefits accrued by users of ALA products and services by multiplying the estimate of the expenditure or value base by the estimated efficiency gain attributable to ALA products and services. The bulk of the estimates produced represent improvements in producer surplus achieved via efficiency gains. 4.1 Costs Establishing the costs involved throughout the entire inputs to impact pathway is an important exercise of a cost-benefit analysis. This includes both the input costs incurred by ALA and its partners, as well as any usage and adoption costs borne by clients, external stakeholders, intermediaries and end users. Given the length of the project and commercial confidentiality issues, we were unable to identify usage and adoption costs borne by intermediaries and end users of ALA markers. For the purpose of this evaluation, we only included ALA investment it has received from the Australian Government. The ALA has been funded by number of Australian Government programs, primarily National Collaborative Research Infrastructure Strategy (NCRIS), but also previously the Collaborative Research Infrastructure Scheme (CRIS) and the Education Investment Fund (EIF). Estimates of the funding by program are shown in Error! Reference source not found.. Table 4-1. ALA funding arrangements ($ million) 2007­08 2008­09 2009­10 2010­11 2011­12 2012­13 2013­14 2014­15 2015­16 2016­17 NCRIS $1.65 $1.65 $1.65 $1.65 $1.65 $1.48 $4.28 $4.60 $4.70 EIF $10.00 $10.00 $10.00 CRIS $2.80 Total (all years, all sources): $56.11 million As outlined in Table 4-1, the Australian Government contributed $56.11 million to the ALA between 2007-08 and 2016-17 in nominal terms. These contributions were CPI adjusted and discounted using a real discount rate of 7%, which translated to $81.47 million in 2016-17 dollars. An annual costs basis of $8.15 million is assumed for this analysis. 4.2 Benefits As outlined in Section 2 of this report, during the survey a number of questions were asked relating to expenditures relevant to different uses of the ALA (data access, transformation, or delivery) and investments informed by the use of the ALA. Furthermore, respondents were also asked to estimate the % efficiency gains achieved from using the ALA. We then analysed the respondents’ data to establish some indicative estimates of potential benefits.4 Assuming that the survey is relatively representative of typical users of the ALA, it is possible to obtain indicative estimates of the likely relative gains from the use of the ALA for the main categories of use. Table 4-2. Analysis of survey responses – economic questions Use of ALA Total annual spend from respondents ($) Average annual expenditure ($ per respondent) Weighted average efficiency gain (%) Data access $2,700,000 $77,000 12.7% Data transformation $2,450,000 $70,000 12.7% Data delivery $3,600,000 $102,000 12.3% Meeting Australian biodiversity commitments $6,800,000 $195,000 12.5% Investment informed by ALA $4,800,000 $137,000 13.5% Total $20,300,000 12.8% Key findings from the analysis of survey responses include: . The survey responses were dominated by researchers (around 46%), resulting in the skewing of the use of the ALA and the values towards data management issues rather than the efficiency gains that result from the enhanced investment decisions attributable to the ALA. Industry and government respondents accounted for around 26% and 23% respectively. . Given the fact that the respondents were only a subset of potential users, any aggregate estimates of efficiency gains based on the survey responses alone will be significant underestimates. However, even responses from the small sample size achieved indicate annual benefits from efficiency gains are around $2.6 million. . The survey responses generally indicate relative efficiency gains from the use of the ALA for both data activities and better-informed actions and investments are generally around 12-13%. Given that only a sample of ALA users responded to the survey, it is probably the weighted average efficiency gains that provide the most insight. Where relevant aggregate estimates of total expenditure could be 4 As the survey allowed for ranges of responses (e.g. $100,000 to $200,000, or 5-10%), in this analysis we have used the mid-point of the range provided by respondents to develop these estimates. Responses were capped within the ranges provided in the survey instrument unless the respondent provided specific data. The weighted average efficiency gain was estimated using response data where the respondent was able to provide both an estimate of expenditure and an estimate of efficiency gains for that relevant use of the ALA. obtained, it would then be possible to estimate a range for the total benefits to the nation attributable to the ALA. Aggregation of the value of likely expenditure that could be enhanced through use of the ALA Because of the relatively small sample of ALA users surveyed in this review, it is necessary to estimate the aggregate value of relevant expenditure on biodiversity management by key users. The efficiency gains indicated from survey respondents can then be applied to the estimated expenditure base to estimate the potential benefits from the use of the ALA. There are no official estimates of expenditure influenced by the ALA. However, there is some data that can be analysed to provide an indication of the potential range of relevant expenditure. The ABS, through its ongoing program of expanding the national accounts framework has previously estimate annual general government sector in environmental protection (environmental and natural resource management) at between $1.5 and $1.8 billion. Analysis of relevant commonwealth, state and territory budget statements indicate annual expenditure on biodiversity protection including reserves management is typically around $1.6 billion.5 The annual public expenditure on biodiversity management is in the range of $1.5-$1.8 billion. Based on the information currently available, it is reasonable to assume that the use of the ALA could enhance the efficiency of this spend. 4.3 Results – The Threshold Test Because we do not know the actual aggregate value of expenditure influenced by the ALA and the efficiency gains across that whole expenditure pattern, it is not possible to do a full cost-benefit analysis and estimate a definitive BCR. However, based on the information available it is possible to estimate the threshold for the relative efficiency gain from using the ALA that would be required to justify ongoing expenditure on the ALA. That is, how much of an efficiency gain is necessary to justify spending $8.15 million per annum on the ALA, where the BCR is greater than 1. To estimate this threshold, we used the annual relevant expenditure for the Commonwealth (around $280 million per annum) and the whole general government sector (around $1.6 billion) as the basis for the expenditure that could benefit from the ALA. We used the $8.15 million in expenditure as the costs. The threshold efficiency gain required to justify the continuation of the ALA and the reported efficiency gains from the survey are shown in Figure 4-1. The key points to note from the analysis are: . Analysis of survey responses indicated that weighted average efficiency gains from using the ALA were around 12.8% of relevant expenditures. . To justify the ALA (BCR is greater than 1) for Commonwealth expenditure only, an efficiency gain of 2.9% of total relevant expenditure would be required. This is significantly lower than the stated efficiency gains from survey respondents, including responses from Commonwealth officials. . If the broader measure of relevant expenditure is used (all general government expenditure on biodiversity from the Commonwealth and all states and territories), an efficiency gain of 0.5% of total relevant expenditure would be required. Again this is significantly lower than the stated efficiency gains from survey respondents. However, it should be noted that not all states and territories are using the ALA to inform planning, management and investment. 5 The Australian Government expenditure on biodiversity and national parks management is typically around $280 million. Analysis of relevant state and territory budgets indicates significant expenditure at the state level including NSW (around $270 million per annum), Victoria (around $150 million per annum), Queensland (around $400 million per annum), South Australia (around $130 million per annum), Western Australia (around $210 million per annum), Tasmania (around $90 million per annum), NT (around $60 million per annum), and the ACT (around $20 million per annum). Economic analysis shows that only small gains in the efficiency of biodiversity expenditure are needed to justify the continuation of the ALA. These required efficiency gains are significantly less than those indicated though the ALA Impact Survey and consultation. Efficiency gain thresholds 14.0% Commonwealth Total Australia Reported efficnecy gains Clearly the benefits of the ALA significantly exceed the costs and any BCR is likely to be significantly higher than 1. For example, if the survey results are applied to Commonwealth expenditure on biodiversity and national parks, the annualised economic benefit is estimated at $26.9 million in 2016 dollars with a benefit-cost ratio of 2.6. Box 1 – What is an acceptable benefit-cost ratio for public investments? In economic analysis, any project with a benefit-cost ratio (BCR) greater than 1 is acceptable as the benefits of the project are greater than the costs. When comparing projects, the project with the highest BCR is generally considered superior. It is instructive to compare the economic analysis conducted in this project to other public projects currently being funded by the Australian Government. A review of physical1 and non-physical infrastructure projects being considered indicates a range of BCRs for various projects including: . NSW WestConnex (BCR = 1.7) . Victoria Melbourne Metro Upgrade (BCR = 1.17) . Tasmania. Just add water (An Innovation Strategy for Tasmania: Focus on Food Bowl Concept (BCR = 1.6) . Houghton(2011) Case Study of Australian Bureau of Statistics (BCR=1.3-5.3) . Beagrie and Houghton (2014) three UK research data centres (BCR= 2.5-12) . Lateral Economics 2014 Returns from Australian Government investment on research data (BCR = 1.5) The information in this box shows that the BCR for non-physical infrastructure is higher than physical infrastructure projects which typically lie in the 1-2 range. The BCR exceeds 2 for most non-physical projects. 4.4 Sensitivity analysis We undertook sensitivity analysis to test the robustness of the economic analysis results to changes in assumptions and uncertain parameters. The results of the sensitivity analysis are shown in Table 4-3. Table 4-3. Results of sensitivity analysis Central Pessimistic Optimistic Variable assumption assumption assumption Efficiency gains (%) 12.8% 8% 15% Commonwealth Biodiversity Expenditure ($m 2016 dollars) 280 200 360 Attribution to ALA % 75% 50% 100% Discount rate % 7% 9% 5% Benefit-Cost Ratio 3.5 1.1 7.5 Annual benefits ($m 2016 dollars) 26.9 8 54 The last two columns of the table show the effects of varying the key assumptions (and their associated parameter values) at the same time. As mentioned above, the annual benefit associated with Commonwealth biodiversity expenditure ranges from $8m to $54 in the cases explored in the sensitivity analysis. We observed that NPV of benefits is particularly sensitive to changes in the discount and attribution rates, efficiency gains and government expenditure. The pessimistic and central (baseline) scenarios perhaps offered conservative yet realistic forecasts of future benefits. In this we estimated that the benefit-cost ratio is between 1.1 and 3.5. 5 Conclusions There has been significant growth in the use and uptake of the ALA and it has influenced cultural change in the biodiversity data collection, research, assessment, and policy communities. As both a direct and indirect consequence of the ALA, there are new products and services generating value for research, industry, government and community groups, and increases in productivity, efficiency and innovation (applications) in various sectors across these key stakeholder groups. There are areas where its value and influence can be improved especially in the environmental legislation compliance area but only in collaboration with the relevant agencies and actors in the information supply chain. Similarly there are opportunities to improve the ALA’s ability to engage and interact with others in the building of a value proposition for a new business or service or approach (e.g. app, methodology, platform). 5.1 Uptake, Usage and Adoption KEQ: To what extent has there been uptake and usage of a) the data delivered through ALA and b) the tools, services and infrastructure of the ALA by its four key stakeholder groups (research, industry, government, community) and “other platforms and countries”? There has been significant uptake and usage of a) the data delivered through ALA and b) the tools, services and infrastructure of the ALA by its four key stakeholder groups (research, industry, government, community) and other platforms and countries. The total number of records held in the ALA has risen from around 20 million in January 2012 to over 60 million in July 2016. The total number of records downloaded from the ALA now totals over 8.5 billion. Google Scholar results for the “Atlas of Living Australia” show that references to the ALA have been increasing steadily each year and now number almost 1050. The number of people using the ALA and number of sessions of use has also been increasing since 2009, with the number of user sessions now totalling almost 1.6 million. The ALA Impact Survey showed almost three quarters of respondents use the ALA over other data sources to discover and access data or information for their needs. The spatial and mapping tools were the most commonly used services, but analysis tools, including visualisation, data discovery tools, data capture, collection and upload tools and species profiles were also popular. 5.2 Influence on Cultural Change KEQ: To what extent has ALA influenced cultural change in the following behaviours? a. Open sharing of data b. Improved use of information standards in data collection and information management c. Use of open services/web services/ APIs – linkages and integration and interoperability between platforms and services and data d. Improved community connectivity and partnerships The ALA has had a substantial influence on the open sharing of data, improved use of information standards in data collection and information management, linkages and integration and interoperability between platforms and services and data, and improved community connectivity and partnerships. Aspects of the ALA and its use that ALA Impact Survey respondents were most positive about, and supportive of, in terms of proven benefit and value were: . linking and integrating different data types through the ALA (e.g. through the spatial portal), thereby adding value to the information . helping to improve the fitness for purpose, consistency and accessibility of online data through the embedding of standards in ALA systems . helping the community and organisations to undertake on-ground interventions and actions with respect to biodiversity. 5.3 New Products and Services KEQ: Has ALA amplified and improved the creation, delivery and competitive advantage of new products and services and enterprises by others? The ALA has amplified and improved the creation, delivery and competitive advantage of new products and services and enterprises by others. In particular, respondents to the ALA Impact Survey indicated the ALA is valuable in supporting and informing local, state and national government policies and programs and also assists greatly in the production of reports, papers and publications. Some examples of reports produced with the assistance of the ALA include: . The conservation of lizards in the Northern Territory. . The future of Eucalyptus trees in Australia. . Papers confirming information about specimens and species identification. . The National Climate Change Adaptation Research Facility (NCCARF) reports, which are often underpinned by ALA data, e.g. report on biodiversity refuges (Reside et al, 2013). Respondents to the ALA Impact Survey pointed out that people can and do use the ALA to identify issues that may occur in relation to them undertaking a new enterprise, for example a mining venture or a fish farm. Another example provided was the pharmaceutical development process from a species. If ALA data shows a species is demonstrably rare and poorly distributed, there may be little or no point in pursuing pharmaceutical development. Conversely, if the ALA shows the species is widespread and common, it may provide a very good opportunity for pharmaceutical development. Survey respondents also indicated the ALA has helped increase understanding of the level of government investment in actions, programs and outcomes and their alignment with government policies. 5.4 Productivity and Efficiency KEQ: Has the ALA resulted in productivity and efficiency gains in information management and delivery of services? The ALA has resulted in productivity and efficiency gains in information management and delivery of services. Estimated gains in productivity or work efficiency through use of the ALA were estimated to be around 10-15% from a sample of 45 survey respondents from government, industry, research and community groups. This was estimated to translate to $2.6 million in savings each year for the survey respondents and their associated agencies and groups alone. Researchers generally estimated higher relative efficiency gains than other respondents. This is likely to be due to a greater technical understanding of the complexities and costs of developing the data themselves. Government respondents generally estimated lower efficiency gains than other respondents, particularly with respect to expenditure and investments. While the use of third-party researchers for data analysis may explain the data-related results, the reactively lower efficiency gains identified for expenditure and investment potentially indicate that government expenditure and investment decisions are influenced by broader considerations. Overall, the survey data indicates significant relative efficiency gains from the use of the ALA across the broad spectrum of typical uses, and across the diversity of different user types. 5.5 Applications and Derivatives KEQ: To what extent has ALA helped in the application of knowledge and activities in key sectors including through the creation of “information derivatives”? The ALA has helped significantly in the application of knowledge and activities in key sectors, including through the creation of information derivatives. Fifty three percent of respondents in the ALA Impact Survey agreed or strongly agreed that the ALA had helped in the application of research to address key national and global challenges. None of the respondents disagreed with that statement. In addition, 41 per cent of respondents agreed or strongly agreed that the ALA had helped the Government meet its international biodiversity and development treaty obligations, with only 6 per cent disagreeing with that statement. Examples provided by interviewees of the use of the ALA in underpinning research to address key national and global challenges included: . Habitat and species priorities for conservation. . Predicting the ranges of invasive species. . Predicting the influence of climate change on biodiversity and pest species. . Improvements in data and data management standards and tools, including software. . Input to pure research on first principles around ecological understanding. 5.6 Economic Quantification of Impact With the limited information and resources available and initial economic analysis demonstrated that the ALA is an economically viable investment, where benefits derived by users are significantly higher than the costs of delivering the service. Furthermore, the benefits realised could be significantly increased through increased uptake and use of the ALA in informing planning, management and investment decisions. The ALA Impact Survey 2016 indicated annual benefits from efficiency gains are around $2.6 million. If the survey results are applied to Commonwealth expenditure on biodiversity and national parks, the annualised economic benefit is estimated at $26.9 million in 2016 dollars with a benefit-cost ratio of 3.5. 6 Limitations and Future Directions This evaluation has employed a mixed method to assess the research impacts arising from the ALA. It combines quantitative and qualitative methods to illustrate the nature of economic, environmental and social impacts. In cases where the impacts can be assessed in monetary terms, a cost-benefit analysis (CBA) has been used as a primary tool for evaluation. As a methodology for impact assessment, CBA relies on the use of assumptions and judgments made by the authors. This relates to economic indicators for impact contribution, attribution and counterfactual. These limitations should be considered when interpreting the above analytical results. Given the scope and budget for the analysis, we acknowledge that there are some limitations with regard to the evidence base of impacts. For example, we had limited knowledge about the benefits, arising from the additional use of the data facilitated by the ALA (i.e. the value of the data re-use). In addition, the likely environmental impacts were not quantified but treated as potential benefits due to the lack of reliable data. There were also implementation difficulties such as survey design (e.g. Questions to differentiate users and depositors, and in quantifying efficiency and costs), and it required a good deal of time to customise the questions and pilot test each survey. This evaluation represents an initial attempt to test the process and analysis framework. Nonetheless, insights into the impact and value of the ALA have been generated that can be further refined, improved and built upon. To continue to confirm and gather further evidence of these impacts and more accurately quantify the value of the ALA, we recommend the survey be hosted as an open survey on the ALA website to encourage continuous feedback. This means there will be a larger pool of respondents and survey responses will provide leads for future narratives. The survey could be simplified to request a standard response to each of the key evaluation questions developed for this evaluation. If it is preferred that the current survey or similar be repeated, recommended versions of the questions are provided at Appendix B. We understand that impact evaluation for research infrastructure is an evolving practice and suggest that as part of its evolution, it needs to address some key evidence constraints by planning for impact and monitoring progress towards it. It is also important to engage with users and other stakeholders to collect data/information and ensure a robust and thorough investigation of all the triple-bottom-line outcomes and impacts. It is therefore recommended that further effort and expenditure be assigned to better quantify the economic benefit derived from the ALA, thereby increasing confidence in the CBA undertaken here. It should be noted there currently exist rules of thumb that suggest program evaluations should involve around 5–10% of a program budget. Recommendations Our recommendations are relevant as appropriate to funders and depositors, data centres and repositories and other future studies. Recommendation 1: Continue support for the ALA impact study. The present study could profitably be followed by further studies to provide a more comprehensive picture of the ALA impact and value. Recommendation 2: Track changes over time. Value and perceptions of value change over time. A series of metrics should be determined that could be captured on an ongoing basis to monitor ALA performance in key areas. Recommendation 3: Conduct more granular analysis. A set of further case studies should be determined for examination to add to the impact evaluation. There is significant scope to examine the value of specific collections, or the economic value of service to specific groups. Recommendation 4: Further develop the methods. There would be benefits from further developing, refining and exploring applications of the methods used in this study (i.e. impact survey tool, monetisation techniques). Recommendation 5: Study the wide value and impact of NCRIS. The framework developed in this study could be applied across NCRIS and other publicly funded research infrastructure seeking to measure where open data infrastructure benefits or makes broader contributions to society beyond those contributions to academic knowledge. 7 References Booth, T.H., L.M. Broadhurst, E. Pinkard, S.M. Prober, S.K. Dillon, D. Bush, K. Pinyopusarerk, J.C. Doran, M. Ivkovich, A.G. Young (2015) Native forests and climate change: Lessons from eucalypts. Forest Ecology and Management. Volume 347, Pages 18–29 Booth, T. H., Williams, K. J. and Belbin, L. (2012), Developing biodiverse plantings suitable for changing climatic conditions 2: Using the Atlas of Living Australia. Ecological Management & Restoration, 13: 274–281. Booth, T.H. (2016) Estimating potential range and hence climatic adaptability in selected tree species. Forest Ecology and Management, v. 366, p. 175-183 Beagrie, N., and Houghton, J. (2014) The Value and Impact of Data Sharing and Curation: A synthesis of three recent studies of UK research data centres. Available at http://repository.jisc.ac.uk/5568/1/iDF308_­ _Digital_Infrastructure_Directions_Report,_Jan14_v1-04.pdf Donker, F.W., Loenen, B. Breg, A.K. (2016) Open Data and Beyond. International Journal of Geo-Information, 48. Higson, C. and Waltho, D. (2010) Valuing Information as an Asset. White paper for the London Business School. 20pp. Houghton, J. and Gruen, N. (2014) Open Research Data. Report to the Australian National Data Service (ANDS). Lateral Economics (2014) Open for Business: How Open Data Can Help Achieve the G20 Growth Target. A Lateral Economics report commissioned by Omidyar Network. Available at http://lateraleconomics.com.au/wp-content/uploads/omidyar_open_business.pdf Moody, D. L., and Walsh, P. (1999) Measuring the Value of Information-An Asset Valuation Approach. ECIS Jun. 1999: 496-512. Attachment A ALA impact survey questions ALAImpacts TheALAImpacts study is exploring the use of open source biodiversity data, information and tools. It is being conducted byAlluvium Consulting and CSIRO’s Performance and Evaluation Team with funding received fromALA. The last years have seen significant developments in increasingly access to data, information and tools fromALA-allowing the use and re-use of large quantities of biodiversity information. TheALA Impacts study is exploring how biodiversity data, information and tools fromALAare being used, and by whom. You are invited to participate in a survey that asks for your opinions on biodiversity data, information and tools fromALAand invites you to share details of any projects you have undertaken or explored using data, information and tools fromALA. There are 25 questions in this survey. The majority of questions are multiple choice and there are no large 'essay' questions. The survey will take between 10 and 20 minutes to complete, depending on whether you wish to give optional additional information about open data projects you have worked on. All respondents who provide an e-mail address on the last screen will be entered into a draw to win one of four $50 Coles/Myer Gift Cards. E-mail address will only be used for the purpose of the draw, and to inform respondents when the final report is published. E-mail addresses will be stored separately from responses to protect anonymity. HOW WILL MY INFORMATION BE USED? The results from this survey will be used as part of anALAevaluation report forALA’s internal and external use. To contribute to shared learning, aggregate statistics from this survey may be used to produce scientific journal publications and conference presentations. No personally identifying information will be included in this. CONFIDENTIALITY All information provided by you will be treated confidentially. No responses will be attributed to individuals in the analysis of the survey and data will be de-identified.Any data collected as part of this study will be securely stored as per CSIRO’s Recordkeeping Procedures. Your participation in this study is completely voluntary. You are free to withdraw by stopping the survey at any time, and you are free to skip any questions you do not want to answer. If you wish to remove you information from this study after you have completed the survey, you can do so by contacting the evaluator at any time. ETHICAL CLEARANCEAND CONTACTS If you have questions at any time about the evaluation or the procedures, you may contact the evaluator, Dr Jian Wang (jwang@csiro.au) or 02 6276 6174. This study has been approved by CSIRO’s Social Science Human Research Ethics Committee in accordance with the National Statement on Ethical Conduct in Human Research (2007).Any concerns or complaints about the conduct of this study can be raised with the Manager of Social Responsibility and Ethics on (07) 3833 5693 or by email at csshrec@csiro.au. 1.[CONSENT] I have read the statement above and I agree to take part in this study. YES NO 2.Establishing your identity code First 2 letters of your Mother's maiden name Month of your birth (January = 01) The first 2 letters of your first name 3.Which kinds of data and information fromALAdo you work with? Please choose all that apply: Off Occurrence records Off Environmental data Off Species descriptions or profiles, including taxonomy/names Off Images Off Conservation status of species Off Spatial Layers Off Genetic or phylogenetic data Off Literature Off Organisational/collection/dataset information Other (please specify) 4.Which kinds of tools, services and infrastructure fromALAdo you work with? Please choose all that apply: Off Spatial and mapping tools Off Analysis tools including visualisation Off Sandbox Off Data capture, collection and upload Off API / web services Off Alerts and annotations, personal profile management Off Data discovery – search, navigate, filter, lists Off Hubs Off Species profiles Off Profiles tool Other (please specify) * 5.Please give ashortdescription of an instance where you useddata,information andtools fromALA. For example:"I downloaded and explored occurrence records and environmental data. I wrote a report for the environment agency as a result of what I learnt from theALA". 6.Was the main focus of your use of data, information and tools fromALAprimarily on local, national or international issues? Off Local Off National Off International Off Not geographically specific Off Do not know 7.Where did you previously (prior toALA) discover and access information/data for your purposes/project? Off Individual data providers Off Own data sources (existing) Off Own data sources (new – had to go and collect specifically) Off Other Data aggregation / repository service (Please specify which ones) 8.Where do you currently go to discover and access information/data for your purposes/project? Off Primarily theALA Off ALAonly and nowhere else Off Individual data providers Off Own data sources (existing) Off Own data sources (new – had to go and collect specifically) Off Other Data aggregation / repository service (Please specify which ones) 9.To what extent hasALAand your use of it, influenced or changed any of the following behaviours, practices or culture in relation to open sharing of data, standards, use of open services and partnerships. My useof theALAhas changed my understanding and acceptance of open access using creative commons My useof theALAhas influenced my decision to provide data free of change – previously fees for download and access. The embedding of standards inALA systems has helped improve the fitness for purpose, consistency and accessibility There is an improvement in the amount of “trusted” or quality data on-line as a consequence of the establishment of theALA The linking and integration of different data types through the ALA(e.g. through spatial portal) has added value to information TheALAhas influenced me to ensure my services/platforms are available as open infrastructure Strongly Neither disagree Disagree Disagree nor agree Agree StronglyAgree Don'tknow/N/A Strongly Neither disagree Disagree Disagree nor agree Agree StronglyAgree Don'tknow/N/A TheALAhas enabled communities or groups to connect, collaborate and self-organize in groups of interest to improve their own activities, avoid re­inventing the wheel, reduce duplication, provide critical mass, and attract new customers / clients / My use of theALAhas influenced my “advocacy/promotion” to about any of the issues above (e.g. open access, standards, quality) Other (Please provide any specific examples of changes in behaviours or practices) 10.HasALAamplified and improved the creation, delivery and competitive advantage of new products and services and enterprises? Strongly Neither disagree Disagree Disagree nor agree Agree StronglyAgree Don'tknow/N/A I use theALAas a tool in building and improving skills in science, technology, engineering and mathematics (STEM) I have used the information/data/tools/ services ofALAas part of a value proposition for a new business or service or approach (e.g. app, methodology, platform) Other (Please provide any specific examples of new products and services) Questions 11 to 20 are designed to elicit your views on the potential productivity and efficiency gains attributable to the use of theALA. This is separated into questions relating to time and resources spent in data access, transformation and delivery for activities, decision-making and investment (questions 11 and 16) and questions relating to efficient allocation of resources such as government funding and business expenditure (questions 17 and 20). How to calculate the annual expenditure for time and resources · Formula: Cost = Price x Time x Quantity x Frequency. · Example: a data access activity takes 1 staff member (Quantity) 3 hours to complete (Time) and the hourly cost of the staff is $20 (Price). This activity occurs 2 times per year (Frequency). Hence the total annual cost = $20 x 3 x1 x2= $120 11.What is your organisation’s approximate annual expenditure that relates to biodiversity dataACCESS including data discovery, licence, data extraction and download? Off Less than $50,000 Off $50,000 to $100,000 Off $100,000 to $150,000 Off $150,000 to $200,000 Off $200,000 to $250,000 Off More than $250,000 Off Don’t know Off Not applicable to my role Off Other I can only talk at the individual/division/faculty (state the level and an approximate $ figure) 12.To what extent do you think your use of theALAhas improved your organisation’s data management efficiency in relation to the time and resources spent in biodiversity data ACCESS (e.g. 0%, 5% etc.)? 0% 0% to 5% 5% -10% 10%-15% 20%-25% Don’t know Not applicable to my role Other (please specify an approximate %) 13.What is your organisation’s approximate annual expenditure that relates to biodiversity data TRANSFORMATION including understanding the data, cleaning/formatting the data, making data fit for purpose, linking data with other data? Off Less than $50,000 Off $50,000 to $100,000 Off $100,000 to $150,000 Off $150,000 to $200,000 Off $200,000 to $250,000 Off More than $250,000 Off Don’t know Off Not applicable to my role Off Other I can only talk at the individual/division/faculty (state the level and an approximate $ figure) 14.To what extent do you think your use of theALAhas improved your organisation’s data management efficiency in relation to the time and resources spent in biodiversity data TRANSFORMATION (e.g. 0%, 5% etc.)? 0% 0% to 5% 5% -10% 10%-15% 20%-25% Don’t know Not applicable to my role Other (please specify an approximate %) 15.What is your organisation’s approximate annual expenditure that relates to biodiversity data/information DELIVERY including using the data, delivery of a product, undertaking analysis? Off Less than $50,000 Off $50,000 to $100,000 Off $100,000 to $150,000 Off $150,000 to $200,000 Off $200,000 to $250,000 Off More than $250,000 Off Don’t know Off Not applicable to my role Off Other I can only talk at the individual/division/faculty (state the level and an approximate $ figure) 16.To what extent do you think your use of theALAhas improved your organisation’s data management efficiency in relation to the time and resources spent in biodiversity data/information DELIVERY(e.g. 0%, 5% etc.)? 0% 0% to 5% 5% -10% 10%-15% 20%-25% Don’t know Not applicable to my role Other (please specify an approximate %) 17.What is your organisation’s approximate annual expenditure that relates to meetingAustralian Government biodiversity obligations? Off Less than $200,000 Off $200,000 to $400,000 Off $400,000 to $600,000 Off $600,000 to $800,000 Off $800,000 to $1,000,000 Off More than $1,000,000 Off Don’t know Off Not applicable to my role Off Other I can only talk at the individual/division/faculty (state the level and an approximate $ figure) 18.To what extent do you think your use of theALAhas improved the efficiency of this expenditure (e.g. 0% 5% etc.)? 0% 0% to 5% 5% -10% 10%-15% 20%-25% Don’t know Not applicable to my role Other (please specify an approximate %) 19.What is your organisation’s approximate annual data, information management and/or biodiversity related business investments informed by the use of theALA? Off Less than $200,000 Off $200,000 to $400,000 Off $400,000 to $600,000 Off $600,000 to $800,000 Off $800,000 to $1,000,000 Off More than $1,000,000 Off Don’t know Off Not applicable to my role I can only talk at the individual/division/faculty (state the level and an approximate $ figure) 20.To what extent do you think your use of theALAhas improved the efficiency of this business expenditure (e.g. 0% 5% etc.)?.)? Off 0% Off 0% to 5% Off 5% -10% Off 10%-15% Off 20%-25% Off Don’t know Off Not applicable to my role Other (please specify an approximate %) 21.To what extent hasALAhelped in the application of knowledge and activities in key sectors including through the creation of “information derivatives”? (actual or anticipated) Strongly Neither disagree Disagree Disagree nor agree Agree StronglyAgree Don'tknow/N/A development treaty obligations. Other (please provide any specific examples or additional information) 22.Your Location ACT NSW QLD VIC WA SA TAS NT International (Please indicate which country) 23.YourAge 18-24 25 -34 35 -44 45 -54 55 -64 65 and over 24.Do you see yourself primarily as a: Data provider / holder Data user Prosumer – provider and user Other (please specify) 25.Please select the main sector you are working in. Off Research -university and other research organisations, including museums and collections Off Government – local, state and national government, policy, planning, implementation, decision making, investment Off Industry – businesses/private sector, NRM organisations, agriculture/biosecurity, environmental assessment etc Community -Non-for-profit community groups or NGOs, students, citizen scientists, education, general public, communicators Off etc Other (please specify) 26.How were you made aware of theALA’s existence? Off Word of mouth Off Internet search Off Colleague/friend/teacher Off Social media Off Newsletter/blog Off Conference/forum/seminar Other (please specify) 27.Would you like to entered into a draw to win one of four $50 Coles/Myer Gift Cards? Yes No If Yes, please provide your email address 28.Would you like to volunteer to be a follow-up interview participant to receive a $20 Coles/Myer Gift Card? Yes No If Yes, please provide your email address If you have questions at any time about the evaluation or the procedures, you may contact the evaluator, Dr Jian Wang (jwang@csiro.au) or 02 6276 6174. Attachment B ALA impact survey questions for future use 1. Uptake, Usage and Adoption In your opinion, to what extent has there been uptake and usage of the data delivered through ALA by Research groups? a) Not at all b) A little bit c) Quite a lot d) Extensively to completely In your opinion, to what extent has there been uptake and usage of the tools, services and infrastructure of the ALA by Research Groups? a) Not at all b) A little bit c) Quite a lot d) Extensively to completely In your opinion, to what extent has there been uptake and usage of the data delivered through ALA by Industry? a) Not at all b) A little bit c) Quite a lot d) Extensively to completely In your opinion, to what extent has there been uptake and usage of the tools, services and infrastructure of the ALA by Industry? a) Not at all b) A little bit c) Quite a lot d) Extensively to completely In your opinion, to what extent has there been uptake and usage of the data delivered through ALA by Government? a) Not at all b) A little bit c) Quite a lot d) Extensively to completely In your opinion, to what extent has there been uptake and usage of the tools, services and infrastructure of the ALA by Government? a) Not at all b) A little bit c) Quite a lot d) Extensively to completely In your opinion, to what extent has there been uptake and usage of the data delivered through ALA by Community Groups? a) Not at all b) A little bit c) Quite a lot d) Extensively to completely In your opinion, to what extent has there been uptake and usage of the tools, services and infrastructure of the ALA by Community Groups? a) Not at all b) A little bit c) Quite a lot d) Extensively to completely In your opinion, to what extent has there been uptake and usage of the data delivered through ALA by other countries? a) Not at all b) A little bit c) Quite a lot d) Extensively to completely In your opinion, to what extent has there been uptake and usage of the tools, services and infrastructure of the ALA by other countries? a) Not at all b) A little bit c) Quite a lot d) Extensively to completely 2. Influence on Cultural Change To what extent has using the ALA changed your understanding, use and acceptance of the creative commons licencing framework? a) Not at all b) A little bit c) Quite a lot d) Extensively to completely In your opinion, to what extent do you think the embedding of standards in ALA systems has improved the fitness for purpose, consistency and accessibility of all online biological and ecological data? a) Not at all b) A little bit c) Quite a lot d) Extensively to completely In your opinion, to what extent has there been an improvement in the amount of “trusted” or quality data online as a consequence of the establishment of the ALA? a) Not at all b) A little bit c) Quite a lot d) Extensively to completely In your opinion, to what extent has ALA influenced others to ensure their own services/platforms are available as open infrastructure/architecture (eg software downloadable on GitHub; APIs all available for services/products of others)? a) Not at all b) A little bit c) Quite a lot d) Extensively to completely In your opinion, to what extent has the linking and integration of different data types through the ALA (e.g. through spatial portal) added value to information and services? a) Not at all b) A little bit c) Quite a lot d) Extensively to completely In your opinion, to what extent has ALA enabled “communities” or groups to connect, collaborate and self-organize in groups of interest to improve their own activities, avoid re-inventing the wheel, reduce duplication, provide critical mass, and attract new customers / clients / partners? a) Not at all b) A little bit c) Quite a lot d) Extensively to completely 3. New Products and Services To what extent has ALA assisted you in the production of reports and publications? a) Not at all b) A little bit c) Quite a lot d) Extensively to completely To what extent has ALA assisted you in building and improving skills in science, technology, engineering and mathematics (STEM) as well as interest and awareness of the STEM sector? a) Not at all b) A little bit c) Quite a lot d) Extensively to completely To what extent have you used ALA as part of a value proposition for a new business or service? a) Not at all b) A little bit c) Quite a lot d) Extensively to completely 4. Productivity and Efficiency In your opinion, to what extent has the ALA resulted in productivity and efficiency gains in information management and delivery of services? a) Not at all (0% productivity or efficiency gain) b) A little bit (> 10% productivity or efficiency gain) c) Quite a lot (> 40% productivity or efficiency gain) d) Extensively to completely (>80% productivity or efficiency gain) 5. Applications and Derivatives In your opinion, to what extent has the ALA helped improve regulatory compliance by Industry and Government with respect to environmental legislation? a) Not at all b) A little bit c) Quite a lot d) Extensively to completely In your opinion, to what extent has the ALA helped improve Government decision-making regarding investment of public funds and meeting its international biodiversity treaty obligations? a) Not at all b) A little bit c) Quite a lot d) Extensively to completely In your opinion, to what extent has the ALA helped in the application of research to key national and global challenges? a) Not at all b) A little bit c) Quite a lot d) Extensively to completely In your opinion, to what extent has the ALA helped the community (NRM bodies) to undertake on-ground interventions with respect to biodiversity (restoration, community, surveys etc – MERIT data)? a) Not at all b) A little bit c) Quite a lot d) Extensively to completely