September 2014 – June 2021 Final Evaluation Report Indigenous STEM Education Project Lisa Walker and Christopher Banks COPYRIGHT © Commonwealth Scientific and Industrial Research Organisation 2021. CITATION Walker, L. & Banks, C. (2021) Indigenous STEM Education Project Final Evaluation Report: September 2014 – June 2021. Canberra, Australia: CSIRO. Acknowledgements Acknowledgement of Country Aboriginal and/or Torres Strait Islander peoples have longstanding scientific knowledge traditions. These traditions have developed knowledge about the world through observation, using all the senses; through prediction and hypothesis; through testing (trial and error); and through making generalisations within specific contexts. These scientific methods have been practised and transmitted from one generation to the next and contribute to particular ways of knowing the world that are unique and complementary to Western scientific knowledge. This respect encompasses the recognition of Aboriginal and Torres Strait Islander contexts for technologies and concepts; their application in the past, present, and future, including supporting Science, Technology, Engineering, and Mathematics (STEM) career pathways for Aboriginal and/or Torres Strait Islander students; and reaffirming the ingenuity and creativity of Aboriginal and/or Torres Strait Islander peoples’ knowledge systems. A deep respect for these Aboriginal and/or Torres Strait Islander cultural practices and knowledge underpins the philosophy and practice of the Indigenous STEM Education Project. The Indigenous STEM Education Project team acknowledges the Traditional Owners of the lands with whom this project is collaborating, their vibrant living cultures and knowledge systems and acknowledge the Countries on which the evaluation work took place. We pay our respects to Elders past and present and thank all community members who provide the leadership to ensure meaningful and effective engagement with Aboriginal and Torres Strait Islander communities for the six distinct but complementary STEM education programs that make up this project. We specifically acknowledge the Traditional Owners of the lands on which the project operated and the evaluation case study research was conducted (see page 12). The Commonwealth Scientific and Industrial Research Organisation (CSIRO) acknowledges that Aboriginal and Torres Strait Islander peoples make extraordinary contributions to Australia in cultural, economic, and scientific domains; for example, incorporating Indigenous knowledges of ecological and social systems is vital to the achievement of sustainable development. Other acknowledgements CSIRO wishes to acknowledge the significant knowledge and leadership of Aboriginal and/or Torres Strait Islander scientists, educators, and program leaders that have made the development and implementation of the Indigenous STEM Education Project possible. The authors would like to thank the external peer reviewer, Dr Jessa Rogers from Macquarie University, whose insightful feedback improved the report significantly. We would also like to thank the late Professor Tom Cooper who retired from Queensland University of Technology in 2019 and was instrumental in the development of PRIME Futures. CSIRO acknowledges the contributions of former and current members of the evaluation team who supported the program monitoring and evaluation methodologies, data collection, and analyses, including Kate Cherry, Jessica Fidler, Dr Caja Gilbert, Karlie Noon, Dr Kirsten Sadler, and Dr Michael Tynan. We also acknowledge the contributions of and thank former and current members of the project teams who supported the program monitoring and evaluation methodologies, data collection, and analyses. We also thank CSIRO’s Indigenous Engagement Office, and particularly Louisa Warren, for her insightful advice and guidance. Finally, and most importantly, the students, teachers, teacher assistants, principals, Aboriginal and/or Torres Strait Islander Education Workers, parents/carers, heads of department, heads of curriculum, STEM professionals, and mentors that helped organise and took part in the evaluation research are gratefully acknowledged. The time that was given to the evaluation team, and the knowledge that was shared, made the evaluation possible. Contents Acknowledgements.................................................................................................................................3 List of figures...............................................................................................................................................4 List of tables................................................................................................................................................4 Acronyms......................................................................................................................................................6 Executive summary...................................................................................................................................7 Introduction.................................................................................................................................................9 Purpose of this evaluation report...................................................................................................................................13 Project theory of change and impact pathways.............................................................................................................13 Methodology..............................................................................................................................................15 Key messages....................................................................................................................................................................15 Research reflection and position....................................................................................................................................16 Results...........................................................................................................................................................17 To what extent did the individual programs achieve their intended outcomes?........................................................17 Key messages....................................................................................................................................................................17 ASSETS ....................................................................................................................................................................17 Bachelor of Science (extended).............................................................................................................................26 Indigenous STEM awards......................................................................................................................................33 I2S2 ....................................................................................................................................................................37 Science Pathways for Indigenous Communities..................................................................................................44 PRIME Futures.........................................................................................................................................................52 What are the whole-of-project outcomes and what elements have contributed towards achievement of these outcomes?...............................................................................................................................................................59 What key factors influence the sustainability of the Indigenous STEM Education Project?.......................................65 Adapting to COVID-19..........................................................................................................................69 Conclusion and Recommendations.................................................................................................71 References..................................................................................................................................................73 Appendix A: Context.............................................................................................................................77 Appendix B: Updated Impact Statements..................................................................................79 List of figures Figure 1. Individual programs within the Indigenous STEM Education Project.....................................................................7 Figure 2. Indigenous STEM Education Project – progress towards outcomes........................................................................7 Figure 3. Indigenous STEM Education Project participation numbers (includes participation up to September 2020).....9 Figure 4. Names of the Aboriginal and Torres Strait Islander language, social, or nation groups where participants lived and where participating schools were located..............................................................................................11 Figure 5. Indigenous STEM Education Project timeline..........................................................................................................12 Figure 6. Example of diagram showing progress towards outcome and anticipated timeframes.....................................16 Figure 7. Summer school locations, applications, and participants......................................................................................18 Figure 8. School locations of ASSETS summer school participants.......................................................................................19 Figure 9. ASSETS progress towards outcomes.........................................................................................................................20 Figure 10. Students attending the 2017 and 2018 summer schools increased their knowledge of prerequisite subjects.21 Figure 11. Home locations of Bachelor of Science (Extended) students.................................................................................26 Figure 12. Bachelor of Science (Extended) - progress towards outcomes...............................................................................28 Figure 13. Indigenous STEM Awards categories........................................................................................................................33 Figure 14. Home locations of Indigenous STEM Award recipients and finalists.....................................................................33 Figure 15. List of CSIRO initiatives survey respondents have been involved with..................................................................35 Figure 16. I2S2 program participation numbers.........................................................................................................................37 Figure 17. Locations of schools participating in I2S2.................................................................................................................37 Figure 18. I2S2 progress towards outcomes ..............................................................................................................................39 Figure 19. 2020 Snapshot of Science Pathways for Indigenous Communities participation numbers.................................44 Figure 20. Locations of schools that have participated in Science Pathways for Indigenous Communities........................44 Figure 21. Science Pathways for Indigenous Communities – progress towards outcomes ..................................................46 Figure 22. Example pages from the Malleefowl Science Inquiry Illustration of Practice Guide............................................51 Figure 23. End of program participation numbers, September 2019......................................................................................52 Figure 24. Locations of schools participating in PRIME Futures..............................................................................................52 Figure 25. PRIME Futures – progress towards outcomes..........................................................................................................54 Figure 26. Changes in teaching approaches: Teachers saw the biggest change in their use of RAMR and Big ideas.........55 Figure 27. Key components to support whole-of-school change............................................................................................57 Figure 28. Indigenous STEM Education Project – progress towards outcomes......................................................................60 Figure 29. Principles of good practice in remote learning.......................................................................................................69 List of tables Table 1. Summary of key findings and recommendations..........................................................................................................8 Table 2. Evaluation activities for each program.........................................................................................................................15 Table 3. Snapshot of ASSETS alumni engagement in broader STEM initiatives in 2019 and 2020.........................................24 Table 4. Bachelor of Science (Extended) enrolment figures......................................................................................................26 Table 5. Bachelor of Science (Extended) retention figures by cohort......................................................................................29 Table 6. Key elements contributing to the ongoing sustainability of the Project...................................................................65 Table 7. Program-level adaptations due to COVID-19................................................................................................................70 Table 8. Recommendations arising from the evaluation of the Indigenous STEM Education Project...................................71 Acronyms ABCAustralian Broadcasting CorporationACARAAustralian Curriculum, Assessment, and Reporting AuthorityAIEOAboriginal and Islander Education OfficerAboriginal Summer School for Excellence in Technology and Science – one of ASSETSthe Project’s six programsCEdOCSIRO Education and OutreachCSIROCommonwealth Scientific and Industrial Research OrganisationIndex of Community Socio-Educational Advantage - a scale that allows for fair and reasonable ICSEAcomparisons among schools with similar studentsI2S2Inquiry for Indigenous Science Students – one of the Project’s six programsNAPLANNational Assessment Program – Literacy and NumeracyPISAProgram for International Student AssessmentPRIME FuturesPurposeful Rich Indigenous Mathematics Education Futures – one of the Project’s six programsQUTQueensland University of TechnologyRAMRReality-Abstraction-Mathematics-ReflectionSTEMScience, Technology, Engineering, and MathematicsYDCYuMi Deadly Centre - a research centre at QUT that delivered the PRIME Futures programYDMYuMi Deadly Maths - a mathematics pedagogical framework Executive summary The Indigenous STEM Education Project was an ambitious project that aimed to increase interest and academic achievement among Aboriginal and/or Torres Strait Islander students in STEM subjects and related professions. The six programs which comprised the Project (Figure 1) influenced student outcomes in multiple ways: through direct program support; by better equipping teachers and schools to deliver culturally responsive STEM education; and by celebrating best practice and individual excellence. Throughout the six years of the Project, it has reached just under 24,000 Aboriginal and/or Torres Strait Islander students, 2,768 teachers and assistant teachers, and 603 schools. Evaluation evidence indicates that it has left a lasting impact on participating educators, students, and schools, and that the integration of Aboriginal and/or Torres Strait Islander knowledges and contexts through hands-on, inquiry-based lessons is beneficial for all students, regardless of their cultural backgrounds. Figure 2 summarises the progress against outcomes for the overall Indigenous STEM Education Project. Outcomes are listed in expected order of achievement, from short-term outcomes through to longer-term. This highlights that the Indigenous STEM Education Project is seeing transformative progress in short- term and some medium-term outcomes and effective/ emerging progress for longer-term outcomes. • Aboriginal Summer School for Excellence in in Technology and Science (ASSETS) • Bachelor of Science (Extended) • Indigenous STEM Awards • Inquiry for Indigenous Science Students (I2S2) • PRIME Futures • Science Pathways for Indigenous Communities Figure 1. Individual programs within the Indigenous STEM Education Project Figure 2. Indigenous STEM Education Project – progress towards outcomes Figure 2. Indigenous STEM Education Project – progress towards outcomes 8. Jurisdictions, CSIRO, universities & partners scaling up1. Innovative, place‐based Indigenous contextualised and/or led STEM curriculum (inquiry‐based) delivered in schools and university and documented in school plans5. Increased student aspiration, sense of value, cultural identity and school belonging6. Schools are culturally competent in delivering Two‐way science in partnership with Elders, families and communities flow on benefits to broader curriculum/ teaching 3. Increased parental, family & community engagement & recognition of role models 4. Increased student engagement, results and recognition 2. Increased teacher capacity in: inquiry; place‐based, hands‐on curriculum development; and delivery in an Indigenous context/ Indigenous led 7. More Indigenous (and non‐Indigenous) students pursuing STEM education pathways, enrolment in STEM years 10‐12 & university, STEM careers & leadership opportunities Table 1 groups the evaluation findings into five key themes and lists recommendations1 considered important for others looking to affect change in Aboriginal and/or Torres Strait Islander education or STEM education more broadly.2 Table 1. Summary of key findings and recommendations FINDINGS RECOMMENDATIONS TEACHER AND SCHOOL OUTCOMES Overall, the Project was successful in improving teacher confidence in, and capability to deliver, an innovative Aboriginal and Torres Strait Islander contextualised STEM curriculum. Through train-the-trainer models in PRIME Futures and professional development and support in I2S2 and Science Pathways for Indigenous Communities, the Indigenous STEM Education Project changed how thousands of students were taught STEM. There is less evidence of whole- of-school change, but the Project has shown that shifts are possible when teachers are given the time and resources to implement new approaches and are supported by invested school leadership. Where possible, programs should tailor professional learning activities to the appropriate level and if necessary, offer multiple levels rather than adopt a one-size-fits-all approach. Programs should prioritise face-to-face interactions and the creation of avenues for information sharing between individual teachers and school clusters. FAMILY AND COMMUNITY OUTCOMES The Project was able to support the development and strengthening of authentic relationships between many Aboriginal and/or Torres Strait Islander families and communities and local educators. These relationships were pivotal in helping teachers integrate local Aboriginal and Torres Strait Islander perspectives into the Australian curriculum as part of I2S2, PRIME Futures, and Science Pathways for Indigenous Communities. Some program staff and educators highlighted personnel turnover, both in schools and program teams, as presenting key challenges for continuity of relationships. Program teams need to support educators to liaise with local community groups and knowledge holders to develop joint educational goals that include community-driven indicators of success and to support the integration of place-based Aboriginal and Torres Strait Islander knowledges into the curriculum. Programs should more explicitly explore how they can contribute to teacher retention, for example through incentives and facilitating innovative and reflective practice. STUDENT OUTCOMES The Project has generally had a positive impact on student engagement, aspirations, results, and recognition. By supporting teachers in delivering hands- on, inquiry-based STEM that connects with students’ everyday experiences and existing knowledge, whole cohorts of students were exposed to a new way of learning STEM using Aboriginal and Torres Strait Islander contexts. Evaluations of PRIME and I2S2 highlighted that both programs contributed to increased academic results for low-achieving Aboriginal and/or Torres Strait Islander students, and low-achieving non-Indigenous students. Feedback from participants in ASSETS and the Indigenous STEM Awards highlights that students valued the multi-layered support offered and that their STEM ambitions were further shaped, influenced, and advanced by the many opportunities that resulted from participation in these programs. The ongoing relationships students established with program Alumni networks often provided valuable personal and professional connections for students. Across the programs, there was less evidence to assess some of the longer-term outcomes, including the impact of the Project on STEM career and education pathways. Programs should have in place formalised plans to develop and maintain alumni networks. The application of an inquiry-based, hands-on, culturally responsive pedagogy should be viewed as best practice regardless of students’ cultural backgrounds. BEST PRACTICE, CULTURALLY RESPONSIVE STEM EDUCATION OUTCOMES Overall, the Indigenous STEM Education Project has made progress promoting the uptake of best practice, culturally responsive STEM education pedagogies, although the results varied by program, school, and teacher. The Science Pathways for Indigenous Communities Program was successful in embedding Indigenous ecological knowledge within the Australian curriculum in a number of schools in the Northern Territory and Western Australia. The Program also drove the development of “Two-way Science: An integrated learning program for Aboriginal desert schools” (Deslandes et al., 2019) and a number of online resources to further spread the Two-way Science pedagogy. There is also evidence from the other programs of their effectiveness to introduce a culturally responsive STEM pedagogy. Programs aiming to change teaching pedagogies should engage with school leadership as well as individual teachers to embed new approaches within a school environment and ensure teachers are provided with time and resources to reflect on and refine their practice. Programs should adopt a long-term horizon (5+ years) for affecting change and work with funding partners to secure necessary resources over this time frame. STEM EDUCATION SECTOR OUTCOMES Overall, the Indigenous STEM Education Project was primarily focused on increasing participation numbers and continuing the operation of the program through seeking new funding opportunities, with less emphasis placed on driving more systems-level changes in jurisdictional/institutional practices and creating cultural shifts/embedding new knowledge broadly within the education sector. However, the Western Australian Government has been formally exploring a Two- way Science Initiative based predominantly on Science Pathways for Indigenous Communities, which if approved, will provide educators with resources and learning opportunities to further expand Two-way Science in government schools across the state. In addition, the Bachelor of Science (Extended) program continues to operate and strengthen its practice. Despite not all programs leading to systemic changes in the education sector, systemic changes were observed within CSIRO, and within individual schools and communities. Programs should have a strategic approach to systems-level change and engaging with key policy stakeholders and government departments. Evidence-based teacher professional learning programs should also include learning opportunities for pre-service teachers to assist in embedding new pedagogies as early as possible. 1 A synthesis of ‘what works’ in Indigenous STEM education based on the Indigenous STEM Education Project has been published (Banks, 2021). 2 Detailed recommendations on the six individual programs are in the respective case study evaluation reports: https://www.csiro.au/en/education/Programs/Indigenous-STEM-Education-Project/Monitoring-and-evaluation Introduction The Indigenous STEM Education Project was a partnership between CSIRO, Australia’s national research science agency, and the BHP Foundation, an independent charity established by BHP to support large, long-term community projects by not-for-profit organisations. The Project operated from 2014-2021, although elements of the Project will continue to be delivered from 2021 onwards, such as CSIRO providing online resources to teachers and the University of Melbourne’s Bachelor of Science (Extended). A timeline of the Project is presented in Figure 5. The Project’s overarching goal was to provide supported pathways that improve the participation and achievement of Aboriginal and/or Torres Strait Islander students in STEM subjects and careers. The Project comprised six programs covering a range of beneficiaries and geographic areas; Appendix A contains a discussion of the context in which the programs were developed and delivered. Three of these were universal programs: Inquiry for Indigenous Science Students (I 2S2) and PRIME Futures, which were science inquiry and maths programs implemented in metropolitan and regional communities, and Science Pathways for Indigenous Communities, which used Indigenous Ecological Knowledges as the basis for teaching science in remote communities. Three of the programs were targeted: the Aboriginal Summer School for Excellence in Technology and Science (ASSETS) and the Indigenous STEM Awards, which supported, celebrated, and extended high-achieving Aboriginal and/ or Torres Strait Islander students and STEM champions, and the Bachelor of Science (Extended) at the University of Melbourne, which provides an alternate pathway to a university science degree for students who may not have undertaken the required science and/or mathematics subjects. The Indigenous STEM Education Project was a broad and ground-breaking initiative, one that has connected with teachers, students, and communities across the country. As highlighted in Figure 3, the Indigenous STEM Education Project connected with 23,904 students and 2,768 educators3 across 603 schools.4 While the six programs that comprised the Indigenous STEM Education Project were established as independent programs, there were a common set of underlying principles, informed by best practice and co-design and feedback from cultural knowledge holders. These principles guided program activities and informed how change was affected and included: • Strengths-based – focusing on the capacity, skills, and knowledge of the program, people, and communities, while acknowledging barriers. • Built on high expectations – setting high expectations for Aboriginal and/or Torres Strait Islander students. • Promoting inquiry-based STEM pedagogies5 - teaching based on learning strategies involving student-centred research and investigation. • Privileging Aboriginal and Torres Strait Islander culture and place – ensuring Aboriginal and Torres Strait Islander knowledges, voices, and experiences are at the forefront of program design and delivery. • Ensuring flexible, culturally responsive program development and delivery. Figure 3. Indigenous STEM Education Project participation numbers (includes participation up to September 2020) 3 The term ‘educators’ is used interchangeably with ‘teachers’ in this report to be inclusive of positions involved in delivering, supporting, and leading STEM education, including participating teachers, teaching assistants, Aboriginal and Torres Strait Islander Education Workers, Heads of Curriculum/Department, and Principals. 4 These figures involve some estimation of the percentage of Aboriginal and/or Torres Strait Islander students at some schools, and do not include participation for Term 4, 2020. Figure 3. Indigenous STEM Education Project participation numbers (includes participation up to September 2020) Figure 3. Indigenous STEM Education Project participation numbers (includes participation up to September 2020) Figure 3. Indigenous STEM Education Project participation numbers (includes participation up to September 2020) 23,906 Students 2,769 Educators 603 Schools People from all states and territories have participated in the Indigenous STEM Education Project, including students and teachers from small remote primary schools, large urban high schools, and tertiary institutions. As well as geographic differences, Aboriginal and/or Torres Strait Islander participants have their own unique cultures, customs, scientific traditions, languages, and laws (Australian Institute of Aboriginal and Torres Strait Islander Studies (AIATSIS), 2021). This diversity has been a particular source of strength for the Indigenous STEM Education Project, allowing the Project to be enriched by local knowledges and perspectives. In an effort to showcase this diversity, Figure 4 lists the names of the Aboriginal and Torres Strait Islander language, social, or nation groups where participants were from (Bachelor of Science (Extended) and Indigenous STEM Awards), went to school (ASSETS) and where participating schools were located (PRIME Futures, Science Pathways for Indigenous Communities, I2S2). Figure 4 has been developed primarily using the AIATSIS Map of Indigenous Australia (AIATSIS, 2021). In compiling this list, it is recognised that there are a number of shortcomings including that: the source map lists only the larger Aboriginal and Torres Strait Islander groups across Australia; there are often variations in the names of groups; and the boundaries of groups overlap and are not fixed (AIATSIS, 2021). We also acknowledge that many participants have cultural connections that extend beyond their home location listed at the time of participation and that colonisation has displaced many peoples from their traditional lands. Notwithstanding these shortcomings, it is the intention that the list serves to recognise and highlight the diversity of the many individuals that have participated in the Indigenous STEM Education Project. 5 Inquiry-based STEM pedagogies are the process by which students learn through their active learning (thinking) and participation (doing) related to the problem, concept, or phenomenon. Science Inquiry Skills is a key strand (area) of the national science curriculum, alongside Science Understanding and Science as a Human Endeavour. Figure 4. Names of the Aboriginal and Torres Strait Islander language, social, or nation groups where participants lived and where participating schools were located * Māori tribe Amangu Amangy Arrernte Awabakal Badtjala Banggarla Banjima Baraba Barkindji Barna Barranbinya Barunggam Bayali Bibbulman Bidjara Bigambul Biri Biripi Boonwurrung Buandig Bundjalung Dainggatti Danggali Darkinung Darumbal Dharug Djabuganjdji Djabwurung Djirbalngan Doolboong/ Miriwoong Eora Gabalbara Gangulu Gayiri Geawegal Giraiwurung Giya Goreng Gubbi Gugu-Badhun Gumbainggir Gundungurra Gunggandji Gunibidji Gureng Guugu-Yimidhirr Iningai Jaburrara Jawoyn Jukun Kalaako/Malpa Kalaamaya Kalaw Ya Kamilaroi Kaniyang Kariyarra Kartan Kaurna Pangkarra Kokatha Kuku-yalanji Kunjen Kurarra Kureinji Kuring-gai Kuthant Kuwarra Larrakia Latje Luritja Mayi-Kulan Meru Minang Muralag Narangga Nawu Ngadjuri Ngāti Awa* Nganyaywana Ngarabal Ngarigo Ngarluma Ngarrindjeri Ngatatjara Ngāti Awa Ngunawal Ngurraiillam Nhanta Nhantu Nhuwala Nuenonne Nukunu Nyamal Nyanganyatjara Nyawaygi Palawa Palyku Pangkarra Paredarerme Peerapper Peramangk Pinjarup Pintupi Quandamooka Taungurung Tharawal Thul Garrie Waja (Bindal) Tiwi Tjalkanti Tjupany Tommeginne Turrbal Tyerrernotepanner Wadi Wadjigu Wailwan Wajuk Waka Walmatjarri Wangkathaa Wardandi Wargamaygan Warnindilyakwa Wathaurong Watjarri Waveroo Wawula Wemba Wiljali Wiradjuri Wirangu Wongaibon Wonnarua Worimi Wudjari Wulgurukaba Wunumara Wurundjeri Yadhaigana Yidinjdji Yinggarda Yolngu Yugambeh Yuggera Yuin Yuwi Figure 5. Indigenous STEM Education Project timeline Figure 5. Indigenous STEM Education Project timeline 2014First Indigenous STEM Awards presented I2S2 expanded operations to include SA Seven PRIME Futures presentations delivered at the YuMi Deadly Centre Sharing Summit held in October 2017. Launch of online ACARA Illustrations of Practice Videos showcasing Two-way Science approach Publication of Two-way Science book - An Integrated Learning Program for Aboriginal Desert Schools Science Pathways for Indigenous Communities shifted to a cluster model to encourage collaboration and leadership between schools PRIME Futures held two-day YuMi Deadly Centre Sharing SummitProject team members co-presented with Indigenous program participants at The National Aboriginal & Torres Strait Islander Education Conference First Bachelor of Science (Ext) student completed their degree201520162017PRIME Futures commenced in 15 schools First cohort of BSC (Ext) students completed year 1 I2S2 piloted in 14 schools in QLD and NSW ASSETS summer schools extended to Townsville and NewcastleI2S2 expanded to 45 schools in QLD, NSW and WA PRIME Futures delivered to 43 schools with 165 teaching staff trained Science Pathways for Indigenous Communities operating in 3 schools in WA and 3 in NT Cumulative total of 17 students enrolled in Bachelor of Science (Ext) program I2S2 commenced Online Pilot Program I2S2 Sharing SymposiumPRIME Futures team members presented at International STEM in Education Conference, Brisbane Indigenous STEM Awards held first pre-announcement winners gathering in Sydney ASSETS Adelaide summer camp featured on Radio National, 499 students in Alumni network Science Pathways for Indigenous Communities operating in 15 schools with 971 students Al programs pivoted to virtual program delivery models due to impact of COVID-19 I2S2 launched Online Learning Program Western Australian investigating Two-way Science Initiative based on Science Pathways for Indigenous CommunitiesI2S2 launched a new online learning module 'Teaching Mathematics' Science Pathways For Indigenous Communities launches six-part online learning experienceSecond Evaluation Report First Evaluation Report Third Evaluation Report ASSETS Case Study ReportASSETS Destination ReportPRIME Case Study ReportI2S2 Case Study ReportIndigenous STEM Awards Stories of Change Report Final Evaluation ReportYumi Deadly Maths program joined ISEP as partof PRIME FuturesASSETS joined ISEPand held SummerSchool in AdelaideScience Pathways for IndigenousCommunities based on TangentyereCouncil's Land and Learning programcommenced operating in three communitiesin the NT2018201920202021 Purpose of this evaluation report This report is the last in a series of four overall evaluation reports for the Indigenous STEM Project. The first, second, and third evaluation reports examined program implementation and assessed short-term outcomes (Cherry, Banks, Mudhan, & McNeilly, 2019; Ma Rhea et al., 2018; Tynan & Noon, 2017). This report evaluates the success of each of the six programs that make up the Indigenous STEM Education Project by looking at the achievement of short-term, intermediate, and long-term program outcomes. It also takes a whole- of-project focus, examining the overall impact, key strengths, and important learnings. Finally, the report assesses the Project’s enduring benefits and the sustainability of outcomes. It also provides an update on the individual program responses to the disruption caused by the COVID-19 pandemic. The Indigenous STEM Education Project is unique in its reach and scope, and therefore, the evaluation lessons are expected to be important to researchers and practitioners interested in STEM education and Aboriginal and/or Torres Strait Islander education. This evaluation report, and the three reports that preceded it, contribute to the growing body of literature on what works in Aboriginal and/or Torres Strait Islander education and particularly STEM education. As recognised in the Productivity Commission’s Indigenous Evaluation Strategy, too often, evaluations of programs targeting Aboriginal and/ or Torres Strait Islander peoples occur late in project timelines and are not properly integrated into the policy and program development cycle (Productivity Commission, 2020). The usefulness of evaluations is also limited by a focus on accountability more than finding ways to improve outcomes for Aboriginal and/or Torres Strait Islander peoples. Monitoring and evaluation have been central components of the Indigenous STEM Education Project from its inception. Consistent with recognised best practice, there has also been a focus on listening to the voices and experiences of Aboriginal and/or Torres Strait Islander peoples and integrating key learnings back into each of the individual programs. The key evaluation questions are: To what extent did the individual programs achieve their intended outcomes? What were the whole-of-project outcomes and the common elements that contributed towards their achievement? What key factors influenced the sustainability of the Indigenous STEM Education Project? As the project progressed, the focus of the evaluation reports shifted from understanding the effectiveness of initial program implementation (first report), assessing achievement of early outcomes (second and third reports), to looking at longer-term outcomes and the Project’s lasting impact (this report). The various authors of these evaluation reports have been able to build on learnings and feedback from earlier reports to adjust and refine each iteration. This aligns with the Project’s commitment to flexibility and action learning, and is why the reports have evolved and do not follow identical structures. Project theory of change and impact pathways The overall Indigenous STEM Education Project, and the six programs that comprise it, all have their own impact statements. These statements, based on CSIRO’s Impact Model, describe the logic and assumptions of each program and articulate expected outputs, outcomes, and longer-term impacts. The Impact Statements are available at https://www. csiro.au/en/education/Programs/Indigenous-STEM- Education-Project/Monitoring-and-evaluation. The Indigenous STEM Education Project was based on the initial hypothesis that the delivery of innovative programs will lead to improved engagement, attendance, and academic achievement among Aboriginal and/ or Torres Strait Islander students in STEM. Three of the programs (I2S2, PRIME Futures, Science Pathways for Indigenous Communities) focused on delivering culturally responsive, hands-on, inquiry-based6 STEM pedagogies. The three other programs, ASSETS, Bachelor of Science (Extended), and the Indigenous STEM Awards, looked to provide opportunities, recognition, and/or mentoring for students and others. The Indigenous STEM Education Project’s impact statement and the impact statements of the six individual programs were developed in 2017 and have been updated again during the development of this final report. During this review, a number of additional outcomes were identified by program staff and stakeholders, and have been included. Some outputs/ outcomes were also re-categorised or updated. Due to limited data, this report is not able to assess the program’s performance against all outcomes, particularly longer-term ones. See Appendix B for the updated impact statements. 6 Inquiry-based learning is a pedagogical approach to teaching STEM that emphasises investigation and problem solving. As outlined by the Department of Education, Skills and Employment (2020a) “inquiry-based learning prioritises problems that require critical and creative thinking so students can develop their abilities to ask questions, design investigations, interpret evidence, form explanations and arguments, and communicate findings.” As part of the Lowitja Institute’s work to develop an evaluation framework to guide evaluations involving Aboriginal and/or Torres Strait Islander peoples, one of the key criterion for ethical evaluations is ‘allow[ing] stakeholder needs and perspectives to shape program and evaluation development”; this, in turn, leads to “evaluation results that are more relevant and consistent with stakeholder priorities” (Kelaher et al., 2018, p. 42). This is also in line with the responsibilities under the AIATSIS Code of Ethics for Aboriginal and Torres Strait Islander research, which specifies that research “should respond to priorities determined by Aboriginal and Torres Strait Islander peoples and have key objectives that demonstrate intended beneficial impacts and outcomes, either at a local level or more broadly” (AIATSIS, 2020, p. 18). These key principles are central to the Indigenous STEM Education Project’s monitoring and evaluation activities. The process of using evaluation findings to revise impact statements and adjust program delivery, where appropriate, follows an action learning approach similar to the cycle of “reflecting, planning, acting, data collecting, analysing, reflecting again, re-planning” (Aubusson, Brady, & Dinham, 2005, p. 2). This cycle of learning and growing helps integrate practice learnings into future programs, improve program delivery, and inform future actions. As part of this process, evaluators and Program teams worked to ensure Program activities and their underlying Impact Statements were informed by participant experiences and Aboriginal and Torres Strait Islander understandings and world views. As the project matured, so did the views of what constitutes success. A particular focus has been on creating space to hear from different stakeholders, communities, and participants and understand what success means to them. Consequently, the focus has shifted from student attendance and academic success to following individual students, teachers, and communities along their STEM learning/career journeys and providing culturally responsive opportunities to further their STEM goals. Methodology Key messages • Evaluation activities for the six programs that comprise the Indigenous STEM Education Project were guided by a set of key principles and intentional decisions to centre the voices and experiences of Aboriginal and/or Torres Strait Islander participants. • Detailed case study evaluation reports have been produced for ASSETS, Bachelor of Science (Extended), I2S2, Indigenous STEM Awards, PRIME Futures, and Science Pathways for Indigenous Communities. These reports adopt a multi-methods approach incorporating some of the following elements: surveys, semi- structured interviews, focus groups, reflective journals, and assessments of jurisdictional and program data. • This final evaluation report assesses the evidence for the achievement of outcomes using a strengths-based scale that assesses whether the achievement of the outcome was transformative, effective, or emerging. Program monitoring and evaluation processes have been embedded across all the programs of the Indigenous STEM Education Project. While these processes are tailored to the unique situations of each program being evaluated, there are several key principles and intentional decisions that have guided the collection of evaluation data across the Project. There has intentionally been limited direct monitoring of student performance; instead, there has been a focus on qualitative data collected through participant/stakeholder surveys and case studies. This allows for the voices of Aboriginal and/or Torres Islander peoples to not only be heard but centred throughout the evaluation (Muir & Dean, 2017). Where appropriate data were available, standardised jurisdictional or national- level data were also used as part of a mixed-methods approach to supplement the qualitative information. Detailed case study reports have been produced for each program and are available here: https://www.csiro.au/ en/education/Programs/Indigenous-STEM-Education- Project/Monitoring-and-evaluation. The core of the case study framework is the triangulation of data from the perspectives of key stakeholders in the relevant program elements: students, teaching staff, and program/support staff. Rather than duplicate the case study results here, this report summarises the key case study findings and aligns these results to the individual program outcomes. Where available, additional qualitative and quantitative data that was not available at the time that the case study reports were published will also be included in this analysis. Table 2 provides a summary of the key evaluation activities undertaken for each program. All evaluation activities were approved by CSIRO’s Social and Interdisciplinary Science Human Research Ethics Committee in accordance with the National Statement on Ethical Conduct in Human Research 2007 (updated 2018). All participants (including parents/carers for those under 18) were provided with participant information sheets and gave their free, prior, and informed consent to participate. More details on the specific methodologies used to assess key performance indicators are listed under each program in the Results section of this report. Table 2. Evaluation activities for each program ASSETS BACHELOR OF SCIENCE (EXTENDED) I2S2 INDIGENOUS STEM AWARDS PRIME FUTURES SCIENCE PATHWAYS FOR INDIGENOUS COMMUNITIES Surveys* Program staff reports Semi-structured interviews**/yarns Focus groups** Jurisdictional/administrative data Teacher assessments Reflective journals Post-event/professional development questionnaire Sharing symposium *Includes student, teacher, and principal surveys and pre/post surveys, biannual surveys, and exit surveys. **Semi-structured interviews and focus groups were conducted by members of the CSIRO evaluation team. This report examines the evidence that outcomes have been achieved using the following strength-based scale: Transformative – The outcome was fully achieved, and there was evidence of substantial, widespread positive shifts. Effective – The outcome was mostly or fully achieved, and there was evidence of positive shifts. Emerging – The outcome was showing early signs of achievement, and there was evidence of positive shifts beginning to be developed. The whole-of-program outcomes are presented in a circle format that also highlights the expected timeframe for the achievement of each outcome. Short-term outcomes are listed in the upper left quadrant of the figure; as you move around the circle in an anti-clockwise rotation, the timeframe for the expected achievement of the outcome increases. See Figure 6 for an example. Figure 6. Example of diagram showing progress towards outcome and anticipated timeframes Research reflection and position The CSIRO evaluation team consisted of a number of researchers during the six-year period of planning, data collection, analysis, and report writing. Several team members were Aboriginal, including a Wiradjuri woman and a Gamilaraay woman, and the remaining evaluation team members were non-Indigenous. Non-Indigenous team member world views, while varying, through enculturation, were largely based on Western modes of theoretical knowledge (Dew, McEntyre, & Vaughan, 2019). Aboriginal team members’ voices were frequently given prominence to counter these modes of knowledge, as were the voices of many evaluation participants. Through the evaluation of the Indigenous STEM Education Project, the evaluation team had the opportunity to influence and be influenced by the research through processes of reflexivity and development (Attia & Edge, 2017). The evaluation team’s development and methodology were informed by the writings of Aboriginal and/or Torres Strait Islander researchers and their colleagues (for example, Hogarth, 2017; Jackson-Barrett, Price, Stomski, & Walker, 2015; Martin & Mirraboopa, 2003; Nakata, 2002, 2007; Rigney, 2006; Yunkaporta, 2009) and conversations with Aboriginal and/or Torres Strait Islander CSIRO colleagues with expertise in engaging with Aboriginal and/or Torres Strait Islander peoples and communities. These perspectives informed the development of a general approach that the evaluation team aimed to apply throughout the project, including recognising the strengths, complexity, and researcher positions; recognising researchers as learners; and a focus on utilisation and contributing to positive change. The time spent with program participants, often in schools and on Country, provided the evaluation team with the opportunity, to a limited degree, to better understand the environments in which the project was being delivered. Figure 6. Example of diagram showing progress towards outcome and anticipated timeframes Short‐term outcomesLonger‐term outcomes ACHIEVEMENT OF PROGRAM OUTCOMES Results To what extent did the individual programs achieve their intended outcomes? Key messages • The evaluation showed that the six programs that comprise the Indigenous STEM Education Project were effective in meeting their desired outcomes. All programs had one or more transformative outcomes, resulting in substantial, widespread positive impacts for participants. • I2S2, PRIME Futures, and Science Pathways for Indigenous Communities have successfully supported teachers to deliver hands-on, inquiry-based STEM lessons that integrate Aboriginal and Torres Strait Islander knowledges and are aligned to the Australian science/mathematics curriculum. • ASSETS, Bachelor of Science (Extended), and the Indigenous STEM Awards have guided, mentored, and/or recognised STEM students (and others), and connected them with further opportunities to explore and achieve their STEM education and career goals. • Across the six programs, staff have worked to provide multi-layered, individualised development opportunities to students and educators that have had transformative impacts in the classroom and for individual participants. • Of all six programs, Science Pathways for Indigenous Communities has had the biggest impact on developing systems-level change through the Two-way Science Initiative currently being investigated by the Western Australia Department of Education. • Programs were most effective at supporting teachers to incorporate new pedagogies when school leadership was also invested in whole-of-school change. As well as detailing program performance against intended outcomes, this section also highlights some of the individual stories of students, teachers, and schools that have participated in one or more of the programs within the Indigenous STEM Education Project. These stories provide a small sample of the impact the Project has had on individual participants and are symbolic of the hundreds of unique stories that could be shared. ASSETS Program description The Aboriginal Summer School for Excellence in Technology and Science (ASSETS) program provided an opportunity for high-achieving Year 10 Aboriginal and/ or Torres Strait Islander students, with an interest and aspiration in STEM, to explore the study and career options available to them. The ASSETS program has been run intermittently in Adelaide from 1992 to 2013 with a number of funding partners. The program was incorporated into the Indigenous STEM Education Project in 2014. The ASSETS program had three primary components 1. an intensive nine-day residential summer school, 2. an ongoing, tailored leadership program, and 3. an integrated and overarching cultural program. The central component of the program was a culturally, socially, and geographically responsive7 nine-day residential summer school, which focused on three key areas: • Cultural leadership and inclusion of Aboriginal and Torres Strait Islander knowledges overseen by a Program Patron who was a local leader in Aboriginal and/or Torres Strait Islander education and was supported by a member of the Indigenous support unit of the host university and two or more additional mentors. • Academic development with a focus on the scientific inquiry process and practical team- based inquiry projects supported by academics and knowledge holders from the local area. • Leadership and personal development activities that helped students develop education plans and increase awareness of STEM careers and pathways. Following the summer school, students were invited to networking events with STEM professionals and were also able to access STEM-related work placement programs facilitated by ASSETS program staff. ASSETS program staff also used online forums to share news and ideas and connect students to additional STEM opportunities. Students, their families, and teachers were provided with ongoing, personalised assistance through Years 11 and 12 and into tertiary education. During the summer school, students were provided with opportunities to explore, share, and strengthen their connections to Aboriginal and Torres Strait Islander cultures8. As outlined in the ASSETS evaluation case study report, the ASSETS program was unique in its explicit focus on exploring cultural identity and Aboriginal and Torres Strait Islander scientific knowledges. It adopted a Two-way Science inquiry learning pedagogy, which involved connecting Aboriginal and Torres Strait Islander scientific knowledges and cultures with mainstream Western STEM literacy and careers. As illustrated in Figure 7, the program held its first summer school in Adelaide as part of the Indigenous STEM Education Project in 2014. The program was extended to Townsville and Newcastle in the 2015-16 summer holidays and three more summer schools were held in these locations over the following years. Due to funding, the summer school was only held in Adelaide and Newcastle during the 2019-20 summer holidays. The program switched to a virtual work placement model in 2020 due to the COVID-19 pandemic. Figure 7. Summer school locations, applications, and participants Figure 7. Summer school locations, applications, and participants 2014-2015 2015-2016 2016-2017 2017-2018 Host city: Adelaide 30 Applicants 28 Participants Host cities: Townsville, Newcastle, Adelaide 119 Applicants 98 Participants Host cities: Townsville, Newcastle, Adelaide 175 Applicants 101 Participants Host cities: Townsville, Newcastle, Adelaide 118 Applicants 104 Participants 742 Total applicants 499 Total participants 67% of applicants accepted 2018-19 2019-20 Host cities: Townsville, Newcastle, Adelaide 194 Applicants 99 Participants Host cities: Newcastle, Adelaide 106 Applicants 69 Participants 2019-2020 2018-2019 2017-2018 2016-2017 2015-2016 2014-2015 7 Each summer school had a unique theme, for example focusing on marine research and science. 8 There was substantial variation in the pre-existing cultural connectedness of ASSETS participants, ranging from emerging to strong. Key achievements • 499 students participated in the summer schools and 93 work placements were undertaken as part of the ASSETS program. As highlighted in Figure 8, ASSETS summer school participants came from schools across Australia. • The program helped to increase participants’ awareness of and aspirations for STEM education and career pathways. • The summer schools provided an inclusive and safe environment for students to explore, share, and strengthen their connection to culture and learn more about connections between Aboriginal and Torres Strait Islander scientific knowledges and its connections to Western STEM. • The ASSETS program created lasting connections between students and widened students’ professional networks. Measuring success The Impact Statement in Appendix B provides an overview of the key outcome pathways for the ASSETS program. Due to limited data, this report is not able to assess the program’s performance against all outcomes. Where appropriate, outcomes have also been grouped together for reporting purposes. The key outcomes were: Short-term • Students have a better understanding of and confidence pursuing STEM career pathways with subject choice referencing prerequisites for university STEM courses. • Students are more aware of the relevance of Aboriginal and/or Torres Strait Islander scientific knowledges for STEM and are able to explore, share, and strengthen connections to culture. • An ongoing channel for parents/carers , communities, and schools seeking further advice and/or opportunities. • Growth in student and professional networks. Intermediate • High aspiration for students with a focus on STEM careers. • Participation in broader STEM initiatives – e.g., work placements, Awards program, CREST, BHP Science Awards, university STEM programs. • Schools, jurisdictions, stakeholders valuing summer school and leadership program resulting in increased demand. Long-term • Continued success in STEM subjects in Years 11-12, particularly direct university entry from ATAR. • Students connected with post-secondary internships, cadetships, scholarships, and other unique opportunities. • Cohorts of role models, both mentors and students, contributing to Two-way science and championing Indigenous knowledges. • Active, engaged, skilled, and growing alumni community and professional network. • Identification of future funding partners. Evaluation methods Progress against outcomes was assessed using a variety of measures. In 2019, a detailed Case Study evaluation was undertaken (Banks, Mudhan, & Fidler, 2019). Supporting this Case Study, a Destination Report was also completed in 2020 (Banks & ASSETS Alumni, 2020). The Case Study utilised a multi-methods approach incorporating the following methods: • Semi-structured interviews with students, program staff, and program partners who attended the January 2017 summer school in Newcastle. • Online surveys used as part of the ongoing program monitoring, which included pre- and post-summer school participant surveys, surveys of parents/carers of participants, and destination surveys for formers participants. Figure 8. School locations of ASSETS summer school participants Figure 8. School locations of ASSETS summer school participants The Destination Report utilised a qualitative methodology consisting of semi-structured interviews with 10 past ASSETS participants. The interviews were designed to understand the impact, if any, that ASSETS had on alumni over a period of 5 to 6 years and to find out about their STEM journeys, decision-making, and challenges. Key findings of the ASSETS program Figure 9 provides a summary of progress towards outcomes for the ASSETS program. New or updated outcomes, and outcomes with no available data, have been excluded from this analysis. Figure 9. ASSETS progress towards outcomes Students have a better understanding of and confidence pursuing STEM career pathways with subject choice referencing prerequisites for university STEM courses Level of evidence: high High aspiration for students with a focus on STEM careers Level of evidence: high A key goal of ASSETS was to support students’ existing aspirations and foster a passion for STEM. The program also worked to ensure students had a clear knowledge of the pathway to pursue further education and eventual employment in a STEM field, and to provide a safe environment to explore connections to culture. Qualitative information in the ASSETS Case Study and Destination Reports showed that the program had a transformative impact on students’ understanding and confidence in pursuing a STEM career, including improving students’ understanding of Year 11 and 12 prerequisite subjects. ASSETS was also effective in supporting students’ high aspirations for STEM careers. As reported in the ASSETS Case Study and highlighted in Figure 10, survey data showed increases in the proportion of students understanding what a prerequisite subject was and what prerequisites were required for their desired university course. For example, students attending the 2017 and 2018 summer schools were asked if they knew the prerequisite subjects for what they wanted to study at university. The results showed that the proportion of students who agreed or strongly agreed that they knew their required prerequisite subjects increased by 36 per cent (2017) and 37 per cent (2018) after attending a summer school. Figure 9. ASSETS progress towards outcomes 1. Students have a better understanding of and confidence pursuing STEM career pathways with subject choice referencing prerequisites for university STEM courses5. Participation in broader STEM initiatives –e.g. work placements, Awards program, CREST, BHP Science Awards7. Continued success in STEM subjects in Years 11‐12, particularly direct university entry from ATAR3. Growth in student and professional networks 2. Students are more aware of the relevance of Aboriginal and Torres Strait Islander scientific knowledges for STEM and are able to explore, share, and strengthen connections to culture6. Schools, jurisdictions, stakeholders valuing summer school and leadership program, resulting in greater demand4. High aspiration for students with a focus on STEM careers9. Cohorts of role models, both mentors and students, contributing to Two‐way science and championing Indigenous knowledges8. Active, engaged, skilled, and growing alumni community and professional network Figure 10. Students attending the 2017 and 2018 summer schools increased their knowledge of prerequisite subjects Source: Banks et al. (2019). Conversations with the program team highlighted that many students entered the program with high career aspirations and that the goal of the program was to encourage and deepen students’ interest in STEM. Evaluation findings indicated that student confidence increased and students’ understanding and knowledge of STEM education and career options were broadened. Contributing factors included: • Access to tailored and responsive content delivered by STEM professionals. • The program’s emphasis on developing broader skills (e.g., teamwork). • Increases in student confidence levels due to feeling more prepared and informed about future options. • Work placement program and contact with STEM professionals contributed to support and bolstered students’ career aspirations with placements in STEM organisations helping students to get a better understanding of what is involved in a STEM career. As a result of attending ASSETS, I have found a new career I am interested in doing. Therefore, I have changed my subjects to have the prerequisites to get into that field. This program has given me insight into a career I never knew existed. It influenced me to change my subjects to best suit that career, so I have the prerequisites to get in. [Work placement had] helped me to better understand the environment of a workplace and opened my mind to other career options. I have higher social confidence and a better knowledge in STEM careers. ASSETS has presented me with a better perspective of what I want to do in the future, and made me rethink what uni[versity] courses I wanted to take. Figure 10. Students attending the 2017 and 2018 summer schools increased their knowledge of prerequisite subjects Continued success in STEM subjects in Years 11-12, Level of evidence: particularly direct university lowentry from ATAR Per cent of students who agree or strongly agree As already discussed above, by nature of the program, students selected to participate in ASSETS were high achieving. According to program staff, these students already had good results in science and mathematics prior to joining the program. Improving student results was not a core focus of the program; rather, the program looked to connect students with STEM professionals and mentors and develop broader life skills and connections to culture. The ASSETS Case Study presented the results of an analysis of student grades and highlighted that the program was effective in supporting students’ continued success in STEM subjects in Years 11 and 12. Students were asked to self-report grades in STEM subjects for Year 11. Data showed that 98 per cent of students scored in the highest three assessment bands9 in Science, 94 per cent in Maths, and 75 per cent in Technology and Engineering (n = 100). Further analysis compared Year 11 and 12 alumni grades with Year 10 grades. The results for Year 12 students indicated that the number of ‘A’ and ‘B’ grades increased from 60 per cent to 66 per cent (n = 11). The results for Year 11 showed an opposite trend with the number of ‘A’ and ‘B’ grades decreasing from 81 per cent to 65 per cent (n = 14). As recognised in the ASSETS Case Study, these results may be influenced by the small sample size and differences between the two cohorts. The results from two destination surveys conducted in June 2017 with participants in the 2014-15 and 2015-16 summer schools indicated that 97 per cent of respondents completed Year 12 and 67 per cent were attending university or TAFE. This result is perhaps not surprising given that ASSETS was aimed at high-achieving students; nevertheless, and as highlighted in the Case Study, it is positive that participants were exceeding national statistics on high school retention and university attendance. ASSETS changed the way I worked at school, it made me work harder to reach my goals. Thanks to the mentors and their stories of how they got where they are, I have developed more determination to work towards university and complete school to the best of my ability. The ASSETS program was designed to help students explore their connection to culture and to share and engage with the local cultures at the summer school location. The program also aimed to increase students’ awareness of the contributions that Aboriginal and/or Torres Strait Islander peoples have and continue to make in STEM. Evidence in the Case Study and Destination Report highlighted that the ASSETS program had a transformative effect on many students and changed the way many students felt about their cultural identities. The program staff recognised that students participating in the ASSETS program had varying levels of cultural connections, and program staff worked to create a safe and responsive space at each summer school where participants could engage in culture and explore Aboriginal and Torres Strait Islander STEM knowledges. Feedback from students focused on a number of key themes including: • The uniqueness of the summer school program and particularly the integration of culture and STEM knowledge, which was often not a part of students’ regular schooling experiences. • The enjoyment felt while participating in cultural activities, often for the first time, and exploring more about Aboriginal and Torres Strait Islander cultures in a safe and open environment. • Increased recognition of the application of Aboriginal and Torres Strait Islander scientific knowledges. • Interest in deepening engagement with the home communities, including using STEM projects to bring benefits to their communities. • A small number of students indicated that they would have preferred a more comprehensive cultural program. There was also some evidence that students were interested in using the knowledge gained at ASSETS to benefit their local communities. ASSETS gave me a great opportunity to further connect with my culture, as well as learning interesting scientific concepts/theories. I haven’t been exposed to that side of my identity really much at all before this. My family’s small and they sort of don’t really have…a super cultural family. So having that experience was just incredible and it really made me value my culture and sort of want to learn more about it and become more immersed in it and everything like that. Students are more aware of the relevance of Aboriginal and/or Torres Strait Islander scientific Level of knowledges for STEM and are able evidence: highto explore, share, and strengthen connections to culture 9 The top three assessment bands are Very High Achievement (VHA), High Achievement (HA), and Sound Achievement (SA). I definitely was at the end of the scale where I didn’t have a strong connection to my culture but actually attending all that cultural stuff and also just being around other students who did have a deeper connection was quite inspiring… it made me…want to…dig deeper and question things and find out more because…my family…didn’t really talk about it that much just because my nana didn’t like to talk about it and…so…it…gave me the confidence to…actually… find out more. I did appreciate having that opportunity to speak with people and to do those cultural things. As noted in the Case Study, most students did not make explicit connections between the relevance of culture for their STEM careers but found that having Aboriginal and/or Torres Strait Islander patrons and mentors helped to show students what could be possible. However, one student was able to connect the cultural activities and STEM knowledge to their desire to work with Indigenous communities. I did enjoy the cultural activities very much and I recognised…how to apply [it] to my field…it has a big impact on my field because there’s quite a lot of opportunities to be able to go out to rural areas and work with the Indigenous people out there. Four outcomes relating to participant networks, engagement in STEM initiatives, and engagement with alumni community have been grouped together. The ASSETS program aimed to connect students with their peers and with STEM professionals to encourage, maintain, and grow students’ interest in pursuing STEM education and careers. Increasing connections can help students share learning aspirations, overcome challenges, and expose students to new opportunities. Data from the Case Study and Destination Reports indicates that the program had an effective impact on the growth in student and professional networks and an effective impact on student participation in broader STEM initiatives. Reports from Program staff also highlighted examples where ASSEETS students and mentors have gone on to champion and advocate for Two-way Science in their furth study or career. As noted above, participants entered the program with very different backgrounds and with varying levels of connections to other Aboriginal and/or Torres Strait Islander students interested in STEM. Regardless of the existing connections, the evaluation found that students benefited from being around other likeminded students in an inclusive and accepting environment. The Destination Report highlighted that many alumni found the connections with peers and forming of friendships the most valuable part of ASSETS. Parents/carers also noticed the growth in student networks and saw it as a positive support network for participants. While not all students formed lasting relationships, many students did report making continuing and lasting connections with their peers. ASSETS program staff worked to maintain these peer-to-peer connections following the summer school and set up a social media group where alumni could communicate with one another, share achievements, and support one another. This group, which in August 2020 had 360 members, was also used by ASSETS staff to connect ASSETS alumni with other STEM opportunities. ASSETS staff also worked to connect alumni with work experience and other STEM-related opportunities to help build their professional networks and interest in STEM. While evidence of professional network building was not as strong as that for peer networks, many students understood the potential benefits of being in close proximity with STEM professionals and felt privileged to be able to build such relationships. ASSETS staff also connected alumni to various opportunities within CSIRO (such as cadetships and casual employment) and in other STEM organisations. Table 3 provides a snapshot of some of the activities alumni participated in in 2019 and 2020. Growth in student and professional Level of evidence: networks highParticipants engaged in broader STEM initiatives – e.g., work Level of evidence: placements, Awards program, mediumCREST, BHP Science Awards Cohorts of role models, both mentors and students, Level of evidence: contributing to Two-way science mediumand championing Indigenous knowledgesActive, engaged, skilled, and Level of evidence: growing alumni community and mediumprofessional network. Table 3. Snapshot of ASSETS alumni engagement in broader STEM initiatives in 2019 and 2020 2019 2020 • Four ASSETS alumni commenced as CSIRO Cadets • Three ASSETS alumni joined CSIRO as Vacation Scholars • ASSETS mentors attended the 2019 GARMA Festival and presented workshops in the Youth Forum with ISEP staff • Alumnus and mentor selected for CSIRO’s Indigenous Time at Sea Scholarship • ASSETS alumni studying education attended 68th National Science Education Conference (Darwin) • Two ASSETS alumni received the Aboriginal and Torres Strait Islander Secondary Student STEM Achievement Award ASSETS program staff also spoke about a circle of engagement with some students coming back to the program as mentors and other mentors continuing to engage with participants and the program well into their STEM careers. I…have loved the connection [the social media] page has provided to allow communication between participants… and also sharing individuals’ achievements in science and technology. This has impacted the way I motivate myself towards my students…I regularly keep in contact with ASSETS participants as they impacted my life in just a positive way. I love to check up and show support to them in their achievements in life and just to re-spark the connection that was kindled on the ASSETS summer school. They [STEM Professionals] were really cool to meet. That was really amazing. They’re really smart people and just to be able to meet with them and do the activities with them and they taught us how to use all the equipment. I definitely felt like there was a group of students there that definitely had my back and I formed a close connection to. By the end of it, there was just this general sense of comradery within the entire cohort in general. …the networking with other Indigenous students was really powerful for me because I hadn’t really been exposed to that very much at my school back home. It was just crazy that there were all these kids who liked science just as much as I did, or wanted to go to uni[versity], and I was like: Oh crap, I can actually do that. Because I’m the first in my family to graduate from high school, I haven’t really been exposed to people with higher education aspirations…at that point. The evaluation activities undertaken for the Case Study and Destination Report focused on the experiences of students and sought information from parents/carers and ASSETS program staff and mentors to further assess progress towards student-focused outcomes. While overall opinions of the program were not directly sought, there is evidence that students and parents/carers valued the ASSETS program and the impact it continues to have on alumni. As highlighted above, forming connections and relationships with peers and STEM professionals, and the opportunity to connect with their cultural heritage were all highly valued by participants. This evidence is backed up by strong application numbers for the program each year; as highlighted in Figure 7 at the beginning of this section, there has consistently been more applications than places available. Schools, jurisdictions, stakeholders valuing summer Level of school and leadership program, evidence: lowresulting in greater demand Bachelor of Science (Extended) Program description The Bachelor of Science (Extended) program10 is a four- year degree program at the University of Melbourne that provides a supported pathway for Aboriginal and/ or Torres Strait Islander students to complete a Bachelor of Science degree. The program was funded through the Indigenous STEM Education Project from 2015 to 2019 and is an ongoing course at the University of Melbourne.11 The Bachelor of Science (Extended) course is designed for Aboriginal and/or Torres Strait Islander students who show potential in science but might not otherwise have access to such an opportunity. During their first three semesters (typically one and a half years), students take a number of integrated science, mathematics, and communication subjects that allow them to refine and consolidate their prior STEM knowledge. Students are also provided access to services to enable them to become familiar with the University, its processes, and staff, and are offered academic skill development sessions. During the three semesters students progressively transfer to a standard Bachelor of Science course structure. Students receive ongoing assistance for the duration of their degree and are also connected to the wider support structures for Aboriginal and/or Torres Strait Islander students at the University of Melbourne, including through Murrup Barak (Melbourne Institute for Indigenous Development). The Bachelor of Science (Extended) course is unique in that it provides a scaffolded approach to assist students for the duration of their whole degree and focuses on incorporating Aboriginal and Torres Strait Islander perspectives in the science curriculum at the University of Melbourne. In this regard, University staff work to develop and deliver a first-year science and mathematics curriculum that integrates Aboriginal and Torres Strait Islander science knowledge. Table 4 provides a summary of first-year enrolment figures for the Bachelor of Science (Extended); Figure 11 highlights that program participants came from various locations across Australia. Key achievements • The program enrolled 45 students between 2015 and 2019. Many of these students successfully transitioned into the Bachelor of Science and the retention rates are comparable with retention rates for Aboriginal and/or Torres Strait Islander Bachelor students in Australia. • The Bachelor of Science (Extended) program was successful in creating a culturally responsive, supportive environment where students were able to connect with lecturers and support staff. • University staff were able to develop and implement an engaging first-year curriculum that integrated Aboriginal and/or Torres Strait Islander knowledges into key science subjects. The program has also provided a catalyst for Faculty of Science academics and staff to rethink their engagement with Aboriginal and/or Torres Strait Islander knowledge holders, academics and students. • At the end of 2020, three students had graduated from the course while approximately two thirds of students continued to work towards completion of their degree. Table 4. Bachelor of Science (Extended) enrolment figures INTAKE YEAR NUMBER OF STUDENTS ENROLLED 2015 12 2016 5 2017 8 2018 10 2019 10 Total 45 10 The program is referred to in the present tense as it is an ongoing program delivered by the University of Melbourne. The BHP Foundation funding period (2015-19) is distinct from the current University of Melbourne funding period (2019 onwards). 11 Unless otherwise specified, the evaluation of this Program focuses on the funding period which ended in 2019. Figure 11. Home locations of Bachelor of Science (Extended) students Figure 11. Home locations of Bachelor of Science (Extended) students Measuring success The Bachelor of Science (Extended) Impact Statement in Appendix B provides an overview of the key outcome pathways for the Bachelor of Science (Extended) program. Similar to the other programs, this impact statement has been updated to provide more detail on key inputs and activities, refine the pathways and timeframes to achieve change, and highlight participant priorities. A key addition worth noting is the inclusion of the following outcome - Aboriginal and/or Torres Strait Islander students succeeding in tertiary education, including graduating from Bachelor of Science (Extended). While the goal of the Bachelor of Science (Extended) was to see students succeed in tertiary STEM education, student success in tertiary education, regardless of the specialty, was also an important result. Students often change courses as they refine their education interests and career aspirations. It is important that this was not seen as an adverse result for the program. The key outcomes were: Short-term • Strong student engagement, retention, and results. • Increased student aspiration. Intermediate • Students successfully transition into Bachelor of Science. • University building stronger relationships/partnerships with local Aboriginal organisations and communities to increase Aboriginal science knowledge. • Innovative, place-based, culturally responsive science curriculum and pedagogy incorporating Indigenous science knowledge delivered by University of Melbourne academics. Long-term • Best practice in science extended courses and Aboriginal and/or Torres Strait Islander engagement identified and adopted by the University of Melbourne and other universities. • Strong student engagement with development opportunities – study abroad, exchange, scholarships, awards, prizes, volunteering, and leadership opportunities. • Aboriginal and/or Torres Strait Islander students succeeding in tertiary education, including graduating from Bachelor of Science (Extended). Evaluation methods Progress against the key Bachelor of Science (Extended) outcomes was assessed using a variety of measures and sources. In June 2017, CSIRO commenced a Case Study evaluation to explore pathways and processes of the program, and the extent to which program goals had been achieved (Mudhan, Banks, Gilbert, & Sadler, 2019). The Case Study involved individual and group interviews with 24 people, including students, program staff, and key stakeholders. All students who were currently enrolled or had withdrawn from the course were invited to participate in the research. Six currently enrolled students participated. In May 2019, the University of Melbourne commissioned ORIMA Research (ORIMA Research, 2019) to conduct a qualitative evaluation of the Bachelor of Science (Extended) program and assess the extent to which it was preparing students for success in the standard Bachelor of Science program. Where relevant, these results are also included in the discussion below. Key findings of the Bachelor of Science (Extended) program Figure 12 provides a summary of progress towards outcomes for the Bachelor of Science (Extended) program. New or updated outcomes, and outcomes with no available data, have been excluded from this analysis. Figure 12. Bachelor of Science (Extended) - progress towards outcomes Figure 12. Bachelor of Science (Extended) - progress towards outcomes 1. Strong, effective partnerships established with schools and other stakeholders2. Increased community and parental engagement5. Increased aspiration, sense of value and school belonging3. Increased teacher capacity in Two‐way Science using On‐Country contexts and cultural competence4. Increased student engagement and attendance7. The program’s Two‐way Science learning approach and resources are culturally responsive, community‐ based and embedded across school curriculums: 6. Enhanced student results Strong student engagement, retention, and resultsLevel of evidence: mediumIncreased student aspirationLevel of evidence: mediumStudents successfully transition into Bachelor of ScienceLevel of evidence: high Three outcomes relating to student experiences of the program have been grouped together. The Case Study evaluation data indicated that the Bachelor of Science (Extended) had an effective impact on student engagement and retention and was effective at supporting existing student aspirations. The program was also found to be transformative in assisting students successful transition to a standard Bachelor of Science. These results are consistent with the assessment from the University of Melbourne’s external evaluator, which found that the course was effective at ‘bridging the gap’ between high school and university (ORIMA Research, 2019). The transition to university is challenging for many students across Australia; this is particularly so for students that move away from home to study (Baik, Naylor, & Arkoudis, 2015), as is the case for many Bachelor of Science (Extended) students. The foundation year program provided by the University of Melbourne aimed to ease the transition for Aboriginal and/or Torres Strait Islander Students studying science. Feedback from students highlighted the following factors as important to their ongoing success in the course: • The ability to develop strong personal connections with lecturers and support staff who were committed to the success of the program and students. • Multi-layered, culturally aware, and competent support and advice offered by various University personnel including from Murrup Barak. • Connections with other Aboriginal and/or Torres Strait Islander peers from similar backgrounds. • Development of a culturally responsive, open learning environment. While students in the Bachelor of Science (Extended) course did report feelings of loneliness and disconnection, they also recognised that the advice, connection, and opportunities provided to them through the course helped to overcome some of the challenges faced. Students provided mixed feedback on their levels of engagement with course content. While some students indicated that the content was overly broad and consequently, not interesting, other students were more engaged with the content and found the inclusion of Indigenous knowledges particularly interesting. The mixed results were confirmed by teachers, who recognised that students brought a range of interests into the classroom and correspondingly had different levels of engagement with various topics. The feedback could also be reflective of the multidisciplinary nature of the first-year subjects offered in the course, with students not sharing the same level of interest across all discipline areas. Some students also commented that the pace of learning in the first year was slow and that this was initially a source of frustration. As students progressed further in the course, there was more general agreement about the benefit of providing a solid base of learning for future Bachelor of Science subjects. First-year retention has been identified as critical for establishing a pattern of study and academic engagement (James, Krause, & Jennings, 2010). Table 5 highlights the retention figures for the course and shows that after completing their first year, most students continued in the program. The retention rates for subsequent years vary by cohort, and for the 2015 and 2016 cohorts, decreased slightly in 2019. While small cohorts make it hard to draw strong conclusions from the data, with the exception of the 2019 results for the students enrolled in the course in 2016, most of the annual retention figures compare favourably to the retention rate of 71.2 per cent for all Aboriginal and/or Torres Strait islander Bachelor students in Australia in 2017 (Universities Australia, 2017). As outlined in Taylor, Lalovic, and Thompson (2019), a range of factors affect the retention of Aboriginal and/or Torres Strait Islander Students in tertiary education. Family and peer support have been identified as pivotal to student well-being and to students remaining at university. Conversely, family/community obligations and financial hardships have also been identified as significant sources of stress for students. The strong retention rates of the Bachelor of Science (Extended) are a good indication that the program’s multi-layered support structure has been successful in guiding students through their first year and establishing a solid foundation for future academic success. Table 5. Bachelor of Science (Extended) retention figures by cohort 2016 ENROLMENT 2017 ENROLMENT 2018 ENROLMENT 2019 ENROLMENT OVERALL 2015 Cohort (12 students) 75% (9 out of 12) 78% (7 out of 9) 100% (7 out of 7) 71% (5 out of 7) 42% (5 out of 12) 2016 Cohort (12 students) n/a 100% (5 out of 5) 100% (5 out of 5) 60% (3 out of 5) 60% (3 out of 5) 2017 Cohort (12 students) n/a n/a 88% (7 out of 8) 100% (7 out of 7) 88% (7 out of 8) 2018 Cohort (12 students) n/a n/a n/a 80% (8 out of 10) 80% (8 out of 10) 2019 Cohort (12 students) n/a n/a n/a n/a n/a All cohorts (12, 17, 25,35 enrolled) 75% (9 out of 12) 86% (12 out of 14) 95% (19 out of 20) 79% (23 out of 29) 66% (23 out of 35) Shaded cells show proportion of first-year students re-enrolling in their second year of the course Both students and teachers highlighted low and inconsistent attendance rates for some students as an area for improvement, bearing in mind that many students had 100 per cent attendance rates. While students often had many varied reasons for low attendance, including participation in extra-curricular activities, illness, and attending cultural events, there was a clear consensus among teachers that attendance was linked to achievement in the program. Attendance was positively impacted by a range of factors, including participation in the Indigenous Tutorial Assistance Scheme offered by the University and adjustment of course content and topics to meet the interests of participants. Teachers also recognised that these changes take time and planning and must be balanced with the need to cover the required curriculum. The support that they offered and the sense of community that they were fostering here, I thought it was a fantastic opportunity to get back in touch with my Indigenous heritage because it wasn’t something that I was offered at school. (Student) …my…teacher…would organise times for me to meet them and we’d go through everything…and what I need to work on, and I think that’s something that really helped. (Student) It is fair to say, I think, that class attendance is, for most students, crucial to success. (Teaching Staff) I think the students do find the content engaging. They have different disciplinary interests, so they’ll find different bits more interesting than others, and they’ll find different parts of it more challenging than others too. (Teaching Staff) The Case Study evaluation results showed that the program was effective at developing innovative place- based Indigenous contextualised curriculum content and there had been effective progress by the University in building stronger relationships/partnerships regarding Indigenous scientific knowledges.12 Qualitative feedback indicated that students appreciated topics that connected their existing Indigenous knowledges with western scientific knowledge. Students reported increased motivation when they could use their existing knowledge (e.g., stories passed down to them by Elders) as a foundation to build further scientific understanding. University of Melbourne academics generally had positive attitudes towards incorporating Aboriginal and Torres Strait Islander knowledges into the curriculum. Academics did acknowledge that the ability to incorporate such knowledge differed for each subject/topic, for example, teachers found it is easier to incorporate Aboriginal and/or Torres Strait Islander knowledges into biology, than for chemistry. Staff also acknowledged that not all academics had the same level of confidence in integrating Indigenous science perspectives, for example, some staff worked with existing culturally responsive pedagogies, and that support from colleagues was often needed. Overall, feedback from staff to program evaluators was that the curriculum development was heading in the right direction and was seen as a work in progress. One unintended outcome of the program reported by University staff was the transformational nature of the degree program on the staff involved. Specifically, it was conveyed that staff have embarked on changes to their teaching practice and other initiatives such as furthering Indigenous engagement within their Faculty, modelling for other Faculties the co-creation of processes to support students with Murrup Barak, and sharing teaching practice with colleagues both within and beyond the University. Staff emphasised that each cohort of students came from varying locations around the country, and many had strong connections to their communities and culture. Students were often very open to sharing information about their particular culture/s and connecting staff with key people in their communities to further inform curriculum development and connect the University to Elders who, in turn, provided further input into the program. This cyclical relationship helped to further improve the program. Innovative, place-based Indigenous contextualised Level of evidence: curriculum content developed and mediumintegrated into the University of Melbourne’s science curriculumUniversity building stronger relationships/partnerships Level of evidence: regarding Indigenous mediumscientific knowledges with Indigenous organisations 12 The evidence was not as strong for the latter outcome as it was not the primary focus of the evaluation. The Case Study also highlighted that staff were generally adopting a careful and cautious approach to ensure they were being culturally respectful and engaging with the appropriate people regarding integration of Aboriginal and Torres Strait Islander knowledges into the curriculum. Staff also expressed a desire to connect further with local knowledge holders and Indigenous organisations. Some students shared concerns regarding the appropriateness of local place-based Indigenous knowledges being shared with all Aboriginal and/or Torres Strait Islander students and its lack of inclusivity to other groups. This emphasises the challenges associated with generalising Indigenous knowledges and teaching them more broadly across different groups. I thought it was really important that those Indigenous sort of sources and readings and lectures were acknowledged… with the inclusion of fire into Indigenous knowledge and practices…that was quite fantastic. (Student) …you need to embed Indigenous knowledge perspectives in your curriculum, and so it’s a very complicated thing and in some ways we’re still right at the start of that...we’re all together starting to think through how and in what ways can you engage with Indigenous knowledge. (Teaching Staff) It definitely made it a lot easier to learn because it [Indigenous knowledge] was really quite fascinating. In school we didn’t get taught about Indigenous practices, I only learnt that from my Aunty. So it was really nice to have those things taught in class, it was a really good way to stay motivated. Particularly towards the end of semester…it’s like the pressure for the exams hasn’t quite hit yet, you just hit quite a slump. (Student) The Bachelor of Science Program (Extended) has been effective in helping students succeed in tertiary education, and in particular, in tertiary science education at the University of Melbourne. The Bachelor of Science (Extended) program has had an emerging impact on student engagement with development opportunities. At the end of 2020, the Bachelor of Science (Extended) program had celebrated the graduation of three students from the program. At the time of publication, four students were on track to complete their degree in 2021. More completions are expected in the coming years as many students have decided to reduce their course loads, which extends the duration of the degree beyond four years. Anecdotal evidence from program staff indicates that they are expecting an increasing number of students from each cohort to complete their course. Data from the University of Melbourne indicates that at August 2020, 17 per cent of students were continuing their tertiary studies in other courses at the University of Melbourne or elsewhere. Due to limitations in data, there is no information available to track the outcomes of students who transferred out of the Bachelor of Science (Extended) program into another program at the University of Melbourne or at another institution, however the result is not viewed as a negative result but rather a natural occurrence as students further refine their field of interest. Across the Australian higher education sector, there is a concerted push to increase both retention and completion rates for Aboriginal and/or Torres Strait Islander students (Universities Australia, 2019) and a number of key strategies have been identified as important to increasing these rates, including: • strong partnerships with local Elders and community, • enabling courses, • dedicated Indigenous centres, • fostering culturally safe environments, • dedicated tutorial assistance, and • pastoral care and access to specialist advice. These strategies are central components of the Bachelor of Science (Extended) program and emphasise that the course is leading the way in supporting Aboriginal and/ or Torres Strait Islander students in tertiary education. Aboriginal and/or Torres Strait Islander students succeeding in tertiary education, including graduating from Bachelor of Science (extended) Level of evidence: medium Strong student engagement with development opportunities – study abroad, exchange, scholarships, awards, prizes, volunteering, and leadership opportunities Level of evidence: medium Data from the University of Melbourne highlights that a number of Bachelor of Science (Extended) students have been engaged with other programs at the University, specifically: • One student joined the study abroad program. • Five students have taken internships for credit • Six students have volunteered as mentors for the University of Melbourne’s Residential Indigenous Science Experience (RISE) and the Victorian Indigenous Engineering Winter School (VIEWS) • Six students have been demonstrators for laboratory science sessions with Melbourne Indigenous Transition School (Year 7 students) • Four students have been recognised with leadership awards by Agilent. Students have also received bursaries and financial grants to recognise and support their continued success in the course. Indigenous STEM awards Program description The Indigenous STEM Awards program (‘Awards Program’) recognised, rewarded, and celebrated the achievements of Aboriginal and/or Torres Strait Islander students and scientists studying and working in the Science, Technology, Engineering, and Mathematics (STEM) fields, as well as the integral role schools, teachers, and mentors have in supporting Aboriginal and/or Torres Strait Islander students in pursuing STEM education and careers. The Awards Program ran from 2016 to 2020, Figure 13 provides a snapshot of the nine Awards categories . During this time, the Awards Program recognised the achievements of 44 recipients and 120 finalists (this includes four school Award recipients). Finalists and recipients were selected by a panel comprising a range of professionals from CSIRO, the BHP Foundation, and other related STEM organisations. Dr Aunty Kaye Price AM served as the Chair of the judging panel. The Awards Program started as an initial pilot program in 2016. In 2018 and 2019, secondary student recipients were given the opportunity to travel to the United States to participate in the International Science and Engineering Fair (ISEF). Commencing with the 2019 cohort, secondary student Award recipients received monetary prizes in place of international travel. Commencing in 2018, recipients (except the student winners not yet in high school) also participated in a per-announcement gathering where they worked with Awards Program staff to co-design each winner’s preferred presentation/celebration ceremony, usually with their local community. Recipients were also provided with personal development opportunities, including media and communications training, and ongoing mentoring. Key achievements • Since its inception, the Indigenous STEM Awards have recognised the achievements of 44 recipients and 120 finalists (this includes four school Award recipients). As illustrated in Figure 14, participants have come from across Australia. • The Awards Program has worked with Award recipients to design culturally responsive, individualised presentation/celebrations, often with the recipient’s local community. • Participants in the Awards Program have reported increased recognition of their achievements in STEM and of Aboriginal and/or Torres Strait Islander knowledges. The program has also successfully opened doors and connected participants with many new opportunities. • Creation of a webpage of Award recipients and finalists, showcasing high-achieving Aboriginal and/or Torres Strait Islander STEM students and professionals. • Aboriginal and Torres Strait Islander STEM Professional Award • Aboriginal and Torres Strait Islander STEM Professional Early Career Award • School Award • STEM Champion Award • Teacher Award • Aboriginal and Torres Strait Islander Tertiary Student STEM Achievement Award • Aboriginal and Torres Strait Islander Secondary Student STEM Achievement Award • Aboriginal and Torres Strait Islander Student Maths Award • Aboriginal and Torres Strait Islander Student Science Award Figure 13. Indigenous STEM Awards categories Figure 14. Home locations of Indigenous STEM Award recipients and finalists Figure 14. Home locations of Indigenous STEM Award recipients and finalists Measuring success The Indigenous STEM Awards, similar to the other programs within the Indigenous STEM Education Project, has its own Impact Statement. The Impact Statement for the Indigenous STEM Awards was updated in 2021 to reflect the key program activities and experiences of participants more accurately (see Appendix B). The key outcomes were: Short-term • Indigenous STEM achievement and best practice in STEM teaching showcased and promoted • Network of high-achieving Aboriginal and/or Torres Strait Islander peoples pursuing STEM education or careers Intermediate • Active, engaged, skilled and growing Awards Alumni network • Participants engaged with other CSIRO STEM programs Longer-term • Increasing number and involvement of champions who are mentoring Indigenous STEM students • Increased recognition and championing of the role of family, community and mentors to success in STEM • Raising student aspirations to pursue science education and careers • Increased awareness and uptake of science inquiry pedagogy Evaluation methods As one of the smaller programs within the Indigenous STEM Education Project, the Awards Program was not prioritised for a detailed evaluation. Accordingly, data has not been collected against all the key outcomes listed in the Awards Program Impact Statement. To help understand the impact the Awards Program has had on participants, in early 2021, a short online survey was sent to Award recipients and finalists; semi-structured interviews were also held with two Award winners and three members of their immediate networks. Snowball sampling was used to identify those interviewees based on recommendations from the Award recipient. Thematic content analysis was used to analyse the interview data. Findings from the survey and interviews have been published in a short, ‘Stories of Change’ report (Walker, Gilbert, Harris, Salmon & Banks, 2021). The Stories of Change report is unique among the case study evaluations undertaken for the Indigenous STEM Education Project due to the prominent focus on participants’ experiences rather than a purely evaluative approach. Combining the stories of two Award winners with survey data makes it possible to identify common themes an see how these relate to the intended outcomes of the Awards Program. Key findings of the Indigenous STEM Awards program Recognition Recognition of individual efforts in STEM was a clear theme that emerged from the survey and interviews with Award recipients. As outlined in the Stories of Change report (Walker et al., 2021), for some Award recipients, the additional recognition had a direct impact on their confidence and motivation levels. While some Award recipients received media coverage, and highlighted this visibility as important to further promoting their work and the importance of STEM for Aboriginal and/or Torres Strait Islander students, others suggested that more could be done by CSIRO to liaise with media and promote the Awards Program and individual STEM achievements. CSIRO’s a big name. To be recognised by CSIRO, you know you’re going to be put somewhere so that people are going to see it…What we know, and what we do, it’s acceptable…it’s okay to share it. It’s okay to promote it. And it’s good. It’s real, it’s worthwhile. The recognition served as great motivation to continue my efforts. Meeting and engaging with other awards winners provided me with confidence and new ideas. The impact of this award has been pervasive in many facets of my life, both professional, community, and family. The external recognition and validation of the importance and impact of my work at [school] and the wider community has led to a marked increase in my confidence. A School Award recipient echoed the comments of individual recipients. Recognition that the STEM opportunities we provide are exceptional and we need to continue with them, share with others, and involve more students. The recognition and honouring of Aboriginal and Torres Strait Islander individuals and scientific knowledges also contributed to broader outcomes of reconciliation, that is, the public acknowledgement of culture and knowledges helped interrupt and reverse the oppression of knowledges as a result of colonisation. Opportunities Many Award recipients who responded to the survey indicated that the Awards Program provided them with valuable opportunities and connections. Asked to list specific opportunities, responses included further tertiary education, job offers, conference presentations, and additional project funding. The Awards Program also provided the opportunity for participants to receive communication and media training, an opportunity that was highly valued by some participants. What’s happened since I have won that award...I’ve had a lot more people ringing and asking and talking [about Two-way Science]. Going to conferences in Perth with the education department, talking to this person, this person, making connections with everyone. I think one of the greatest bits that I’ve actually been a part of was when we went as a group to do the training for the media and a number of things like that at the Powerhouse Museum. Being able to hear everyone’s stories…Then when I started hearing about what everyone else was doing, I was actually amazed by the amount of great STEM work that Indigenous people were making and a number of those people I’ve still got contact with. The best part of being an Awards recipient is all the connections that I am able to make. I was able to connect with the other winners at the winners gathering (a number of which I now have ongoing relationships with), the connections I made at my presentation, and the connections that I can continue to make now even 3 years on. I am still recognised as an Award winner, and this gives me more opportunity to network with people who I can lean on to help me with my career, navigate the industry, my journey to reconcile Australia, etc. Feedback on the connections and opportunities made by Award finalists was not as strong, with only 33 per cent of respondents agreeing or strongly agreeing that being selected as a finalist helped them secure education/work opportunities and only 17 per cent of respondents agreeing that they made new contacts. These results are not surprising given the Awards Program primarily focused on providing benefits and opportunities for Award recipients. Engaging with the broader network of finalists is an opportunity future awards programs could explore further. One of the characteristics of the Awards Program is that many participants are also engaged in other CSIRO STEM programs, underscoring the interconnectedness that the program involves or contributes to. As highlighted in Figure 15, survey data collected from Award recipients and finalists highlighted that participants have been extensively involved in the Indigenous STEM Education Project and have also connected with other CSIRO STEM initiatives, often through the efforts of Indigenous STEM Education Project staff members. Connection to family, culture, and other Aboriginal and/or Torres Strait Islander peoples interested in STEM The Awards Program sought to build recipients’ connections to other Aboriginal and/or Torres Strait Islander peoples interested in STEM and highlight and promote Aboriginal and Torres Strait Islander knowledges within the community. Data collected through surveys and feedback from participants interviewed highlighted that participants made connections between the increased recognition and confidence they felt as a result of receiving an Indigenous STEM Award and the impact that had on their communities and the acceptance of Aboriginal and Torres Strait Islander knowledges. These impacts were not confined to immediate Award recipients, with families of those recognised also growing in confidence and in some instances, also becoming advocates for Two-way Science. Confidence, recognition, and validation. The Awards highlights and promotes just how integral local, everyday Indigenous peoples are to lifelong learning and community. That backgrounds are not barriers to being a valued voice and engaging with and being role models within the community. It draws the line between the knowledge that Indigenous peoples have always had and ‘accepted’ modern, Western science, thereby shining a spotlight on the good that is being done that otherwise would not have been seen or heard of. These are people that we know. People take cultural knowledge for granted; we’ve come from this massive, long history of assimilation and this awful history where language and culture were things that were stopped deliberately, and, then to suddenly Figure 15. List of CSIRO initiatives survey respondents have been involved with Indigenous STEM Education Project initiatives • Aboriginal Summer School for Excellence in Science and Technology (ASSETS) • ASSETS Work Placement Program • Bachelor of Science (Extended) • Inquiry for Indigenous Science Students Program (I2S2) • Science Pathways for Indigenous Communities Other CSIRO STEM Initiatives • BHP Foundation Science and Engineering Awards • Creativity in Research, Engineering, Science, and Technology (CREST • STEM Professionals in Schools • Sustainable Futures • Young Indigenous Women’s STEM Academy • have that be the focus of celebration, I think it’s a really powerful statement and I think [individual award winners] and all those people out in remote communities who have this long experience, awful experience, history, to see that then completely reframed as a celebration of cultural knowledge and Aboriginal science and STEM and the people who were working with it, was enormous. It wasn’t in the city; all those things are just really powerful. Similar impacts were felt by a School Award recipient: [As a result of the Awards Program] STEM has a higher profile with the Indigenous students. More students are recognising the importance of STEM. The general student and staff population of our school are keener to learn Indigenous STEM concepts and ways of working. 92 per cent of individual respondents agreed or strongly agreed that after receiving an Award they felt a stronger connection to a community of Aboriginal and/or Torres Strait Islander peoples working in or interested in STEM I2S2 Program description I2S2 provided professional learning opportunities to teachers of Year 5 to 9 science students in metropolitan and regional schools. The training supported teachers to embed Aboriginal or Torres Strait Islander scientific knowledges in their classrooms through hands-on inquiry-based projects and sought to increase student engagement and achievement in science. The program focused on building teachers’ capacity to deliver science inquiry lessons, to build strong and positive relationships with parents/carers and the community, and to link Aboriginal and/or Torres Strait Islander knowledge and histories to the science curriculum. The I2S2 program has transitioned through different delivery phases during which the program team worked to provide a best practice teacher professional learning (TPL) program while responding to challenges including funding uncertainties and the impact of COVID-19. While the program was aimed at Aboriginal and/or Torres Strait Islander students, it was delivered to all students in those year levels at participating schools. Commencing in 2015, participating teachers were provided with four face-to-face TPL sessions over two weeks. The sessions were designed to improve teachers’ understanding and capacity in science inquiry and provide them with foundational knowledge and understanding of key cultural considerations to help them recognise the diversity of Aboriginal and Torres Strait Islander cultures and identities. The program team also worked closely with school clusters following completion of the training to consolidate their learning and encourage further collaboration. Activities included class visits, modelling, and team teaching. In early 2019, the program shifted to a blended model that incorporated both face-to-face and online TPL. Teachers participating in the online TPL had access to five modules and a range of webinars, as well as ongoing support and resources from the program team. In February 2020, I2S2 launched a new iteration of the online program intended to run alongside, and support, the face-to-face module. As a result of the COVID-19 pandemic and associated school closures and travel restrictions, the I2S2 program moved entirely online with five self-paced inquiry modules available for teachers and parents/carers to download. Following completion of the modules, participants were given access to I2S2 inquiry-based resources and materials. Figure 16 provides a summary of the overall program participation numbers (to September 2020). As shown in Figure 17, I2S2 engaged with schools in all states and territories. Key achievements • I2S2 connected with 513 schools, 548 teachers, and reached 11,375 students. In addition, a further 499 teachers accessed the online TPL. • I2S2 improved the capacity and confidence of teachers to adopt an inquiry-based science pedagogy and integrate Aboriginal and Torres Strait Islander contexts into their lessons. • The program was successful in increasing student engagement and contributed to improved academic results, particularly for previously low-achieving Aboriginal and/or Torres Strait Islander students and low-achieving non-Indigenous students. • Teachers reported that by valuing Aboriginal and Torres Strait Islanders knowledges, I2S2 increased students’ pride regarding their Aboriginal and/or Torres Strait Islander heritage. Figure 16. I2S2 program participation numbers Figure 17. Locations of schools participating in I2S2 Figure 17. Locations of schools participating in I2S2 Figure 16. I2S2 program participation numbers Figure 16. I2S2 program participation numbers Figure 16. I2S2 program participation numbers 11,375 Students 548 Educators Plus 499 using online TPL 513 Schools Measuring success The Impact Statement in Appendix B provides an overview of the key outcome pathways for the I2S2 program. This Impact Statement has been updated to provide more detail on key inputs and activities, refine the pathways and timeframes to achieve change, and highlight community priorities. The key outcomes are: Short-term outcomes • Teachers adopting inquiry-based pedagogy and improving cultural responsiveness. Intermediate outcomes • Increased teacher capacity in both inquiry and Indigenous contexts. • Increased student engagement and (academic) results. • Increased student aspiration, sense of value, and school belonging. • Increased community and parental engagement and increased cultural competency of schools in delivering Aboriginal and Torres Strait Islander contextualised inquiries in partnership with families and community. • Schools supporting other STEM programs (e.g., ASSETS, CREST, Awards, PRIME Futures). Long-term outcomes • Increased number of Indigenous (and non–Indigenous) students pursuing STEM pathways, including in Years 10 to 12, university, and alternatives. • Identification of ‘best practice’ in high expectations science inquiry education and teacher professional learning, and adoption of this ‘best practice’ by states and territories. One outcome, relating to school cultures of high expectations, was also revised and moved to an impact. • Schools, students, and families increasingly adopting a high expectations focus, contributing to a new cultural norm of Aboriginal and/or Torres Strait Islander students attending higher education and pursuing STEM careers. Evaluation methods Progress against outcomes was assessed using a variety of measures. In 2021, just prior to the completion of this report, a detailed Case Study Evaluation was undertaken (Banks, Fidler, Gilbert, & King, 2021). Similar to other Indigenous STEM Education Project programs, this Case Study methodology was preferred because it highlights the voices of program participants. The Case Study utilised a multi-methods approach incorporating the following methods: • Focus groups with students, teachers, and school leaders in eight schools across three states. • Teacher surveys asking for pre-and post- inquiry assessments of students’ engagement and academic achievements. • Jurisdictional data on achievement, engagement, and STEM course selection at I2S2 schools and comparison schools.13 Data in the Case Study builds on data published in the second and third evaluation reports and assesses progress against outcomes up to September 2019. This report draws on all previously published data and reports additional teacher survey results for 2020. 13 As outlined in Banks et al. (2021), of the three jurisdictions contacted, one provided whole-of-grade level data for each school, another provided data for a small number of schools in the I2S2 program, and a third did not provide any. The lack of class- or student-level detail in the data provided makes it difficult to draw strong conclusions about the impact of the I2S2 program. Key findings of the I2S2 program Figure 18 provides a summary of progress towards outcomes for the I2S2 program. New or updated outcomes, and outcomes with no available data, have been excluded from this analysis. Figure 18. I2S2 progress towards outcomes The program had an effective impact on the capacity of teachers to adopt an inquiry-based science pedagogy and incorporate Aboriginal and Torres Strait Islander contexts into science lessons. Teachers were asked to self-report on changes in their capacity and confidence to deliver inquiry-based science lessons and to incorporate Indigenous contexts in their science lessons. As outlined in the I2S2 Case Study, teachers found that following their participation in professional learning, their confidence increased over time with increased application of the I2S2 approach in the classroom. This was confirmed by program staff assessments and student feedback. The following list highlights some common themes identified by participants: • Teachers reported building confidence over time as they continued to apply the new skills. • As confidence increased, so did teachers’ abilities to amend or extend lessons to suit their specific classroom needs. • Teachers found resources useful and comprehensive. • Efforts by program coordinators to create strong relationships with teachers and demonstrate inquiries had a significant impact on lifting the capacity of individual teachers. • Support from other teachers and other schools contributed to building capacity and confidence. Figure 18. I2S2 progress towards outcomes 7. Increased number of Indigenous (and non– Indigenous) students pursuing STEM pathways, including in Years 10 to 12, university, and alternatives 3. Increased student aspiration, sense of value, cultural identity and school belonging5. Schools supporting other STEM programs (e.g. ASSETS, CREST, Awards, PRIME Futures) 2. Increased student engagement and academic results1. Increased teacher capacity in both inquiry and Indigenous contexts4. Increased community and parental engagement and schools have increased cultural competency delivering Aboriginal and Torres Strait Islander contextualised inquiries in partnership with families and community 6. Identification of ‘best practice’ in high expectations science inquiry education and teacher professional learning, and adoption of this ‘best practice’ by states and territories Teachers commented: …since my time starting [I2S2] last year, there’s definitely significant increases with running all these extra programs, so – it’s becoming easier and easier. I can see that over time, we’re probably going to get more and more confident in saying: ‘No, we don’t have to stick with those points so much. We can move around a little bit here’, as long as we don’t lose sight of what it is that we need to assess and what it is we need to have the kids produce. [I2S2 program Coordinator] was really awesome and they loved the work that we were doing. And they built [a teacher at the school]’s capacity amazingly, so this is her baby. We [the teacher and the Coordinator] ran a practical together and so he has helped me set up this scaffolded one and then we ran it, then he came back and I had it four times I think, that term, four or five times. And each time he helped me get through the next. Once I had done it, I had an idea. But I had never done anything like that before. Students spoke to the changes in teaching styles: Yeah, she teaches in a way that really suits…[and] she’s starting to change the lessons up a little bit, so kids don’t get bored in class and you have more different stuff to do and everything. It’s really good. The evaluation highlighted a number of challenges that teachers faced implementing the I2S2 model. These challenges included: • Logistical – including access to equipment and materials, and the time and physical space to do inquiries. • Lesson delivery – including catering for different levels of achievement, balancing inquiries with other curriculum requirements, and shifting from an explicit teaching pedagogy. • Designing professional development suited to teachers with a range of teaching experiences. • Shifting to an online delivery model created apprehension for some teachers who felt less confident without the face-to-face support traditionally provided by program Coordinators and the specialised local knowledge brought by a local Coordinator. The evaluation results indicated that the program had a transformative impact on student engagement and academic achievement. I2S2 also had a transformative impact on student aspirations, sense of value, and feeling of school belonging. As outlined in the I2S2 Case Study, data on student engagement and academic achievement showed that the program had strong impacts on low-achieving students, defined as those receiving a grade ‘D’ or ‘E’ before participating in an I2S2 inquiry. The program had a small or negligible impact on the overall achievement of students in participating schools; however, for low-achieving students, 2018 data showed that the program had a significant impact, with mean grades increasing from 1.78 before students participated in I2S2 inquiries to 2.34 after.14 The proportion of low-achieving students recording a passing grade in science increased from 0 per cent to 42 per cent in 2018 and 34 per cent in 2019. The results for Aboriginal and/or Torres Strait Islander students followed similar trends, with small changes to the overall group but low-achieving Aboriginal and/or Torres Strait Islander students showing substantial improvements. The data on student engagement followed similar trends to the data on student academic achievement. Overall, mean engagement levels (as reported by teachers) increased modestly in 2018 and 2019, with engagement for low-achieving students increasing substantially. Similar trends were evident when looking at teachers’ perceptions of the engagement of Aboriginal and/or Torres Strait Islander students before and after participating in an I2S2 inquiry. Mean engagement increased by a small amount for all Aboriginal and/or Torres Strait Islander students (the change was not statistically significant) but the increase was quite substantial for low-achieving Aboriginal and/ or Torres Strait Islander students (41.5 per cent of students had increased engagement). The relative changes in academic achievement and engagement for low-achieving students versus the overall group are not unexpected15. As one teacher suggested, “kids who were sitting at quite a high level don’t improve that much”. Qualitative feedback from teachers and students highlighted some of the reasons for the improved results, including: • Increased use of hands-on practice and inquiry-driven science lessons in place of textbook-based lessons. • Students were interested and engaged with Aboriginal and Torres Strait Islander knowledge systems. • Multi-method and flexible assessments allowed students to demonstrate their knowledge and understanding and provided students with more opportunities to succeed. Increased student engagement and academic results Level of evidence: high Increased student aspirations, sense of value, and school belonging Level of evidence: high 14 Grades were converted to numeric scores A = 5, B = 4, C = 3, D = 2, E = 1. 15 Positive motivation and engagement has been found to significantly predict academic outcomes for Aboriginal and/or Torres Strait Islander students (Martin, Ginns, Anderson, Gibson & Bishop, 2021). Feedback from students, teachers, and parents/carers during interviews and focus groups highlighted the increased confidence of students participating in I2S2 inquiries. This was particularly the case for Aboriginal and/or Torres Strait Islander students who were able to contribute their knowledge and connect their culture to classroom lessons. Students shared their appreciation for the ability to connect with Aboriginal and Torres Strait knowledges and, for some students, highlighted how the lessons helped them feel more comfortable contributing in the classroom. Teachers also reported that by valuing students’ existing knowledge, the program generated pride in students regarding their Aboriginal and/or Torres Strait Islander heritage. Some of those boys that got it quickly took a lot of pride in then showing other kids how to do it as well. Kids like [student] who…traditionally don’t do particularly well in a mainstream classroom but once he saw that success, he was keen to show everybody else how to be successful. I think it probably actually makes [Aboriginal and/or Torres Strait Islander students] feel a bit more comfortable that [the] education system is acknowledging the Indigenous people[s] of this country and I think there’s going to be a lasting impact in terms of...Aboriginal and Torres Strait Islander kids that a system is recognising and privileging their knowledge, and the knowledge of their culture. …half of my family is [Aboriginal], so it was interesting to learn stuff about what maybe people from further back in the generations were doing, because they don’t really do it nowadays. …for me, if we do more group activities [it]… help[s] me put my hand up and…not be afraid. Evidence from the schools involved in the Case Study indicated that the I2S2 program had an emerging impact on community and parental engagement and the cultural competence of schools delivering inquiries in partnerships with families and community. As outlined in the I2S2 Case Study, participants expressed a strong desire to increase their levels of engagement and partnerships. However, there were only a few examples of schools involving Elders and parents/carers in the school and with I2S2 inquiries. Relationships with Elders and local Aboriginal and Torres Strait Islander communities were often at early stages and teachers were focused on further building these connections. Feedback from teachers indicated that there was a strong acknowledgement that community and parental support would improve teaching and support I2S2 lessons; however, few examples were provided showing this acknowledgement in practice. Community connections were also dependent on existing capacity in schools, with some schools more able than others to engage with community members and involve them in I2S2 inquiries. Focus group conversations with students highlighted examples of parental and community participation in schools and provided further context including: • Often community and family engagement was centred around cultural events (e.g., NAIDOC week) and artistic practices, for example, painting school buildings. • There was an overall sense that students felt schools were taking steps to create a welcoming environment and students were also aware of the benefits that building links with community members could bring to a school. • Students also highlighted the important role that Aboriginal and Torres Strait Islander teacher assistants play in sharing knowledge and providing support. These comments highlight that creating meaningful partnerships with communities and parents/carers and incorporating Aboriginal and Torres Strait Islander knowledges in subjects beyond art and history are important opportunity areas for many schools. I2S2 is working to fill this gap but evidence shows that there is more that can be done. As explained in the Case Study, a lack of confidence among non-Indigenous teachers to deliver Aboriginal and Torres Strait Islander knowledges or connect to Elders is a significant hurdle for many. It can also be a challenge to create authentic, engaging learning experiences delivered by Elders. More can, therefore, be done to facilitate the transfer of knowledge to teachers to empower them and increase their awareness of local Indigenous scientific knowledges. Increased community and parental engagement, and schools have increased cultural competency delivering Aboriginal and Torres Strait Islander contextualised inquiries in partnership with families and community. Level of evidence: medium Data on STEM subject selection was provided from one jurisdiction and combined with qualitative information on student subject selection to determine that the I2S2 program was effective in increasing students’ STEM subject selections and career aspirations, and was also effective in supporting schools to engage in other STEM programs. As reported in the I2S2 Case Study, comparing schools by level of educational advantage (ICSEA) showed that students in schools with higher education advantage tended to have higher levels of enrolment in STEM subjects. Comparing the proportion of students selecting STEM subjects from with lower levels of educational attainment showed that for the one jurisdiction that did provide data, I2S2 schools had higher levels of STEM participation. Low sample sizes and a lack of evidence from other jurisdictions make it difficult to directly attribute the results to the program. Many students interviewed for the Case Study spoke in general terms about their future education and career aspirations. This is not surprising given that students were aged from 10 to 14 years old. Therefore, it was too early to see results because many students participating in the program were not at the stage to choose senior high school subjects. Those that did talk about careers in STEM focused on medicine, engineering, or biology. A few students said that the program had increased their interest in a science career. Yeah, it did make a difference [in interest in pursuing science] because…there’s a base…a meaning of it, and you just understand that meaning more and more every time you have lessons. Most of the schools involved in the I2S2 Case Study were already involved in a number of STEM programs designed to encourage student interest and participation in STEM; nevertheless, there was some evidence to suggest that students engaged in I2S2 were involved in other STEM programs, such as ASSETS and the Indigenous Science Awards. It is likely that these schools have a general commitment to STEM and that they would be participating in these programs regardless of participation in the I2S2 program; therefore, directly attributing the results to the I2S2 program is difficult. The I2S2 program was effective in encouraging best practice, high expectations science inquiry education programs, although there was limited evidence showing the model was being adopted by other states and territories. As illustrated in the I2S2 Case Study, almost all teachers involved in the Case Study felt that their school encouraged a culture of high expectation, although they were conscious of the ability of one program to impact overall expectations of a student’s ability. Students noticed changes in teaching styles and commented that teaches were: • encouraging students to achieve academic success, • encouraging students to take initiative, • placing an emphasis on student effort, and • encouraging students to challenge themselves. It’s more about the effort than wanting us to get – they like they want us to get good grades, but they want us to try our best in order to get them, not try just because of that reason. There was limited evidence that the high expectations of students in I2S2 inquiries crossed over to other subjects, although some teachers did indicate they had discussions with students regarding why performance in science was not being matched in other subject areas. Overall, the program is recognised as a program expecting teachers to deliver a best practice pedagogy that places high expectations on students. Despite this recognition, it has not yet been adopted by state/territory education departments in a large-scale way. This lack of overarching support leaves the program, and the relationships built with the community, susceptible to teacher and school leadership turnover. Some universities showed interest in using I2S2 expertise and resources to build the capacity of pre-service teachers, for example, the University of Canberra invited the I2S2 to deliver training to pre-service teachers in late 2019; however, there has not been broader scale application of the approach. It, therefore, tends to represent a more targeted example of attempts to incorporate Aboriginal and Torres Strait Islander knowledges into the science curriculum. Schools supporting other STEM programs (e.g., ASSETS, CREST, Awards, PRIME Futures) Level of evidence: medium Increased number of Indigenous (and non–Indigenous) students pursuing STEM pathways, including in Years 10 to 12, university, and alternatives Level of evidence: medium Identification of ‘best practice’ in high expectations science inquiry education and teacher professional learning, and adoption of this ‘best practice’ by states and territories Level of evidence: medium Science Pathways for Indigenous Communities Program description Science Pathways for Indigenous Communities worked with remote schools and communities to develop integrated Two-way Science learning programs that connect Indigenous ecological knowledge with Western scientific knowledge and the Australian curriculum. The program was built around on-Country projects developed through strong community partnerships with Elders and, where they exist, Indigenous ranger groups, scientists, and land management organisations. Science Pathways for Indigenous Communities was built on Tangentyere Council’s ‘Land and Learning’ project and was delivered by the CSIRO in Western Australia and Tangentyere Council in the Northern Territory. A core focus of the program was to support teachers of primary and middle school students to engage with their local communities to develop curriculum and education plans that include on-Country Two-way Science activities. In 2019, the program shifted from supporting individual schools to a ‘cluster’ model that grouped geographically and culturally aligned schools and communities. The goal of this model was to encourage collaboration between networks of schools and communities to support regionally sustainable Two-way Science programs. Throughout 2020, in response to COVID-19 and the impact of travel restrictions and school closures, the program moved to an online learning model – the ‘Science Pathways Adult Learning Program.’ This online program provided professional development for school staff and community members (including Indigenous rangers and partner organisations) to develop and implement a Two- way Science program. The program was predominantly attended by educators from Western Australian schools/ communities but there was also engagement from Northern Territory and South Australia educators. Over 10 weekly, half-hour workshops, teachers were supported to improve key skills and knowledge. Using an action learning approach, educators and community members were encouraged to try new approaches and apply Two-way Science. The Science Pathways for Indigenous Communities Project Team also developed an online collaborative space for program participants to share education resources and build a community of practice. In 2021, CSIRO worked to expand the online learning program and rolled out a six-part, self- directed online learning experience. The new online modules were designed to spread the reach of the program further and have a focus on accessibility for educators new to the Two-way Science method. Figure 19. 2020 Snapshot of Science Pathways for Indigenous Communities participation numbers Figure 20. Locations of schools that have participated in Science Pathways for Indigenous Communities Figure 19. 2020 Snapshot of Science Pathways for Indigenous Communities participation numbers Figure 19. 2020 Snapshot of Science Pathways for Indigenous Communities participation numbers Figure 19. 2020 Snapshot of Science Pathways for Indigenous Communities participation numbers 971 Students 113 Teachers and 77 AIEOS 15 Schools Figure 19. 2020 Snapshot of Science Pathways for Indigenous Communities participation numbers 25 Communities Figure 20. Locations of schools that have participated in Science Pathways for Indigenous Communities Key achievements • As highlighted in Figure 19, in September 2020, the program was operating in 25 communities and 15 schools, working with 113 teachers, 77 AEIOs, and reaching 971 students. Across the full program, a cumulative total of 3,662 students have been engaged in the program. Figure 20 illustrates the locations of the participating schools. • Science Pathways for Indigenous Communities was successful in increasing educator capacity in Two-way Science and encouraging the delivery of on-Country, community-led learning activities tied to broader curriculum goals. • The program published key resources, including “Two-way Science: An integrated learning program for Aboriginal desert schools” (Deslandes et al., 2019) and collaborated with the Australian Curriculum, Assessment and Reporting Authority (ACARA) to develop four Illustration of Practice videos (ACARA, 2019a) to further guide educators in the Two-way Science approach. • Development of a Remote Travel Guide (a risk assessment tool) for use within CSIRO. • The Western Australian Department of Education has developed the Two-Way Science Initiative to scale the Two-way Science approach to Indigenous and non-indigenous schools across the state. Measuring success As outlined in the Science Pathways for Indigenous Communities Case Study (Cherry, Banks, Gilbert, & Fidler, 2020), the monitoring and evaluation team incorporated community feedback and learnings from the Case Study to re-design the program’s logic statement to ensure it captured the key priorities of educators and community members. The updated program logic was also used to inform a revised Impact Statement (Appendix B). The key outcomes were: Short-term outcomes • Strong, effective partnerships established with schools and other stakeholders. • Increased community and parental engagement. Intermediate outcomes • Increased teacher capacity in Two-way Science using Country contexts and cultural competence. • Increased student engagement and attendance. • Increased aspirations (in students), sense of value, and school belonging. Long-term outcomes • Enhanced student results. • The program’s Two-way Science learning approach and resources are culturally responsive, community- based, and embedded across school curriculums. Based on community feedback and program priorities, a decision was made to not pursue the following outcome areas included in the original Impact Statement: • School culture of high Traditional ecological knowledge and STEM expectations leveraged for literacy and other subjects. • Centres of excellence in two-way STEM education. • University teacher training using teacher professional development, Department of Education extending the model to other remote schools. One item listed under ‘Impacts’ was moved to a long-term outcome. Data are not available to report against this outcome: • Alternative STEM career pathways such as rangers, Parks and Wildlife, CSIRO cadet as well as university pathways become expected pathways in remote communities. Evaluation methods The outcomes of the Science Pathways for Indigenous Communities project were assessed using a variety of sources: • A multi-site Case Study evaluation of the program was conducted in 2019 to identify implementation learnings and assess progress towards outcomes, as described in an initial Impact Pathway. Three remote Aboriginal schools/communities participated in the evaluation from across the Northern Territory and Western Australia. • Qualitative data were collected at several points in time during 2018 and 2019. The findings were primarily based on interviews and focus groups with 90 participants, comprising students, families, community members, educators, program partners, and program staff. • School employees at four Western Australia schools participating in the online Adult Learning Program were asked to complete pre- and post-professional development surveys and an end of term survey. • Program monitoring and feedback from CSIRO program staff. Key findings of the Science Pathways for Indigenous Communities program The key evaluation findings of the Science Pathways for Indigenous Communities program are summarised in Figure 21. Figure 21. Science Pathways for Indigenous Communities – progress towards outcomes Partnership development was one of the key components of the Science Pathways for Indigenous Communities program. The results from surveys and case study interviews indicated that the program had a transformative impact on the formation of strong effective partnerships between schools, local rangers, Elders, and program staff. As reported in the Science Pathways for Indigenous Communities Case Study (Cherry et al., 2020), the program helped to: • Create a shared understanding between participating stakeholders, including local teachers and traditional knowledge holders. • Ensure that schools were able to connect with and build on the many existing partnerships between educators and local communities. • Align existing activities often occurring across multiple different programs to broader learning outcomes and increase planning and coordination. • Build a platform for more frequent and effective Two-way Science activities. • Strengthen relationships between non- Indigenous teachers and Aboriginal and Islander Education Officers (AIEO). Activities, such as program staff participating in local community activities, were seen as crucial to building connections and laying the platform for relationships based on mutual trust and understanding. The relationships and attitudes of teachers also were seen by some community members as crucially important to the success of Two-way Science teaching. Strong, effective partnerships established with schools and other stakeholders Level of evidence: high Figure 21. Science Pathways for Indigenous Communities – progress towards outcomes 1. Strong, effective partnerships established with schools and other stakeholders2. Increased community and parental engagement5. Increased aspiration, sense of value and school belonging3. Increased teacher capacity in Two‐way Science using On‐Country contexts and cultural competence4. Increased student engagement and attendance7. The program’s Two‐way Science learning approach and resources are culturally responsive, community‐ based and embedded across school curriculums: 6. Enhanced student results 71 per cent of respondents said that the Adult Learning Program moderately or substantially increased their ability to collaborate with Indigenous community and Aboriginal education workers (n = 21) There’s definitely a closer relationship with the school, [and rangers] and the school itself has that same vision and ideal of being community-focused first, and relationships…in terms of collaboration and networking, and structurally, or institutionally, that has helped. But also, personally, and with individuals and family members, it’s—again, I guess it’s just tightened up, or re-invigorated, or reinforced those kinds of things. If you have a really strong, functioning, planned engagement with a ranger program, then that becomes what you do as a school. And so, when a new principal or someone takes over, they can go, okay, well we’ve got his meeting coming up, and we’ve got this happening next term—it’s much harder to change direction. And so, if you have really engaged local Traditional owner groups and really engaged ranger programs and have an active plan that’s underway, then I think that leads to sustainable outcomes. I don’t think it depends on how new the teacher is entirely; I think it depends on their attitude and interests, how they build relationships with the assistant teachers they’re working with, and how mutually supportive those relationships are. I’ve really enjoyed being able to ask my AIEO ‘what do you want us to teach?’ and to see her confidence grow in teaching the students her knowledge. As well as building relationships between educators and local community stakeholders, the Science Pathways for Indigenous Communities Program team forged enduring relationships with schools, teachers, and community groups. Feedback from educators indicated that program staff had a key role in facilitating collaboration, for example, through start-of-term planning workshops and encouraging schools to take a cross-curriculum approach. The strength of these relationships was emphasised during the COVID-19 pandemic when face-to-face visits and workshops were restricted. In this new environment, educators and key stakeholders were increasingly responsible for maintaining the momentum of Two-way Science in their communities, albeit with ongoing support from the program team. There is a significant body of evidence that demonstrates that engaging parents/carers in their children’s education improves education outcomes and student engagement levels (Council of Australian Governments Education Council, 2019; Higgins & Morley, 2014). Engagement comes in a myriad of different forms that reflect the unique cultures and circumstances facing each school and community, including parental classroom participation, on-Country learning with Elders, and community involvement in school governance. The results from the Case Study demonstrate that the Science Pathways for Indigenous Communities program had an effective impact on school engagement with families and members of the broader community (Cherry et al., 2020). Program staff highlighted the existing and often longstanding engagement schools had with some families and groups within each community. Across the program, increased or improved community and/or parental engagement was noted in the following areas: • Families, communities, and Elders were engaging with teachers and the school for Learning on Country trips. • Community leadership, building shared goals, and including Aboriginal knowledge holders throughout curriculum development and term planning activities. • Community and family engagement before, during, and after Learning on Country trips. • Creation of a welcoming school environment and safe space for communities and families which, in turn, facilitated interaction and enriched student learning experiences. Both [two ways] are important to the kids and even to us. We got the kids involved in both ways. We just teaching the white people way of science, like going to school you’re learning to read and write and to do math. Here as Elders, we got them out in the bush teaching them about culture, what our family taught us. I think it helps them a lot because for their children too, they can pass that onto the next generation. So that’s very important. Passing it down. They’ve [the community] become more involved in the school, and they’ve had a lot more control and leadership in the program with teachers. They’ve led the way, and that’s really important, I think. It’s good for self-esteem, identity, trust between the two, between school and them. We’re all in this together. I sort of see it as them [Elders and community members] now realising that they can have some ownership of what the students learn and that we’re [the school/ teachers] happy to help facilitate that. That helps bring them in, that they know that their knowledge and what they want to teach the students is important to us and…how teachers [can] help them do it. Increased community and parental engagement Level of evidence: high The survey findings specific to the Science Pathways Adult Learning Program further highlighted how teachers and educators have involved members of the local Aboriginal community in Two-way Science planning and teaching activities. As detailed in the Science Pathways for Indigenous Communities Case Study (Cherry et al., 2020), there is strong evidence to support the finding that the program had a transformative impact on the ability of educators to undertake Two-way Science inquiries using on-Country contexts. In the Northern Territory, Aboriginal educators tended to take the lead and were able to build on their existing networks and capabilities. In-language Learning and Development workshops also improved the ability of educators to plan and undertake Two-Way Science in collaboration with non-Indigenous classroom teachers. Across participating Northern Territory and Western Australian communities, feedback showed that the program was able to improve the science skills of teachers who may have not seen themselves as strong in that area. These benefits are not restricted to non-Indigenous staff, with participants also noting the impact on AIEOs. Particularly the teachers for the primary school, they’re not specialists, they’re not science people [the teachers], and [the program facilitator] has got them into this, interested them in science. It makes them better teachers because, all of a sudden, they’re learning words, they’re learning context—that’s really value-adding big time. It has given me a new perspective on how to teach science to support Indigenous students and me becoming the student. For our AIEO, I have watched him grow in confidence in teaching the children, sitting with them, telling them stories, talking about their behaviours, and just encouraging the students to care for their country. Across the program, face-to-face support for teachers was seen as critical in building skills and cultural confidence. This support combined with increased opportunities for structured on-Country learning helped to build teacher connections and confidence in engagement with local Aboriginal and Torre Strait Islander groups. Teachers learning from Elders and allowing space for local Aboriginal knowledge holders to lead the teaching was identified as an important benefit of the program. We don’t know what’s going on in your business, but we know how pivotal it is. And that’s going to impact on here [school] but guess what, that’s okay, this is [the community’s] culture and we, to the best of our ability, can advocate for that. Program participants did highlight the practical realities of working in remote schools and emphasised the logistic challenges associated with taking students on Country. With many factors playing into the Two-way Science, ability to go on Country, transportation, Community Business (e.g., Sorry Business), it is always hard to plan ahead. We do not always have the same access to the above-mentioned aspects of Two-way Science as less remote schools. Find it difficult to organise trips out to Country - only have one vehicle which limits the number of adults and students that come out. Increased teacher capacity in Two-way Science using country contexts and cultural competence Level of evidence: high Looking specifically at the Adult Learning Program, program staff indicated that they were receiving regular evidence that participants were applying the Two-way Science skills in their communities. The evidence included weekly feedback to the team, participants sharing stories about their successes/what worked, and posting pictures on the private social media group. This is supported by survey data from the Adult Learning Program which showed that 85 per cent of respondents said the program moderately or substantially increased their ability to teach Two-way Science. While many educators preferred face-to-face development opportunities, the online learning modules were important in sustaining the program during a difficult and unprecedented time. Three outcomes relating to students’ experiences of the program have been grouped together. As detailed in the Science Pathways for Indigenous Communities Case Study (Cherry et al., 2020), evidence showed that the program had an effective impact on students across these three areas, although the strength of evidence did vary, with teacher perceptions of student engagement and aspirations as well as gains in student learning showing the strongest results. By comparison, there was limited quantitative data to support changes in student attendance, and indeed, attendance was impacted upon by factors outside of the control of the project team, and thus, has become less of a program focus over time. Similarly, student results judged by standardised school reporting were not seen as the best way to understand the learning impact of the program on students of remote communities, many of whom do not speak English as their first language or dialect. Educator and program team feedback on student outcomes and aspirations highlighted that, generally, students were more engaged during Two-way Science activities and Learning on Country. Students are more engaged in learning as a result of the Science Pathways program. The Two-way approach means that the students are able to work outside on Country with family and Elders. As the students are learning through family stories and family knowledge, they have a real connection to what is being taught and how they retain this information. We assess the student performance against themselves. How are you improving? We don’t really care how you improve compared to someone else. How are you, what have you learnt, what you’ve achieved. It makes the ranger program, and the potential employment through the ranger program, much more visible and accessible and aspirational. Many community members agree that young people in this community who are looking for these opportunities have a clearer pathway. Students themselves drew comparisons between Learning on Country and regular, classroom-based learning. Yeah, but it’s [Learning on Country] a bit different [than learning in the classroom] because when you go outside…Elders actually show you the things that they do, instead of showing it on the screen. They can show it in real life. That’s why I like going outside very much because you get to learn more things outside, and you get to see the actual things to actually believe it. One of the key goals of the program was to share resources and support educators to link Two-way Science activities to their broader curriculum. The program focused on delivering easy-to-use resources that could be adapted to fit the unique needs and strengths of each school and were matched to local community priorities. As explained in the Science Pathways for Indigenous Communities Case Study (Cherry et al., 2020), a key focus in the development of resources was centring local language and culture and linking these to curriculum priorities. In each community, first-language speakers were engaged in the process of creating resources that documented Aboriginal ecological and cultural knowledges. Often, science resources were the impetus to embed local culture and language across the curriculum. Feedback from principals specifically highlighted that the resources and support provided by Science Pathways for Indigenous Communities were key to supporting the broader language and culture program at their school. Embedding the program into the school curriculum also helped overcome the common problem of staff turnover in remote schools and contributed to the sustainability of the program in schools. Increased student engagement and attendance Level of evidence: medium Enhanced student results Level of evidence: medium Increased aspirations (in students), sense of value, and school belonging Level of evidence: high The program’s Two-way Science learning approach and resources are culturally responsive, community- based, and embedded across participating school curriculums Level of evidence: high As a graduate teacher especially, coming out and having the Two-way Science as a structure for integrated learning, because it’s not just science, it’s HASS [Humanities and Social Sciences], as well and English and maths too, is such a huge benefit. Like, it’s a really good framework to centre your learning around, and so it makes teaching a lot easier to plan engaging stuff. It’s really good. Two-way comes first, and then we go through the curriculum and work out how we’re going to fit. There’s always a way to link it. The Two-way Science Initiative As of May 2021, the Western Australian Government was planning to implement the Two-way Science Initiative, which would support schools to build partnerships with local Aboriginal communities to develop integrated, culturally responsive learning programs that connect the Western Australian science curriculum to Aboriginal knowledges. The Western Australian Government reports that: Two-way Science provides an opportunity for all students to learn about science from the world’s oldest continuing cultures. The Initiative aims to: • build respect for Aboriginal histories, cultures, people, knowledges, and experiences amongst school staff and students, and • strengthen Aboriginal student wellbeing, engagement, and achievement. The foundation of a Two-way Science learning program is cultural and ecological knowledge taught by local Aboriginal people to students on Country and in the classroom. Two-way Science Project Schools (Project Schools) will participate in a program of professional learning co-designed by the Department in partnership with local and regional Aboriginal organisations. Small teams from each Project School (teachers, Aboriginal and Islander Education Officers, school leaders, and Aboriginal community members) will engage in an ongoing action-learning process through professional learning workshops, an annual conference, and online webinars. Schools will document and share learning with other schools as the Initiative grows. Two-way Science Lead Schools (Lead Schools) will provide a profession-based model of support, sharing expertise and innovations in culturally responsive practices across schools and networks. A related focus will be on identifying local vocational education and training, career, and employment opportunities for students. Phase 1 of the Two-way Science Initiative (proof of concept) from July 2021 to December 2022 will involve three clusters of five schools in targeted education regions in Western Australia. The Western Australian Two-way Science Initiative builds on the publication of a number of resources designed to spread the Two-way Science pedagogy, including: • Two-way Science: An Integrated Learning Program for Aboriginal Desert Schools (Deslandes et al., 2019). This practical teacher guidebook was designed to build the capacity of teachers to undertake Two-way Science.16 The book was collaboratively developed by Science Pathways for Indigenous Communities school communities in Western Australia and had input from program staff in the Northern Territory • Publication of four ‘Illustrations of Practice’ videos featured on ACARA’s’s Australian Curriculum website (ACARA, 2019a). These illustrations showcase the Two- way Science education approach and help teachers from across Australia incorporate the Aboriginal and Torres Strait Islander Histories and Cultures cross- curriculum priority in the Australian curriculum. Figure 22 showcases example pages from the downloadable case study on the Wiluna Remote Community Schools Nganamarra (Malleefowl) science inquiry which forms part of one Illustration of Practice. 16 In March 2021, CSIRO’s publishing team confirmed that in 2021 they had sold 1,050 copies of the print edition, and it had been downloaded 2,308 (PDF document) and 144 (epublication) times. Figure 22. Example pages from the Malleefowl Science Inquiry Illustration of Practice Guide Figure 22. Example pages from the Malleefowl Science Inquiry Illustration of Practice Guide Figure 22. Example pages from the Malleefowl Science Inquiry Illustration of Practice Guide PRIME Futures Program description PRIME Futures was a mathematics program that worked with teachers in mainstream metropolitan and regional schools with higher-than-average Indigenous student populations. The program used the YuMi Deadly Maths approach (YDM) developed and delivered by the YuMi Deadly Centre at Queensland University of Technology (QUT).17 YDM is a mathematics pedagogical framework covering all strands of the Australian Mathematics Curriculum from Foundation (Prep) to Year 9. The framework is founded on the RAMR cycle, Reality- Abstraction-Mathematics-Reflection (RAMR), and has a strong focus on increasing teachers’ capacity to understand Aboriginal and Torres Strait Islander culture and incorporate it into mathematics lessons. PRIME Futures operated as part of the Indigenous STEM Education Project from 2014 to 2019. The YuMi Deadly Centre closed in 2019 following completion of the program. YDM was implemented through the PRIME Futures program with a combination of centrally organised professional development and more informal, school- based activities. YDC provided training to teacher- trainers (five professional development workshops over a two-year period). Teacher-trainers then trialled the pedagogy in their own classes and trained other teachers in their schools. To facilitate this process, schools were invited to prepare a whole-of-school plan to implement the pedagogy in their school. Participating schools were grouped in clusters to enable a staged roll-out of the PRIME Futures Program. shows the locations of participating schools. Key achievements • As illustrated in Figure 23, prior to the conclusion of the program, YDM was being implemented across 10 clusters, comprising 62 schools and approximately 379 teacher-trainers (this includes teacher-trainers who attended any professional development out of the five). • Across the program, 22 per cent of students in participating schools were from an Aboriginal and/ or Torres Strait Islander background, indicating that the program had a potential reach of 6,975 Aboriginal and/or Torres Strait Islander students.18 • The program was instrumental in improving teacher capacity to provide engaging, culturally responsive mathematics instruction. Figure 23. End of program participation numbers, September 2019 Figure 23. End of program participation numbers, September 2019 Figure 23. End of program participation numbers, September 2019 Figure 23. End of program participation numbers, September 2019 32,317 Students 22% (6,975) Aboriginal and/or Torres Strait Islander Students 379 Teacher- Trainers 62 Schools Grouped into 10 clusters Figure 24. Locations of schools participating in PRIME Futures Figure 24. Locations of schools participating in PRIME Futures 17 The YDM resources and approach were initially developed as part of the Teaching Indigenous Mathematics Education project funded by the Queensland Department of Education and Training from 2010-2012. https://research.qut.edu.au/ydc/projects/prime-futures-program/ 18 This is a summary figure only as it is likely that a smaller portion of students in each a school have been taught YDM. Measuring success The updated PRIME Futures Impact Statement is provided in Appendix B. This Impact Statement has been updated to provide more detail on key inputs and activities, refine the pathways and timeframes to achieve change, and highlight community priorities. The key outcomes are: Short-term outcomes • Positive student engagement with new pedagogy. Intermediate outcomes • Improved teacher capacity and capability, and delivery of culturally responsive maths pedagogy. • School classrooms are culturally and socially safe. • Schools demonstrating strengthening relationships with parents/carers and local Aboriginal and/or Torres Strait Islander communities reinforced by comprehensive school plans. • Improved student results. Long-term outcomes • Increasing cohort of students choosing STEM subjects in Years 10 to 12. • Increased teacher and student pride in self, school, and community. • Whole-of-school implementation/delivery of best practice culturally responsive maths pedagogy. The following indicator was removed as it was no longer a program priority • Schools and students applying to Awards program, ASSETS, and other CSIRO programs. Evaluation methods As the key delivery and evaluation partner for the PRIME Futures program, the YuMi Deadly Centre at QUT collected quantitative and qualitative data on the PRIME Futures program. Key data collection methods included: • Teacher and principal surveys – educators were asked to complete a survey every six months, with the first survey completed between three and six months after the program start date (‘biannual surveys’) and the final survey undertaken on exit from the training program (‘exit survey’). The exit survey included several pre-post retrospective questions. Respondents were asked about program implementation, teacher capacity, and aggregate student engagement and academic achievement using a five-point rating scale. • Reflective journals – throughout their time in the program, teachers were asked to respond to open- ended questions in an online journal format. • Workshop evaluations. • YDC cluster coordinator reports. • Cluster reports prepared by the program team on completion of the active phase of training for each cluster of schools. Collected data were used to inform the 2019 Case Study Evaluation into the PRIME Futures program (Carter, Stuetz, Cottier, Nason, & Cooper, 2019). Given that the PRIME Case Study Report was published just after the program finished operating, there is limited new data to include in this evaluation report. The information presented below summarises the key data and aligns the Case Study findings to the outcomes in the program’s Impact Statement. Key findings of the PRIME Futures program The key evaluation findings of the PRIME Futures program are summarised in Figure 25. Data from teachers, principals, and YDC coordinators confirmed that the PRIME Futures program had a transformative impact on student engagement and was effective at improving student results. As outlined in the PRIME Futures Case Study (Carter et al., 2019) there was strong evidence to support increased student engagement from biannual and exit surveys. Improved student results are harder to demonstrate because individual data on student results were not collected. In place of such data, teachers were asked to share information on student outcomes in surveys and reflective journals. As illustrated in Carter et al. (2019), teacher perceptions of student understanding and engagement, defined as a measure of attention, curiosity, interest, optimism, and passion, improved from Survey 1 to Survey 3. These results were confirmed by the results of the exit surveys which asked teachers to reflect on student outcomes before and after PRIME Futures and showed that student engagement was most improved (n = 123, clusters 1 to 10). Asked in exit surveys about changes in observed student engagement for different groups, teachers recorded the highest improvements for students in the lower ability range and Indigenous students (results increased from ‘very little engagement’ to ‘somewhat’ for both groups, n = 122). The perception of increased Indigenous student outcomes was also supported by the principal exit survey results. These findings are consistent with other research conducted on YDM which shows that the program, while originally intended to benefit Aboriginal and/or Torres Strait Islander students, has strong benefits for students in schools in lower socio-economic areas (YuMi Deadly Centre, 2017). Figure 25. PRIME Futures – progress towards outcomes Figure 25. PRIME Futures – progress towards outcomes 6. Whole‐of‐school implementation/delivery of best practice, culturally responsive maths pedagogy3. Schools demonstrating strengthening relationships with parents and local Aboriginal and/or Torres Strait Islander communities reinforced by comprehensive school plans1. Positive student engagement with new pedagogy2. Improved teacher capacity/capability in implementation and delivery of culturally responsive maths pedagogy5. Increasing cohorts of students in Years 10‐12 pursuing maths & STEM pathways4. Improved student results Positive student engagement with new pedagogy Level of evidence: high Improved student results Level of evidence: high Increasing cohorts of students in Years 10-12 pursuing maths and STEM pathways Level of evidence: medium Throughout the year it has been amazing to see the engagement of all the students within my class…My students’ attitudes towards maths and learning have greatly changed. YuMi has been excellent for my Aboriginal and Torres Strait Islander students, but the program is also very beneficial for all the students in my class...I noticed a massive difference in my students’ engagement and their willingness to learn. The survey results highlighted perceived improvements in test results, from 14 per cent in Survey 1 to 33 per cent in Survey 3 (teacher survey n = 123, principal survey n = 42). This result was confirmed by findings from the teacher and principal exit surveys which both showed that average perceptions of student test results increased from ‘poor’ to ‘satisfactory’. Data were not available to assess any flow-on impacts on student subject selections in Years 10 to 12; nevertheless, the impact of the PRIME Futures program on students was emphasised by many teachers, who commented on students’ improved mathematics achievements and cognitive ability. Comparing pre- and post-test data shows that the students all had considerable improvement…I know my maths teaching has improved and my students’ learning has improved because of it. The evaluation results indicated that the program had a transformative impact on teacher capacity (training, resources, time) and capability (skills, knowledge, confidence). As highlighted in the PRIME Futures Case Study (Carter et al., 2019), teacher and principal survey responses showed improvement in all areas measured, with pedagogical skills and teacher confidence perceived as having the largest improvements. On average, teachers indicated that the program had led to moderate improvements in pedagogical skills and confidence.19 Mathematic knowledge and expectations of students and Indigenous knowledge were all somewhat improved (n = 124). Teacher perceptions of change were similar, although slightly above principal responses. Comments in teachers’ reflective journals indicated that teachers believed that the workshops improved their mathematical understanding and teaching skills or had led them to change their teaching approach. Participants also highlighted the value of resources provided during the professional development workshops. As shown in Figure 26, asked in the exit survey to reflect on their Improved teacher capacity and capability, and delivery of culturally responsive maths pedagogy Level of evidence: high * Teacher exit survey n = 125 Figure 26. Changes in teaching approaches: Teachers saw the biggest change in their use of RAMR and Big ideas Figure 26. Changes in teaching approaches: Teachers saw the biggest change in their use of RAMR and Big ideas 19 Results represent average responses for three biannual surveys and one exit survey for clusters 1 to 10 on a 1 to 5 response scale. teaching approaches before and after undertaking professional development workshops, teachers identified their use of RAMR and Big Ideas as the approaches that saw the biggest changes. The use of hands-on activities increased from “somewhat” to “moderately” while the use of worksheets and handouts decreased, as expected given the focus of the professional development. The YDM [workshops] are fabulous. Every time I participate in one, I feel my understandings of mathematical concepts deepening. I am finding better ways to teach my students and therefore feel like they are making greater gains in their learning. They have been instrumental in changing the way I teach each mathematical concept. Reports by cluster coordinators highlighted the difficulty of ensuring continuous attendance across the workshops, originally designed with the expectation that the same teachers would attend each workshop. The lower-than- expected number of teachers participating in all five professional development workshops potentially lowered the effectiveness of the program. Teachers have multiple demands placed on their time, and spending time away from the classroom involves significant preparation; acknowledging this, workshop content was modified so each professional development session could stand alone. I am beginning to be more confident in my approach to teaching students maths with hands-on materials. I am actively thinking about how I can get the students to physically make or solve maths problems and get them more familiar with maths concepts…I feel so much more confident as a maths teacher and look forward to increasing my confidence even further as I get to practice and refine my teaching in using the YDM pedagogy. Having now attended nine days’ worth of YuMi Deadly Maths professional development, I now feel like I have a better idea of the approach…I know my maths teaching has improved and my students’ learning has improved because of it. A cornerstone of the PRIME Futures program was that schools strengthened connections with their local Indigenous community, parents/carers , and families, and that these partnerships were supported by whole- of-school plans. This, in turn, flowed through to classroom YDM activities where teachers could build on relationships and improved cultural knowledge to incorporate Indigenous perspectives into their learning activities. The results published in the PRIME Futures Case Study indicated the PRIME Futures program had an effective impact on strengthening relationships with the local community (Carter et al., 2019). As outlined in the Case Study, many schools in the program already had strong links to the local Aboriginal and/or Torres Strait Islander community and existing programs in place to support these connections. The level of connection between YDM classroom activities and wider school programs to connect with local Aboriginal and/or Torres Strait Islander communities varied between schools. Indeed, some teachers saw their school’s existing activities as sufficient for incorporating Aboriginal and Torres Strait Islander culture. The results from the pre-post exit survey showed that participation had generally improved teacher knowledge of Aboriginal and Torres Strait Islander culture and community. Teachers rated their knowledge of ‘local Indigenous culture and community’ as ‘poor’ before participating in the YDM program and this increased to ‘satisfactory’ after the program (n = 125). Asked about their use of Indigenous contexts, 38 per cent of teachers indicated they had ‘used some Indigenous contexts’ (results from three biannual surveys n = 444). While these figures are lower than anticipated, it is encouraging that teachers perceived their use of Indigenous contexts in the classroom as doubling after participating in the program (exit survey n = 125). I asked parents to explain to me what it was like for them when they went to school and what they would like for their child to explore with maths. I had parents of Aboriginal children come in and do some maths with my kids. We did this outdoor, and the kids responded well. Aboriginal cultural and teaching and learning practices that support students are gradually being embedded more into our teaching. The integration of Aboriginal perspective to my lessons had a great impact on my teaching. Aboriginal students were more responsive to questions/tasks. Students were also more involved in discussions. The PRIME Futures program aimed to influence leadership and pedagogy at the school level to bring about school- level change in mathematics education. This is an emerging outcome area for the program. Whole-of- school systems-level change requires a coordinated effort, both within a school and between schools in a cluster. Schools demonstrating strengthening relationships with parents/carers and local Aboriginal and/or Torres Strait Islander communities reinforced by comprehensive school plans Level of evidence: medium As reported in the PRIME Futures Case Study (Carter et al., 2019), just about all survey respondents (97 per cent) indicated that they had trained one or more colleagues at their school and they believed the majority of their colleagues that used the YDM approach were interested in trying more YDM activities (teacher biannual and exit surveys n = 569). Evidence collected from surveys, reflective journals, and YDC coordinator reports suggested that successful school change was dependent on input and engagement from each school’s leadership team. Figure 27 highlights the key requirements YDC coordinators believe are essential for creating a supportive environment to enable teacher-trainers to spread the key concepts across their school. Where these components were not in place, teachers tended to confine their YDM practices to their classroom. • Support and high expectations from the leadership team. • Key personnel to drive the change and being retained in this role. • Systemic and regional focus supporting the change. • Time and money to support staff in making the change. (Carter et al., 2019, p. 75) Figure 27. Key components to support whole-of-school change We have started our whole-school implementation. Teachers have reacted positively to the professional development and have started implementing the philosophies of a RAMR in their lessons…Enthusiasm has developed in some lower school teachers who are seeing transferability of skills to other curriculum areas based on the idea of abstract concepts. As a whole school, we have committed to using YD maths as our ‘how to teach mathematics’...trying to ensure that all staff understand the RAMR cycle...I am increasingly trying different ways to support staff to implement YDM. Program staff and teachers both highlighted a key barrier to the dissemination of the YDM approach in schools is the challenge teachers face learning new pedagogies while balancing many often competing demands on their time. The results from the teacher biannual and exit surveys revealed that 45 per cent of teachers highlighted the lack of available time both of colleagues and of themselves as a key obstacle in sharing the YDM approach with colleagues. Thus, a key component in the overall success and uptake of the YDM approach at a school is the selection of staff that are willing to shoulder this extra responsibility and commit to a long-term process to see benefits My main challenge is trying to fit it in with the many other hats I wear in the school. Whole-of-school implementation/delivery of best practice, culturally responsive maths pedagogy Level of evidence: medium What are the whole-of-project outcomes and what elements have contributed towards achievement of these outcomes? Key messages • The Indigenous STEM Education Project has increased the confidence and capacity of teachers to provide culturally responsive, inquiry-based STEM education. • Across the Project, educators and project teams were able to build authentic and long-lasting relationships with families and communities to support students’ STEM journeys. These relationships have been key to teachers’ capability and confidence in integrating Aboriginal and Torres Strait Islander knowledges with the Australian STEM curriculum. • Students responded positively to the new pedagogy and across the six programs, there is strong evidence of increased engagement, results, and aspirations. The evaluation results also emphasise that hands-on, flexible teaching approaches that draw on student strengths and existing knowledge are important for all students, regardless of their cultural heritage. • The Indigenous STEM Education Project had mixed results embedding Aboriginal and Torres Strait Islander knowledges and an inquiry-based pedagogy at a whole-school level and across jurisdictions. The evaluation evidence highlights that whole-school change requires a coordinated and collaborative approach within schools that is not always possible given the many demands placed on educators. • Of the six programs, Science Pathways for Indigenous Communities has had the biggest impact on developing systems-level change through the Two-way Science Initiative currently being investigated by the Western Australia Department of Education. As explained previously, each program within the Indigenous STEM Education Project has its own Impact Statement. The overall Indigenous STEM Education Project also has an Impact Statement that focuses on the key outcomes expected across the programs. This Impact Statement has been revised during the preparation of this report. Key changes to the impact statement include: • Listing educators, school leaders, and communities as inputs. Without these key individuals invested in change and building a new path for Aboriginal and/or Torres Strait Islander STEM education, the programs would not have operated. • Highlighting relationships and connection building as key activities. Strong relationships are the core of all the Indigenous STEM Education Project programs. It is through thoughtful, considered, and committed work to build trusting relationships that the Project has been able to achieve the key outputs and outcomes. • Adjusting the timeframe of key outcomes, including specifically separating short-term, intermediate, and long-term outcomes. This change is important for setting realistic expectations about what can be achieved and when. Many of the programs that comprise the Indigenous STEM Education Project were operating in schools and communities with multiple and complex challenges. Time is needed to build trusting relationships between project teams and communities, establish shared goals, and design flexible programs that meet community needs. • Identifying outcomes that are most important to Aboriginal and/or Torres Strait Islander participants, their teachers, and their families, and focusing on outcomes that are more directly attributable to program-specific achievements, rather than outcomes heavily dependent on other factors. Measuring success The updated Impact Statement for the Indigenous STEM Education Project is provided in Appendix B. The key outcomes for the Project are listed below. Short-term • Innovative, place-based Indigenous contextualised and/or led STEM curriculum (inquiry-based) delivered in schools and a university and documented in school plans. • Increased teacher capacity in inquiry; place- based, hands-on curriculum development and delivery in an Indigenous context. Intermediate-term • Increased parental, family, and community engagement and recognition of role models. • Increased student engagement, attendance, results, and recognition. • Increased student aspirations, sense of value, cultural identity, and school belonging. Long-term • Schools are culturally competent in delivering Two-way Science in partnership with Elders, families, and communities and flow-on benefits to broader curriculum/teaching. • Jurisdictions, CSIRO, universities, and partners scaling up (to reach more locations/students). • More Indigenous (and non-Indigenous) students pursuing STEM education pathways, enrolment in STEM Years 10 to 12 and university, STEM careers, and leadership opportunities. The assessment of each outcome is based on a summative evaluation of the relevant results from each individual project, a judgement process informed by evidence, data collected, and the voices of program participants. Key findings of the Indigenous STEM Education Project Figure 28 highlights the achievement of all outcomes identified in the Impact Statement and assesses progress against these outcomes using the same transformative-effective-emerging scale used to measure progress against outcomes for the individual programs. Figure 28. Indigenous STEM Education Project – progress towards outcomes Figure 28. Indigenous STEM Education Project – progress towards outcomes Similar to the discussion of short- and medium-term outcomes presented in the third Evaluation report (Cherry et al., 2019), this section of the report groups Project outcomes into five categories: 1) teacher and school outcomes; 2) family and community outcomes; 3) student outcomes; 4) best practice, culturally responsive STEM education; and 5) STEM education sector outcomes. For each of these categories, the overall results are highlighted, along with a list of the key elements, or building blocks, for success. These elements are critical to the Indigenous STEM Education Project’s success and are instrumental in the achievement of project outcomes. Each category also includes a discussion of the important lessons learnt. Teacher and school outcomes 20 https://research.acer.edu.au/cgi/viewcontent.cgi?article=1010&context=professional_dev OVERALL RESULTS BUILDING BLOCKS FOR SUCCESS/SUCCESS FACTORS One of the key priorities of the Indigenous STEM Education Project was to support and resource teachers to deliver hands-on, inquiry-based STEM lessons that integrate local Aboriginal and Torres Strait Islander knowledges and are tied to the broader curriculum. Evidence shows that the program had a transformative impact on the capacity of teachers to deliver an innovative, place-based, indigenous contextualised STEM curriculum. Through train-the-trainer models in PRIME Futures and professional development in I2S2 and Science Pathways for Indigenous Communities, the project worked to shift teaching pedagogies and drive whole-of-school change. Survey results and interview data from PRIME Futures, I2S2, Science Pathways for Indigenous Communities, and Bachelor of Science (Extended) highlight that educators have gained confidence in applying culturally responsive, inquiry-based approaches in the classroom and that they valued the resources and support provided. These short-term outcomes are having a transformative impact on the way teachers deliver STEM education. Common across all teacher professional development programs is the reality that it takes time for teachers to fully adopt and master new approaches and that repeated practice helps build confidence. Evaluation findings are consistent with ACER’s recommendations regarding primary school STEM, that teachers require ongoing support and the ability to reflect on their practice.20 Research also highlights the reciprocal relationship between teacher confidence in STEM and student engagement and interest (Skamp, 2012). Many teachers also reported increased confidence in bringing Aboriginal and Torres Strait Islander perspectives into their classrooms and that confidence is impacted by the overall level of connections teachers and schools have with local Aboriginal and/or Torres Strait Islander communities. Broader changes to how schools deliver STEM are a medium to longer-term outcome. As outlined by Luke et al. (2013) in their evaluation of the Strong Smarter Learning Communities Project, evidence on school reforms suggests that it takes three to five years to implement a cycle of reform that generates school level gains, and this extends to four to six years to see improvement in Aboriginal and/or Torres Strait Islander education, assuming continuity of leadership and appropriate succession planning. The Indigenous STEM Education Project has been effective in encouraging whole-of-school change, seeing success in some schools and less progress in others. The face-to-face I2S2 model required schools to commit to changing the science pedagogy across all year levels. This whole-school approach was effective in changing practices at a number of schools but was less a focus when the program moved to training individual teachers online. The PRIME Futures teacher-trainer model worked to spread new culturally responsive teaching methods within a school. Evidence shows it works best when supported by invested school leadership. At the tertiary level, the academics at the University of Melbourne had success in implementing an engaging first-year science curriculum for the Bachelor of Science (Extended) program that incorporated Aboriginal and Torres Strait Islander knowledges. The Indigenous STEM Awards was also successful in highlighting best practice teaching and celebrating schools leading the way in Two-way, inquiry-based STEM. • Facilitating the creation of peer- to-peer teacher networks and school-level networks (clusters of schools). • Long-term professional development programs for teachers to support them in their journey to change student and school-level outcomes. • Provision of resources that are easy to access, inexpensive, adaptable, and require limited preparation. • Face-to-face learning is crucial and needed to establish relationships. • Whole-of-school, multi-year planning. IMPORTANT LESSONS • Teachers are required to juggle multiple priorities and often lack the time and resources to apply new approaches. • Whole-of-school change takes time, committed leadership, and a community of teachers willing to engage with each other and key community partners. • Even with an overall shift to digital, it is important to build strong, personal relationships among program staff and educators. OVERALL RESULTS KEY ELEMENTS FOR SUCCESS The six programs that comprised the Indigenous STEM Education Project all set different expectations in terms of parental, family, and community engagement. Evaluation data from across the programs indicates that the Indigenous STEM Education Project had a transformative impact on parental, family, and community engagement and recognition of role models. An intermediate outcome, the involvement of parents/carers , families, and communities takes time and is centred on building authentic and long-lasting relationships between program teams, educators, and family/community members. For all programs, connections to local Aboriginal and Torres community groups and knowledge holders were important to help teachers and program staff integrate local Aboriginal and Torres Strait Islander perspectives into the STEM curriculum and activities. There is strong evidence from Science Pathways for Indigenous Communities, PRIME Futures, ASSETS, I2S2, and the Bachelor of Science (Extended) foundation year program indicating that involving Elders and cultural patrons in school programs has a positive impact on students, increasing the relevance of the STEM content and increasing student engagement. The Indigenous STEM Awards further emphasised the role of community and the importance of building STEM role models by focusing on celebrating Award recipients in their own communities. Data showed that recipients strongly valued this opportunity. Science Pathways for Indigenous Communities encouraged schools to create a welcoming environment where families and communities were able to partner in their children’s learning journeys. While many participating schools already had strong relationships with their community, the Science Pathways for Indigenous Communities program helped educators draw on these connections to conduct successful on-Country learning activities and deliver Two-way Science. ASSETS program staff also successfully worked with participants and their families to encourage students to pursue their STEM ambitions. Staff were effective in providing one-on-one advice to parents/carers and connecting them with different opportunities for their students. • Safe and welcoming school environment. • Staff willing to engage with community members in a variety of different settings. • Program staff working with families on an individual level and making personal contact. • Drawing on existing connections of Aboriginal Education Workers/ Community Education Officers and inviting them to be partners in curriculum development. • Families and communities need to feel consulted and authentically engaged. IMPORTANT LESSONS • Can be challenging for some schools to deepen their connections with the community when there are limited existing connections. • Staff turnover, both in program teams and in schools can present significant challenges for the continuity of relationships. Student outcomes OVERALL RESULTS KEY ELEMENTS FOR SUCCESS At its centre, the Indigenous STEM Education Project is about increasing Aboriginal and/or Torres Strait Islander students’ interest in STEM and encouraging students to pursue STEM education and career paths.21 The Project influenced student outcomes in two ways, through direct program support (ASSETS, Bachelor of Science (Extended)) and by better equipping teachers and schools to deliver STEM (I2S2, PRIME Futures, Science Pathways for Indigenous Communities). This latter approach, with its focus on culturally responsive, hands-on, inquiry-based activities, sought to change how whole cohorts of students were taught STEM. Evidence on student experiences as part of the Indigenous STEM Education Project shows that the project had a transformative impact on student engagement, aspirations, results, and recognition. Closer examination of these results shows that I2S2, PRIME Futures, and Science Pathways for Indigenous Communities, with their focus on inquiry-based teaching and integrating Aboriginal and Torres Strait Islander knowledges with STEM were particularly successful in increasing student engagement and increasing students’ interest in STEM. First-year students in the Bachelor of Science (Extended) program also reported high levels of engagement and connection with the Indigenous contexts included in the science curriculum at the University of Melbourne. Retention of participants in the Bachelor of Science (Extended) program is one of the central student outcomes, facilitated in part by the multi-layered support offered by the University. ASSETS, the Indigenous STEM Awards and the Bachelor of Science (Extended) Program were also able to draw on strong alumni involvement, as mentors, program patrons, and key program advocates, to help mentor subsequent cohorts of students. These ongoing Alumni relationships, both within and between different cohorts often provided valuable personal and professional connections for students. Evaluations of PRIME and I2S2 highlight that they both contributed to increased academic results for low-achieving Aboriginal and/or Torres Strait Islander students and low-achieving non-Indigenous students. These results emphasise that hands-on, flexible teaching approaches that draw on student strengths and existing knowledge are important for all students, regardless of their cultural heritage. The Indigenous STEM Awards and ASSETS targeted high-achieving students and sought to encourage students in their STEM journeys. By offering multi-layered, individualised personal development assistance to students, and connecting them with further STEM opportunities, scholarships, and work placements, these programs have been successful in increasing students’ STEM engagement and ambitions. Feedback from ASSETS alumni indicates that following participation in the summer school, students were more aware of prerequisite subjects and that many were changing their subject selection for Years 11 and 12 to align with their STEM goals. ASSETS students also spoke of the strong connections they were able to make with a cohort of Aboriginal and/or Torres Strait Islander STEM role models. • Practical, hands-on, on-Country activities connected to culture and curriculum. • Ongoing, multi-layered support from program staff. • Facilitating safe (culturally and socially) places for young people to explore their cultural heritage and identities. • Exposure to STEM career opportunities including work placements, traineeships, and cadetships. • Promoting role models and connecting with Alumni networks that show students viable education and career pathways and opportunities. • Valuing students’ existing knowledge and cultural connections. 21 In line with research that shows that student outcomes are shaped over years, not months (Luke et al., 2013), changes in student results, engagement, and STEM aspirations are an intermediate outcome for the Indigenous STEM Education Project. Family and community outcomes Student outcomes continued OVERALL RESULTS KEY ELEMENTS FOR SUCCESS Across the programs, there was less evidence to assess the impacts on STEM career and education pathways. An exception being some ASSETS participants who indicated they were more aware of or were going to change their Year 11 and 12 prerequisite subjects following participation in the summer school. IMPORTANT LESSONS • Culturally responsive, hands-on, inquiry-based STEM education does not just benefit Aboriginal and/or Torres Strait Islander students, it is useful for all students. • Staff turnover has a key impact on the ability of schools to adopt new teaching pedagogies. • Student outcomes are influenced by multiple factors, many of which are out of the control of the programs within the Indigenous STEM Education Project. It is, therefore, important to be realistic in expectations of change and also offer multi-layered opportunities and support. • Programs should be flexible in how they define success for students; for example, students transferring out of the Bachelor of Science (Extended) program to another tertiary program should not be seen as a negative outcome. Best practice, culturally responsive STEM education outcomes OVERALL RESULTS KEY ELEMENTS FOR SUCCESS While there is no standard definition of culturally responsive education in Australia, the Indigenous STEM Education Project adopts an approach similar to Dunbar and Scrimgeour (2009, p.7), which “privileges the cultural identity and social background of students as essential starting points when designing curriculum and approaches to learning”. Encouraging teachers and schools to shift to a culturally responsive STEM pedagogy is a process that requires flexibility and time. It relies on building teachers’ confidence and strengthening connections to local Aboriginal and Torres Strait Islander knowledge holders. It cannot be a one-size-fits-all approach. An overall assessment of the impact of the Indigenous STEM Education Project indicates that the project was effective in promoting the uptake of best practice, culturally responsive STEM pedagogies. Results varied by program, school, and on an individual teacher-by-teacher basis. Schools and educators also came into the programs with different backgrounds and varying levels of cultural competency. The Science Pathways for Indigenous Communities program worked with remote schools that had long-standing connections to local Aboriginal and Torres Strait Islander communities, and curriculum development was often guided by Elders and knowledge holders. The program was successful in using these relationships to embed Indigenous ecological knowledge in the Australian curriculum. The Science Pathways for Indigenous Communities program team also worked closely with communities to document and disseminate the approach through a book, “Two-way Science: An Integrated Learning Program for Aboriginal Desert Schools” (Deslandes et al., 2019) and through the four Illustration of Practice videos available on the ACARA website showing students learning Aboriginal traditional ecological knowledge through the Australian curriculum (ACARA, 2019a). These resources are an important legacy for the Project and help spread the pedagogy to a wider audience. I2S2 and PRIME Futures, by comparison, encouraged teachers and schools to build connections with local Aboriginal and Torres Strait Islander communities and changed how educators delivered STEM education. There was mixed evidence on the effectiveness of these programs to introduce a culturally responsive STEM pedagogy, with some teachers reporting changing their strategies and classroom approaches while other teachers were at earlier stages in their journeys and not ready to commit to full application of the approach. Feedback from teachers participating in I2S2 and PRIME Futures also showed that teachers felt more able to implement new pedagogies when their school leadership actively supported and invested in the change. These findings are similar to those for the Bachelor of Science (Extended) program in relation to the integration of Aboriginal and/or Torres Strait Islander knowledge into the first-year science curriculum at the University of Melbourne. • Community input in curriculum development and planning. • Acknowledging and utilising the expertise and knowledge of Elders, Traditional Owners, and other community members. • Regular on-Country trips connected with classroom activities and broader curriculum. • Applying an open-ended science inquiry approach based on real- world contexts. • Using Aboriginal and Torres Strait Islander knowledges from across the country, but allowing space for local knowledges to be included at the community level. • Building communities of practice among teachers and school- based clusters to help embed new pedagogies and encourage knowledge and skill sharing. IMPORTANT LESSONS • Staff turnover, particularly in remote schools can significantly impact the delivery of a culturally responsive STEM pedagogy. • Schools and educators should establish a long-term commitment to engage with Aboriginal and/or Torres Strait Islander groups and knowledge holders and create avenues for ongoing communication and knowledge sharing. STEM education sector outcomes OVERALL RESULTS KEY ELEMENTS FOR SUCCESS While the overall focus of the Indigenous STEM Education Project was to improve results for Aboriginal and/or Torres Strait Islander students in STEM, the Project also sought to contribute to broader systems-level change in how STEM is taught in Australian schools (and universities) so more students could benefit from the approach. The Indigenous STEM Education Project has made significant progress in increasing the number of students reached by the program. With over 23,000 students and 2,700 teachers participating, the Project has exceeded initial expectations on reach which, in turn, has led to improved STEM outcomes for many more students. Feedback from program managers indicates that overall, the Indigenous STEM Education Project was on balance more focused on increasing participation numbers and continuing the operation of the program through seeking new funding opportunities, rather than driving systems-level changes in jurisdiction/institutional practice and creating cultural shifts/embedding new knowledge broadly within the education sector. Notwithstanding this lack of overall focus, there is some evidence to show that the approach is becoming embedded within CSIRO education programs, partner organisations, and at least one state education jurisdictions. The Bachelor of Science (Extended) continues to thrive at the University of Melbourne, with knock-on effects among faculty and across the University. The School of Biosciences appointed their first tenured Indigenous STEM teaching specialist academic and more broadly staff report growing momentum for the incorporation of Aboriginal and/or Torres Strait knowledges across the core science curricula. More information on the Faculty of Science’s Indigenous research, engagement and inclusion initiatives is available at https://science.unimelb.edu.au/about/ indigenous-research-engagement-inclusion. As highlighted in the Science Pathways for Indigenous Communities results section of this report, as of May 2021, the Western Australian Government was planning to implement a Two-way Science Initiative, which would support schools to build partnerships with local Aboriginal communities to develop integrated culturally responsive learning programs that connect the Western Australian curriculum: science to Aboriginal knowledges. This initiative builds on the work undertaken by the Science Pathways for Indigenous Communities Program and the publication of a number of key resources documenting the Two-way Science approach. Education jurisdictions in Queensland expressed interest in the I2S2 approach, and the I2S2 program team provided training in the I2S2 scope and sequence to the Queensland Solid Pathways team.22 The Indigenous STEM Education Project has also had some inquiries from tertiary training institutions looking at using the approach in training pre-service teachers but, to date, there has been no direct uptake of the pedagogy by these organisations. This is despite research illustrating the importance of working with pre-service teachers to build confidence in new STEM pedagogies and embed change (Cooper, Kenny, & Fraser, 2012). Similarly, opportunities exist to deliver training to better equip pre- service teachers to constructively engage and consult with Aboriginal and/or Torres Strait Islander families and communities (Moreton-Robinson, Singh, Kolopenuk, & Robinson, 2012). • Policy makers and jurisdictions invested in scaling up programs and approaches. • Program teams liaising with local jurisdictions and sharing results and resources. IMPORTANT LESSONS • Programs aiming to achieve broad, systems-level change need a clearly articulated strategy to influence and inform education systems and partners. 22 Solid Pathways supports eligible high-performing Aboriginal and Torres Strait Islander students in Year 4 to Year 9 for continued academic success, university studies, and access to the Queensland STEM Academy. What key factors influence the sustainability of the Indigenous STEM Education Project? Key messages • Despite the conclusion of the Indigenous STEM Education Project, there are a number of key resources that remain available to support teachers and schools interested in applying inquiry-based science, improve cultural awareness, and implement culturally responsive teaching practices. • The six programs that comprise the Indigenous STEM Education Project all worked to build and strengthen relationships between educators, students, and Aboriginal and/or Torres Strait Islander communities. The relationships and connections formed throughout the Project are core to the Project’s legacy and fundamental to improving outcomes for Aboriginal and/or Torres Strait Islander students in STEM. • The Project’s monitoring and evaluation activities have contributed to the body of evidence for what works for both Aboriginal and Torres Strait Islander and non-Indigenous STEM education in Australia. • Overall, the Project did not place a high priority on diffusion of the pedagogy and approach within the broader education sector, although University of Melbourne faculty have engaged nationally with tertiary educators on Indigenous science. Notwithstanding the lack of overall focus, evidence on the effectiveness of the Science Pathways for Indigenous Communities Program has been used by the Western Australian Government in the creation of a new Two-way Science Initiative in the state. There are also examples of key learnings from the Indigenous STEM Education Project being applied in other CSIRO STEM projects. 2021 marked the conclusion of the Indigenous STEM Education Project. It was an ambitious project with an important goal to increase interest and academic achievement among Aboriginal and/or Torres Strait Islander students in STEM subjects and related professions. Throughout the six years of the Project, it reached 23,904 Aboriginal and/or Torres Strait Islander students, 2,768 teachers and assistant teachers, and 603 schools. Although elements of the Project will continue to be delivered by CSIRO and the University of Melbourne, the conclusion of the Project leaves a large gap in the Australian education sector. It is, therefore, useful to assess the sustainability of the key outcomes and consider the Project’s enduring benefits. Sustainability is assessed as both the ongoing operation of the Project and the broad diffusion of the approach (Chelimsky, 2019). The ongoing operation of the Project involves the “continued use of components and activities for the achievement of program outcomes” (p. 72), while diffusion relates to the spread of a project’s principles, pedagogies, and approaches within the broader education sector, also referred to as systems-level change. Assessing the likely sustainability of a program involves judgements about the future and the influences of complex and changing factors out of the immediate control of a program, including, for example, public and policy interest in Indigenous STEM education (Chelimsky, 2019). Table 6 provides a summary of the key elements that contribute to the ongoing sustainability of the Project; these elements will be discussed in turn below. Table 6. Key elements contributing to the ongoing sustainability of the Project ONGOING/ENDURING PROGRAM ELEMENTS Provision of resources and creating a platform for long-term skill development Relationships and connections Building a body of evidence for what works in STEM education Bachelor of Science (Extended) DIFFUSION OF PEDAGOGY AND APPROACH Western Australia Department of Education and Science Pathways for Indigenous Communities Young Indigenous Women’s STEM Academy Generation STEM (Indigenous STEM education component) Bachelor of Science (Extended) – national tertiary educators Ongoing/enduring program elements At the time of publication, no further funding for the Indigenous STEM Education Project had been sourced; consequently, the programs within the Indigenous STEM Education Project will not be continued in their current form (except the Bachelor of Science (Extended)). This outcome, while not unexpected, is nevertheless somewhat disappointing for the students, teachers, and program staff that have worked hard to improve the outcomes of Aboriginal and/or Torres Strait Islander Students in STEM. Despite the lack of funding, a number of key elements will remain available to support the delivery of inquiry- based, hands-on, culturally responsive STEM education. Provision of resources and creating a platform for long-term skill development Many of the programs that comprised the Indigenous STEM Education Project provided resources, training programs, and tools to teachers and schools to help implement inquiry-based, Two-way Science that links Aboriginal and Torres Strait Islander knowledges into primary and high school STEM curriculums. I2S2, PRIME Futures, and Science Pathways for Indigenous Communities developed comprehensive learning programs, lesson plans, assessment tools, and classroom resources to guide educators in learning and applying the approach. Following the completion of the Indigenous STEM Education Project, some of these resources will continue to be available online for future cohorts of teachers to utilise. These resources place Aboriginal and Torres Strait Islander STEM contexts at the fore and encourage teachers and jurisdictions to rethink the historical representations of Aboriginal and Torres Strait Islanders in the Australian curriculum, improve cultural awareness, and implement culturally responsive teaching practices (Patrick & Moodie, 2016). Key examples include: • Five I2S2 online training modules plus an additional ‘Teaching Mathematics’ module will be available for teachers wishing to learn and apply the approach. • Science Pathways for Indigenous Communities will have a number of online modules available for teachers to participate in. These modules build on the resources contained in the Two-way Science: An Integrated Learning Program for Aboriginal Desert Schools book. • The Science Pathways for Indigenous Communities program also worked with ACARA to develop four ‘Illustrations of Practice’ videos that remain available. • QUT maintains an online presence for the Yumi Deadly Centre, which provides access to a large body of information on the YDM learning approach and pedagogy, example inquiries, and teaching strategies that formed the basis for PRIME Futures. • The Indigenous STEM Awards website continues to serve as an online resource for high-achieving Aboriginal and/or Torres Strait Islander STEM students and professionals. The resource is a useful first point of contact for those looking to find Aboriginal and/or Torres Strait Islander STEM specialists. • The ASSETS project team produced a work placement guide and operational framework that can be used by schools, STEM organisations, universities, and businesses. • The Bachelor of Science (Extended) continues to be fully operational and funded by the University of Melbourne. Teachers that participated in I2S2 and PRIME Futures also received guidance in how to share the new pedagogy and approach with fellow educators. This train-the- trainer model helped spread the program beyond those who initially completed the professional learning activities. In considering the impacts of the train-the- trainer model in an educational setting, it is important is recognise that there are a number of factors that impact the ability of participating teachers to train their colleagues in inquiry-based STEM, including the multiple demands already placed on teachers and the level of support provided by school leadership. Relationships and connections Building lasting and authentic relationships among key project stakeholders is an important legacy of the Indigenous STEM Education Project and a cornerstone of the Project’s success. There is a strong body of evidence that shows that relationships matter for education, particularly for student engagement (Pomerantz, Moorman, & Litwak, 2007; Quin, 2017) and that highly effective schools tend to have high levels of parent and community engagement (Lonsdale & Anderson, 2012). The Indigenous STEM Education Project worked to create new and strengthen existing relationships, and empower teachers and schools to engage with Aboriginal and/or Torres Strait Islander communities in a culturally respectful and meaningful manner. The Science Pathways for Indigenous Communities program worked with educators and community leaders to develop whole-school plans to integrate Indigenous ecological knowledge into classrooms and worked with communities, Indigenous ranger organisations, land management and conservation groups, and schools to bring knowledge holders and Elders into the education and curriculum planning process. These relationships were fundamental to the program achieving its core outcomes. I2S2 and PRIME Futures also worked with educators to increase opportunities for schools to collaborate with the local Aboriginal and/Torres Strait Islander communities. While some schools and teachers lacked expertise and knowledge to integrate Indigenous knowledges and perspectives, other schools had considerable success in partnering with other schools, integrating Aboriginal and Torres Strait Islander knowledges, and improving their cultural awareness (Cherry et al., 2019). As well as developing relationships between schools and communities, the Indigenous STEM Education Program also sought to increase connections between Aboriginal and/or Torres Strait Islander students interested in STEM. Feedback from participants in ASSETS indicates that the relationships and connections formed by students, and the understanding that students are connected to broader networks, is a key benefit of the program. The Indigenous STEM Awards also helped to connect students and teachers to a broader network of Aboriginal and/ or Torres Strait Islanders interested in STEM education. The delivery of the PRIME Futures and Bachelor of Science (Extended) programs was contracted to and/or delivered by third parties (QUT and University of Melbourne, respectively). These third parties were responsible for liaising with students and educators as needed throughout the course of the funding period, and both delivered very successful programs. While this arrangement helped to facilitate the delivery of the programs, CSIRO could have done more to establish and further develop relationships with key stakeholders in these programs, and in the case of PRIME Futures, maintain contacts with schools and teachers after the program concluded. Building a body of evidence for what works in STEM education The Indigenous STEM Education Project has worked from the outset to embed evaluation and monitoring activities into the delivery of the six individual programs. The Project has adopted an action learning approach that includes an ongoing feedback loop between participants, evaluators, and the program team. This process has been documented in a series of published evaluation reports and case studies. The reports add to the body of evidence showing what works in Indigenous STEM Education; they show that teachers, when supported by invested school leadership and given adequate training and resources, are interested and able to adapt to new pedagogies. Evidence from I2S2 also shows that these pedagogies not only lead to improvements in the educational outcomes of Aboriginal and/or Torres Strait Islander students, but improve outcomes for all students, particularly students traditionally classified as low achieving. The evaluation reports also support existing research which highlights the need to shift the educational narrative from a deficit driven approach to a high expectation model that centres Aboriginal and Torres Strait Islander knowledges (Sarra, Spillman, Jackson, Davis, & Bray, 2020). Diffusion of pedagogy and approach As well as improving teacher confidence and capacity in delivering culturally responsive STEM and contributing to improved results for Aboriginal and/or Torres Strait Islander students, the Indigenous STEM Education Project also sought to contribute to broader systems- level change in how STEM is taught in Australian schools and scale programs to reach more students. For these goals to be achieved, there must be sustained evidence that the pedagogy achieves the desired outcomes supported by a long-term plan to encourage broad take-up of the approach. In the event ongoing funding was not available, as occurred, this would help lay the platform for further achievement of outcomes and broader-scale impact beyond the immediate programs. Feedback from program managers indicates that overall, the Indigenous STEM Education Project had a focus on continuing the operations of the project, for example, seeking extensions to, or additional sources of funding. It is the consensus of the program teams and evaluators that less focus was placed on overall systems change, embedding and diffusing ideas and pedagogies within the broader education sector. This result is not unique to the Indigenous STEM Education Project and emblematic of a broader lack of focus applied to systematically changing the level of cultural responsiveness within the Australian education sector and integrating Aboriginal and/Torres Strait Islander knowledges and learning into the Australian curriculum (Jackson- Barrett & Lee-Hammond, 2018). Evaluators identified three instances of the Indigenous STEM Education Project influencing broader education approaches or program design. These are discussed in turn below. Western Australia Department of Education and Science Pathways for Indigenous Communities As mentioned previously in this report, as of May 2021, the Western Australian Government was planning to implement a Two-way Science Initiative, which would support schools to build partnerships with local Aboriginal communities to develop integrated, culturally responsive learning programs that connect the Western Australian curriculum: science to Aboriginal knowledges. The Initiative aims to: • build respect for Aboriginal histories, cultures, people, knowledges, and experiences amongst school staff and students, and • strengthen Aboriginal student wellbeing, engagement, and achievement. Similar to Science Pathways for Indigenous Communities, the foundation of the Two-way Science learning program is cultural and ecological knowledge taught by local Aboriginal people to students on Country and in the classroom. The initiative draws on evidence of the effectiveness of the Science Pathways for Indigenous Communities Program and the important work of the program team and educators to document and share the approach. Young Indigenous Women’s STEM Academy The body of evidence gathered for the Indigenous STEM Education Project has contributed to the development of new programs both internal and external to CSIRO. In 2020, the National Indigenous Australian’s Agency (NIAA) committed to funding CSIRO’s Young Indigenous Women’s STEM Academy for 10 years to increase the number of Aboriginal and/or Torres Strait Islander young women undertaking studies and careers in STEM. Evidence from the Indigenous STEM Education Project and CSIRO’s experience in delivering Indigenous STEM education programs were key factors in the NIAA’s support of this project. Generation STEM Based on evidence about the effectiveness of the Indigenous STEM Education Project, the $25 million 10-year Generation STEM initiative (funded by the New South Wales Government and delivered by CSIRO), which works with industry, government, and education sectors to support, train, and retain students in STEM career pathways, will also incorporate a specific Indigenous STEM education component. Generation STEM will employ an Indigenous STEM Education Project Officer to co-design the Indigenous STEM education program elements within its program. The program will be based on the approach and pedagogies applied in the Indigenous STEM Education Project. Adapting to COVID-19 The COVID-19 pandemic has disrupted education across Australia and worldwide. Nearly four million Australian students were required to move to alternative delivery models and commence distance and digital learning (Department of Education, Skills, and Employment, 2020). Throughout 2020, school closures and travel restrictions had a major impact on the operations of the Indigenous STEM Education Project and resulted in program postponements, cancellations, and/or other disruptions. Like all education initiatives, the six programs that comprise the Indigenous STEM Education Project had to adapt content and quickly design alternative delivery options while also working to ensure students and teachers were supported during this challenging time. Digital teaching requires teachers to adapt to one or often multiple online systems, and it “requires new or adapted pedagogies, management and organisation of content, institutional support, and new or adapted ways of engaging and interacting with students” (Department of Education, Skills, and Employment, 2020b, p. 11). These requirements placed heavy demands on teachers and students already adapting to new approaches to teaching STEM under the Indigenous STEM Education Project. A recent study by the Australian Department of Foreign Affairs and Trade examined the lessons from Australia’s history of remote education and distance teaching for mitigating the impacts of COVID-19 (Dabrowski, Nietschke, Taylor-Guy, & Chase, 2020). The study highlighted five principles of good practice in remote education; see Figure 29. These principles are key elements of the Indigenous STEM Education Project and became core components in allowing the Project to continue to succeed in an otherwise difficult environment. Table 7 provides a summary of how each of the individual programs within the Indigenous STEM Education Project were able to adapt their programs to continue supporting teachers and students in their STEM education and career journeys.23 For more information on the Impact of the COVID-19 pandemic on the Indigenous STEM Education Project, see CSIRO’s submission to the Education in Remote and Complex Environments: Standing Committee on Employment, Education, and Training (Mulcahy, 2020). In many instances, the switch to online has seen programs broaden their reach as more teachers have access to the training materials. Counteracting this has been the loss of face-to-face interactions, an element identified by teachers and program teams as important for establishing trusting relationships and in-person modelling of new pedagogical approaches to STEM. 23 The BHP Foundation funding for PRIME Futures and the Bachelor of Science (Extended) programs both finished at the end of 2019 so they are not included in the table. Figure 29. Principles of good practice in remote learning Table 7. Program-level adaptations due to COVID-19 PROGRAM COVID-19 ADAPTATIONS Science Pathways for Indigenous Communities Travel restrictions and temporary school closures in remote Aboriginal communities due to COVID-19 resulted in the postponement of community visits to deliver adult learning programs. Consequently, the Project team developed new delivery modes to support teachers in their continued Two-way Science learning. These modes included the development of weekly online workshops for adult learning and regular meetings with school staff. The Project team also developed an online collaborative space to encourage sharing and collaboration between teachers and build a community of practice. A number of closed social media groups were also created to allow communities to continue learning on Country in family groups. Resource packs, including activity cards and other printed materials connecting Indigenous knowledge to the science curriculum and school program, were also distributed to communities. I2S2 The program was well-positioned to switch to an online delivery model as a result of the disruption caused by COVID-19. In 2019, I2S2 began rolling out online modules intended to support and increase the reach of the face-to-face program. Following the closure of schools in early 2020, the project team shifted all their teacher professional learning activities to online. The program team also converted one of the Year 5 inquiries into a ‘home-based’ inquiry suitable for parents/carers and/or caregivers to deliver to children in a home learning environment. ASSETS The 2020 ASSETS summer school, held in January, was unaffected by COVID-19. Travel disruptions and workplace closures beginning in March 2020 necessitated the postponement of 12 planned work experience placements for ASSETS students and a virtual work placement program was held for the duration of 2020. The program team worked to organise an online event for the cohort of students that attended the January 2020 summer school to further build and strengthen connections between students. Throughout the year, the program team continued to provide one-on-one advice and guidance to students to help them navigate the many education challenges 2020 presented. Bachelor of Science (Extended) The program created a home practical program and posted kits to students so they could continue their science skills during lockdown (see https://www.youtube.com/watch?v=pVGq0Tfg7jU&t=210s for a video summary). As COVID-19 restrictions increased and students returned to their home states, an opportunity was identified to maintain the hands-on nature of practical classes using the services of Australia Post (i.e., kits of postable laboratory equipment for interdisciplinary science). Students used this kit of equipment to complete their practical program that included an environmental survey (chemistry and biology), simple mechanics (physics and biology), and investigations of living things (biology and geography). All students that began the degree program at the start of Semester 1 2020, successfully completed their first semester despite the flip to online learning and the disruption wrought by the onset of the pandemic. All but one student successfully completed their second semester (one student made the decision to withdraw due to Sorry Business). Indigenous STEM Awards The COVID-19 pandemic forced the Indigenous STEM Awards program team to adjust their schedule of planned events for 2020. The program was able to hold a planned in-person pre-announcement gathering in February 2020 with Award winners. This event provides an opportunity for the program staff to co-design each winner’s presentation/celebration ceremony. From mid-March 2020, all presentations were moved to an online platform. Online events typically included a virtual panel with key Aboriginal and/or Torres Strait Islander STEM professionals and were web cast to those unable to attend. A key component of the Awards program has been the ability to celebrate and recognise winners in their communities. The shift online, therefore, represented a significant challenge for program staff and participants. It also provided an opportunity for program staff to become familiar with new streaming technologies and to look at new and innovative ways to bring people together in virtual spaces to celebrate the achievements of Aboriginal and/ or Torres Strait Islanders in STEM. Conclusion and Recommendations Evaluations of initiatives with multiple and complex components, such as the Indigenous STEM Education Project, are often challenging endeavours. With six distinct programs and participants from across the country, it has been a significant undertaking to document and assess the impact of each program and track progress against key outcomes. This report has looked at evidence collected over the six years of the project and has identified key areas where individual programs have excelled, as well as areas for improvement. It has also identified common themes and factors that are important to creating an environment that allows Aboriginal and/or Torres Strait Islander students to excel in STEM education. The evaluation has also developed a strong body of evidence showing what works to empower teachers to deliver hands-on, inquiry-based STEM lessons that integrate Aboriginal and Torres Strait Islander knowledges into the curriculum. The recommendations outlined in Table 8 are based on the evaluation of the overall Indigenous STEM Education Project. They are designed in response to the key themes and lessons learnt across the six programs that comprise the Indigenous STEM Education Project. The recommendations are considered important for others looking to effect change in Aboriginal and/or Torres Strait Islander education and STEM. For detailed recommendations on the six individual programs, readers are directed to the respective Case Study Evaluation Reports. Table 8. Recommendations arising from the evaluation of the Indigenous STEM Education Project THEME RECOMMENDATIONS Teacher and school outcomes 1. Evidence from I2S2, PRIME Futures, and Science Pathways showed that teachers often enter professional learning experiences with different levels of existing knowledge. Where possible, programs should tailor professional learning activities to the appropriate level and if necessary, offer multiple levels rather than adopt a one-size-fits-all approach. 2. Moving programs into an online environment increases program reach and access for many teachers and schools but this is often at the cost of relationships and development of a community of committed practitioners supporting one another to adopt a new approach. Programs should prioritise the creation of communities of practice and other avenues for information sharing between individual teachers and school clusters. Where possible, face-to-face interactions between program teams and educators should be included in all programs to reinforce and strengthen relationships and build personal connections and trust. Family and community outcomes and cultural connectedness 3. Schools and teachers have different levels of connection to local communities and Aboriginal and/ or Torres Strait Islander families and communities. Program teams need to support educators to liaise with local community groups and knowledge holders to develop joint educational goals that include community-driven indicators of success and support the integration of place-based Aboriginal and/or Torres Strait Islander knowledges into the curriculum. It should also be recognised that this is an ongoing process, culture and knowledge are dynamic, and learning/knowledge sharing should be ongoing between stakeholders. 4. Examining how stronger connections between educators and families/communities can contribute to educator retention should be explored24. Students 5. Many of the programs within the Indigenous STEM Education Project benefited from strong alumni involvement, as mentors, program patrons, and key program advocates. All programs should have in place formalised plans to further develop these valuable alumni resources 6. Evidence from I2S2 and PRIME Futures shows that programs working with low-achieving Aboriginal and/or Torres Strait Islander students are also effective at improving outcomes for non-Indigenous students. The application of an inquiry-based, hands-on, culturally responsive pedagogy should, therefore, be viewed as best practice regardless of students’ cultural backgrounds. 24 Quality of relationships with the wider community, and presence/quality/nature of informal support are important contributors to teacher retention in Australia (Mason & Poyatos Matas, 2015). THEME RECOMMENDATIONS Best practice, culturally responsive STEM education 7. The in-person I2S2 model worked with schools and teachers to roll out the inquiry-based science pedagogy across whole year levels within schools. This required a long-term commitment to the approach from teachers, school leadership, and program teams, and was effective at creating an environment that supported teachers to introduce change. Programs looking to change teaching pedagogies should engage with school leadership as well as individual teachers to embed new approaches within a school environment and ensure teachers are provided with time and resources to reflect on and refine their practice. 8. Evaluation results from the Indigenous STEM Education Project support research findings indicating that embedding change in school and tertiary environments takes time (5+ years). Programs should, therefore, adopt a long-term, realistic horizon for affecting change and work with funding partners to secure necessary resources over this time frame. STEM education sector outcomes 9. Programs working to facilitate lasting, systems-level change in the education sector should have a strategy to engage with key policy stakeholders and government departments. 10. Evidence-based teacher professional learning programs should also include learning opportunities for pre-service teachers to assist in embedding the new pedagogy. Project management and evaluation 11. Programs should have a formalised plan to embed action learning, effect continuous improvement, and ensure that key learnings are shared across programs. 12. Programs looking to influence long-term outcomes should establish robust monitoring systems to enable longitudinal tracking of student pathways and, where relevant, results. References Anderson, I. (2020) For the first time, Closing the Gap has a higher education target – Here’s how to achieve it. The Conversation. Retrieved from https://theconversation. com/for-the-first-time-closing-the-gap-has-a-higher- education-target-heres-how-to-achieve-it-147984 Attia, M., & Edge, J. (2017). Be(com)ing a reflexive researcher: A developmental approach to research methodology. Open Review of Education Research, 4, 33–45. https://doi.org/10.1080/23265507.2017.1300068 Aubusson, P., Brady, L., & Dinham, S. (2004). Action learning: What works? A research report prepared for New South Wales Department of Education and Training. Retrieved from https://www.cese.nsw. gov.au/images/stories/PDF/Eval_Rep/Schools/ Action_Learning_What_Works_2004-05_QTAL.pdf Australian Curriculum Assessment and Reporting Authority. (2019a). Australian curriculum: Aboriginal and Torres Strait Islander histories and cultures Illustrations of Practice. Retrieved from https://www.australiancurriculum. edu.au/resources/aboriginal-and-torres-strait-islander- histories-and-cultures/illustrations-of-practice/ Australian Curriculum Assessment and Reporting Authority. (2019b). Australian curriculum. Retrieved from https://www.australiancurriculum.edu.au Australian Institute of Aboriginal and Torres Strait Islander Studies. (2020). AIATSIS code of ethics for Aboriginal and Torres Strait Islander research. Canberra. https://aiatsis. gov.au/sites/default/files/2020-10/aiatsis-code-ethics.pdf Australian Institute of Aboriginal and Torres Strait Islander Studies. (2021). Map of Indigenous Australia. Retrieved from https://aiatsis.gov.au/explore/map-indigenous-australia Baik, C., Naylor, R., & Arkoudis, S. (2015) The first year experience in Australian universities: Findings from two decades, 1994-2014. Melbourne Centre for the Study of Higher Education, The University of Melbourne. Retrieved from https://melbourne-cshe. unimelb.edu.au/__data/assets/pdf_file/0016/1513123/ FYE-2014-FULL-report-FINAL-web.pdf Ball, R. (2015). STEM the gap: Science belongs to us mob too. The Australian Quarterly, 86 (1), 13-19. Banks, C. (2021). What works in Indigenous STEM education. Canberra: CSIRO. Retrieved from https://www. csiro.au/-/media/Education-media/Files/Indigeneous- STEM/Evaluation/21-00120_SER-EDU_FACTSHEET_ WhatWorksIndigenousSTEMEducation_WEB_210414.pdf Banks, C., & ASSETS Alumni. (2020). ASSETS destination interviews data report. Canberra: CSIRO. Retrieved from https://www.csiro.au/-/media/Education-media/Files/ Indigeneous-STEM/Evaluation/20-00365_SER-EDU_ ASSETS-DestinationInterviewsReport_WEB_201117.pdf Banks, C., Fidler, J., Gilbert, C., & King, M. (2021). Inquiry for Indigenous Science Students (I2S2) case study evaluation report. Canberra: CSIRO. Retrieved from https://www. csiro.au/en/education/Programs/Indigenous-STEM- Education-Project/Monitoring-and-evaluation Banks, C., Mudhan, P., & Fidler, J. (2019). Indigenous STEM Education Project: Aboriginal Summer School for Excellence in Technology and Science (ASSETS): Evaluation case study outcomes report. Canberra: CSIRO. Retrieved from https://www.csiro.au/-/media/Education-media/Files/ Indigeneous-STEM/Evaluation/Aboriginal-Summer-School- for-Excellence-in-Technology-and-Science-Case-Study.pdf Carter, M., Stuetz, A., Cottier, C., Nason, R., & Cooper, T. (2019). Indigenous STEM Education Project, Purposeful Rich Indigenous Mathematics Education (PRIME) Futures program: Case study report. Brisbane, QLD: YuMi Deadly Centre, QUT. Retrieved from https://www.csiro.au/-/media/Education- media/Files/Indigeneous-STEM/Evaluation/20-00149_ SER-EDU_PRIMEFuturesCaseStudy_WEB_200522.pdf Chelimsky, E. (2019). Improving the match between sustainability questions and evaluation practice: Some reflections and a checklist. New directions for evaluation, 2019 (162), 69-86. Cherry, K., Banks, C., Gilbert, C., & Fidler, J. (2020). Science Pathways for Indigenous Communities case study evaluation report. Canberra, Australia: CSIRO. Retrieved from https://www.csiro.au/-/ media/Education-media/Files/Indigeneous-STEM/ Science-Pathways/Science-Pathways-for-Indigenous- Communities---Case-study-evaluation-report.pdf Cherry, K., Banks, C., Mudhan, P., & McNeilly, C. (2019). Indigenous STEM Education Project third evaluation report: September 2014-September 2018. Canberra, Australia: CSIRO. Retrieved from https://www.csiro.au/ en/education/Programs/Indigenous-STEM-Education- Project/Monitoring-and-evaluation/Third-report Coalition of Peaks. (2020). National agreement on Closing the Gap. Retrieved from https:// coalitionofpeaks.org.au/wp-content/uploads/2021/04/ ctg-national-agreement-apr-21-1-1.pdf Commonwealth of Australia. (2015). National innovation and science agenda. Department of the Prime Minister and Cabinet. Retrieved from https:// www.industry.gov.au/sites/default/files/July%20 2018/document/pdf/national-innovation-and- science-agenda-report.pdf?acsf_files_redirect Cooper, G., Kenny, J., & Fraser, S. (2012). Influencing intended teaching practice: Exploring pre-service teachers’ perceptions of science teaching resources. International Journal of Science Education, 34(12), 1883-1908. Council of Australian Governments Education Council. (2019). Alice Springs (Mparntwe) education declaration. South Carlton: Education Council Secretariat. Retrieved from https://docs.education. gov.au/documents/ alice-springs-mparntwe-education-declaration Dabrowski, A., Nietschke, Y., Taylor-Guy, P., & Chase, A. (2020). Mitigating the impacts of COVID-19: Lessons from Australia in remote education. Australian Council for Educational Research. Retrieved from https:// research.acer.edu.au/learning_processes/32/ Dandolo Partners. (2020). Evaluation of early learning and schools initiatives in the national innovation and science agenda: Report to the Department of Education. Retrieved from https://www.dese.gov.au/download/4826/ evaluation-early-learning-and-schools-initiatives-national- innovation-and-science-agenda/7195/document/docx Department of Education, Skills, and Employment. (2020a) Inquiry-based learning. Canberra: Australian Government. Retrieved from https://www.dese.gov.au/ australian-curriculum/national-stem-education-resources- toolkit/introductory-material/inquiry-based-learning Department of Education, Skills, and Employment. (2020b). Education in remote and complex environments – Impact of the COVID-19 pandemic on home learning and teaching. Submission 49 - Supplementary submission. Canberra: Australian Government. Retrieved from https:// www.aph.gov.au/DocumentStore.ashx?id=f6aa9cc5- 6ca5-4551-939b-2a73f2fe3361&subId=679844 Department of the Prime Minister and Cabinet. (2021). Closing the Gap in partnership: Students archive their full learning potential. Canberra: Australian Government. Retrieved from https://www.closingthegap.gov.au/ students-achieve-their-full-learning-potential Deslandes, C., Deslandes, S., Broun, D., Hugh, C., Walsh, F., Bradshaw, F., & Griffith, J. (2019). Two-way Science: An integrated learning program for Aboriginal desert schools. Victoria, Australia: CSIRO Publishing. Dew, A., McEntyre, E., & Vaughan, P. (2019). Taking the research journey together: The insider and outsider experiences of Aboriginal and non-Aboriginal researchers. Forum: Qualitative Social Research, 20(1), 18. http://dx.doi.org/10.17169/fqs-20.1.3156 Dunbar, T., & Scrimgeour, M. (2009). Cultural competence in Aboriginal education services delivery in Australia: Some lessons from the Aboriginal health service sector. AARE International Education Research Conference Canberra, 2009. Retrieved from https://www.aare. edu.au/data/publications/2009/scr091593.pdf Education Council. (2015). National STEM school education strategy. Canberra. Retrieved from http:// www.educationcouncil.edu.au/site/DefaultSite/ filesystem/documents/National%20STEM%20 School%20Education%20Strategy.pdf Higgins, D., & Morley, S. (2014). Engaging Indigenous parents in their children’s education. Resource sheet no. 32. produced by the Closing the Gap Clearinghouse. Canberra: Australian Institute of Health and Welfare & Melbourne: Australian Institute of Family Studies. Retrieved from https://www.aihw.gov.au/getmedia/277aadfe-f766-42ed- b93b-86da29d1917e/ctgc-rs32.pdf.aspx?inline=true Hogarth, M. (2017). Speaking back to the deficit discourses: A theoretical and methodological approach. The Australian Educational Researcher, 44(1), 21–34. https://doi.org/10.1007/s13384-017-0228-9 Innovation and Science Australia. (2017). Australia 2030: Prosperity through innovation. Canberra: Australian Government. Retrieved from https:// www.industry.gov.au/data-and-publications/ australia-2030-prosperity-through-innovation Jackson-Barrett, E. M., & Lee-Hammond, L. (2018). Strengthening identities and involvement of Aboriginal children through learning on Country. Australian Journal of Teacher Education, 43(6), 86-104. Jackson-Barrett, E., Price, A., Stomski, N., & Walker, B.F. (2015). Grounded in country: Perspectives on working within, alongside and for Aboriginal communities. Issues in Educational Research, 25(1), 36–479. http:// www.iier.org.au/iier25/jackson-barrett.html James, R., Krause, K-L., & Jennings, C. (2010). The first year experience in Australian universities: Findings from 1994 to 2009. Centre for the Study of Higher Education, The University of Melbourne, and Griffith Institute for Higher Education, Griffith University. Melbourne. Retrieved from https://citeseerx.ist.psu.edu/viewdoc/ download?doi=10.1.1.723.8190&rep=rep1&type=pdf Kelaher, M., Luke, J., Ferdinand, A., Chamravi, D., Ewen, S., & Paradies, Y. (2018). An evaluation framework to improve Aboriginal and Torres Strait Islander health. Melbourne: The Lowitja Institute. Retrieved from https://www.lowitja. org.au/content/Document/evaluation-policy-brief.pdf Lonsdale, M., & Anderson, M. (2012). Preparing 21st century learners: The case for school-community collaborations. ACER Occasional Essays-March 2012. Retrieved from https://research.acer.edu.au/cgi/viewcontent. cgi?article=1015&context=policy_analysis_misc Luke, A., Cazden, C., Coopes, R., Klenowski, V., Ladwig, J., Lester, J., Macdonald, S., Phillips, D., Shield, P., Spina, N., Theroux, P., Tones, M., Villegas, M., & Woods, A. (2013). A summative evaluation of the Stronger Smarter Learning Communities Project: 2013 report (Volume 1 and 2). Australia: Queensland University of Technology. Retrieved from https://eprints.qut.edu.au/59535/1/ SSLC_Evaluation_Report_2013_Volume_I_copy.pdf Ma Rhea, Z., Tynan, M., Banks, C., Phillipson, S., Sadler, L., & McNeilly, C. (2018). Indigenous STEM Education project second evaluation report: September 2014-September 2017. Canberra, Australia: CSIRO. Retrieved from www. csiro.au/en/Education/Programs/Indigenous-STEM/ Monitoring-and-Evaluation/Second-Report Martin, A.J., Ginns, P., Anderson, M., Gibson, R., & Bishop, M. (2021). Motivation and engagement among Indigenous (Aboriginal Australia) and non- Indigenous students. Educational Psychology, 41(4), https://doi.org/10.1080/01443410.2021.1879994 Martin, K. L., & Mirraboopa, B. (2003). Ways of knowing, ways of being and ways of doing: A theoretical framework and methods for Indigenous research and Indigenist re-search. Journal of Australian Studies, 27(76), 203–214. http://dx.doi.org/10.1080/14443050309387838 Mason, S. & Poyatos Matas, C. (2015). Teacher attrition and retention research in Australia: Towards a new theoretical framework. Australian Journal of Teacher Education, 40(11), 45-66. http://dx.doi.org/10.14221/ajte.2015v40n11.3 Moreton-Robinson, A., Singh, D., Kolopenuk, J., & Robinson, A. (2012). Learning the lessons?: Pre-service teacher preparation for teaching Aboriginal and Torres Strait Islander students. Melbourne: Australian Institute for Teaching and School Leadership. Mudhan, P., Banks, C., Gilbert, C., & Sadler, K. (2019). Indigenous STEM Education Project Bachelor of Science (Extended): Evaluation case study outcomes report. Canberra, Australia: CSIRO. Retrieved from https://www.csiro.au/-/media/Education-media/ Files/Indigeneous-STEM/Evaluation/Bachelor-of- Science-Extended-Case-Study-Report.pdf Muir, S., & Dean, A. (2017). Evaluating the outcomes of programs for Indigenous families and communities (CFCA Practice Resource). Melbourne: Australian Institute of Family Studies. Retrieved from https:// aifs.gov.au/cfca/publications/evaluating-outcomes- programs-indigenous-families-and-communities Mulcahy, M. (2020). Addendum to CSIRO submission 19/700 Education in Remote and Complex Environments Standing Committee on Employment, Education, and Training. Retrieved from https://www.aph. gov.au/DocumentStore.ashx?id=63d39cab-8e5e- 47cd-9a9e-5fc1d27ac387&subId=679038 Nakata, M. (2002). Indigenous knowledge and the cultural interface: Underlying issues at the intersection of knowledge and information systems. IFLA Journal, 28(5/6), 281–291. https://doi.org/10.1177/034003520202800513 Nakata, M. (2007). The cultural interface. The Australian Journal of Indigenous Education, 36(S1), 7–14. https://doi.org/10.1017/S1326011100004646 ORIMA Research. (2019). A report on qualitative research to evaluate the Bachelor of Science (Extended). Canberra: ORIMA Research. Pascoe, B. (2014). Dark emu: Black seeds: Agriculture or accident? Broome, Western Australia: Magabala Books. Patrick, R., & Moodie, N. (2016). Indigenous education policy discourses in Australia. In T. Barkatsas & A. Bertram (Eds.), Global learning in the 21st Century. Rotterdam: SensePublishers. Pomerantz, E., Moorman, E., & Litwak, S. (2007). The how, whom, and why of parents’ involvement in children’s academic lives: More is not always better. Review of Educational Research, 77(3), 373-410. Productivity Commission. (2020). Indigenous evaluation strategy, background paper. Canberra. Productivity Commission. (2021). Report on government services. Canberra. Quin, D. (2017). Longitudinal and contextual associations between teacher–student relationships and student engagement: A systematic review. Review of Educational Research, 87(2), 345-387. Rigney, L. I. (2006). Indigenist research and Aboriginal Australia. In J. Kunnie & I. Goduka (Eds.), Indigenous peoples’ wisdom and power: Affirming our knowledge through narratives (Chapter 3). London, UK: Routledge. https://doi.org/10.4324/9781315252414 Sarra, C., Spillman, D., Jackson, C., Davis, J., & Bray, J. (2020). High-expectations relationships: A foundation for enacting high expectations in all Australian schools. The Australian Journal of Indigenous Education, 49(1), 32-45. Skamp, K. (2012). Teaching primary science: Trial- teacher feedback on the implementation of Primary Connections and the 5E model. Retrieved from https:// www. primaryconnections.org.au/about/history/ research-andevaluation/teaching-primary-science.pdf Taylor, E.V., Lalovic, A., & Thompson, S. C. (2019). Beyond enrolments: A systematic review exploring the factors affecting the retention of Aboriginal and Torres Strait Islander health students in the tertiary education system. International Journal for Equity in Health, 18, Article 136. Tynan, M., & Noon, K. (2017). Indigenous STEM Education Project first evaluation report September 2014-June 2016. Canberra: CSIRO. Retrieved from https://www.csiro. au/-/media/Education-media/Files/Indigeneous-STEM/ Evaluation/First-Evaluation-Report-Executive-Summary.pdf Universities Australia. (2017). Universities Australia Indigenous strategy 2017-2020. Canberra. Retrieved from https://www.universitiesaustralia.edu.au/wp-content/ uploads/2019/06/Indigenous-Strategy-v16-1.pdf Universities Australia. (2019). Universities Australia indigenous strategy first annual report. Canberra. Retrieved from https://www.universitiesaustralia. edu.au/wp-content/uploads/2019/06/20190304- Final-Indigenous-Strategy-Report-v2-2.pdf Walker, L., Gilbert, C., Harris, F., Salmon, J., & Banks, C. (2021) Indigenous STEM Awards: Stories of change report. Canberra, Australia: CSIRO. Retrieved from https://www.csiro.au/en/education/ Programs/Indigenous-STEM-Education-Project/ Monitoring-and-evaluation/Case-study-reports YuMi Deadly Centre. (2017). Impact evaluation report. Brisbane. Retrieved from https://research.qut.edu. au/ydc/wp-content/uploads/sites/181/2017/11/ YDC_Impact_Evaluation_Report_March_2017.pdf Yunkaporta, T. (2009). Aboriginal pedagogies at the cultural interface (Unpublished Doctoral thesis). James Cook University, Townsville, Australia. Retrieved from http://eprints.jcu.edu.au/10974 Appendix A: Context As Australia’s national science and research agency, CSIRO is passionate about the power of STEM to unlock a better future for all Australians. Building on this commitment, the Indigenous STEM Education Project was funded by the BHP Foundation in response to a recognised gap in policy and discourse looking at the intersection of STEM education and Aboriginal and/or Torres Strait Islander education. In 2015, the Australian Government launched its National Innovation and Science Agenda, which emphasised the need to improve Australia’s international competitiveness in STEM and reversing declining maths literacy (Commonwealth of Australia, 2015). Australian Education Ministers also endorsed the National STEM School Education Strategy 2016-2026, which sought to coordinate and target efforts to improve STEM education (Education Council, 2015). Despite the long tradition of scientific knowledge in Aboriginal and/or Torres Strait Islander communities, little attention has been paid to systematically including Aboriginal and Torres Strait Islander knowledges within STEM; for example, the National STEM School Education Strategy is silent on Aboriginal and Torres Strait Islander knowledges. Increasingly, voices are calling for more recognition of the scientific and engineering heritage of Aboriginal and Torres Strait Islander cultures (Ball, 2015; Dandolo Partners, 2020; Pascoe, 2014). The Indigenous STEM Education Project also builds on the work of Sarra et al. (2020), who argue for a strengths- based, high-expectations response to historical education models that erroneously position Aboriginal and/or Torres Strait Islander students with underachievement as the problem. As outlined in Patrick and Moodie (2016), this deficit framing and a focus on achieving statistical equality have been key features of education policy in Australia. In contrast, and as recommended by Patrick and Moodie (2016), the Indigenous STEM Education Project centres Aboriginal and Torres Strait Islander knowledges and cultures and seeks to form genuine and equal partnerships with students, schools, and communities. In the six years since the Indigenous STEM Education Project was launched, there have been several key policy and research developments. In 2017, the Australian Government launched Australia 2030: Prosperity through innovation, which focuses on adapting the education system to support STEM skills and better preparing students for post-school STEM occupations (Innovation and Science Australia, 2017). It also recognised that occupations currently requiring STEM skills are outstripping overall employment growth. In 2019, the Australian Education Ministers agreed to the Alice Springs (Mparntwe) Education Declaration, a national statement outlining Australia’s education goals. The declaration aims “to give all young Australians equal access and opportunity to become confident and creative individuals, successful lifelong learners, and active and informed members of the community” (Council of Australian Governments Education Council, 2019, p.20). STEM learning areas are identified as a key national focus for school education in Australia and as critical to equip students to engage productively in a world of rapidly changing technology. As outlined in Banks et al. (2021), there has been a gradual shift by ACARA to incorporate Aboriginal and/or Torres Strait Islander histories and cultures into the Australian science curriculum at all levels. In 2018, in response to feedback from the community and educators, ACARA released 95 elaborations to provide practical examples and background information for teachers looking to incorporate Aboriginal and Torres Strait Islander history, culture, knowledge, and understanding into their teaching of core scientific concepts (ACARA, 2019b). In 2020, the Australian states and territories along with the Coalition of Aboriginal and Torres Strait Islander Peak Organisations signed the National Agreement on Closing the Gap. This 10-year agreement replaced the 2008 Indigenous Reform Agreement. Unlike its preceding agreement, the 2020 agreement was negotiated and signed by Aboriginal and Torres Strait Islander representatives (Coalition of Peaks, 2020). Relevant to the Indigenous STEM Education Project, the new agreement includes, for the first time, a higher education target for Aboriginal and/or Torres Strait Islander students – that 70 per cent of Aboriginal and/ or Torres Strait Islander Australians between 25 and 34 years of age have a tertiary qualification by 2031. Recent data looking at school attainment and STEM performance for Aboriginal and/or Torres Strait Islander students shows mixed results. As highlighted in the Productivity Commission’s Report on Government Services (Productivity Commission, 2021), there is an increasing number of Aboriginal and Torres Strait Islander students attaining a Year 12 or equivalent qualification, increasing from 39.4 per cent of Aboriginal and/or Torres Strait Islander students in 2001 to 63.2 per cent in 2016 (Department of the Prime Minister and Cabinet, 2021). The proportion of Year 6 Aboriginal and/or Torres Strait Islander students at or above proficient standard in science achievement increased from 19 per cent in 2009 to 35 per cent in 2018. Results for Aboriginal and/ or Torres Strait Islander proficiency in information and communication technologies have fluctuated significantly since between 2008 and 2017. In 2017, the results for Year 6 students were unchanged, while the results for Year 10 students reduced from 32 per cent in 2008 to 24 per cent in 2017. These data are estimates only and have relatively large confidence intervals; see Productivity Commission (2021) for more details. As argued by Anderson (2020), to support achievement of the higher education target in the Closing the Gap 2020 Agreement, schools need to ensure Aboriginal and/or Torres Strait Islander students who complete Year 12 are academically equipped for further university education or vocational training. Broadening students’ options and creating stronger pathways in secondary school mathematics and science will help redress the under- representation of Aboriginal and/or Torres Strait Islander students in STEM, address what Ball (2015) refers to as “the leakage of Indigenous talent to other areas”, and open the door to high paying jobs in science-related fields (p. 17). Appendix B: Updated Impact Statements