Generation STEM evaluation report 2019–22 Building a bigger, stronger and more diverse STEM pipeline in New South Wales Impact and Evaluation, CSIRO Education and Outreach March 2023 Generation STEM is managed by CSIRO and made possible by the NSW Government’s $25 million endowment to the Science and Industry Endowment Fund (SIEF). Citation Banks C, O’Brien M, and Miller K (2023) Generation STEM evaluation report 2019–22: Generation STEM. Canberra, Australia: CSIRO. Copyright © Commonwealth Scientific and Industrial Research Organisation (CSIRO) 2023. To the extent permitted by law, all rights are reserved, and no part of this publication covered by copyright may be reproduced or copied in any form or by any means except with the written permission of CSIRO. Disclaimer CSIRO advises that the information contained in this publication comprises general statements based on social research. Every effort has been made to ensure the accuracy of the findings presented in this report. However, given its interim nature and other constraints inherent with the methodologies, the reader should exercise some caution when interpreting the findings in this report. CSIRO is committed to providing web-accessible content wherever possible. If you are having difficulties with accessing this document, please contact csiro.au/contact. Acknowledgments The authors would like to thank the large number of Generation STEM participants and stakeholders who have engaged in this evaluation by completing surveys, interviews, and focus groups and/or supplying the Commonwealth Scientific and Industrial Research Organisation (CSIRO) with education data. Their time and effort to has helped inform the insights discussed in this report. Thanks to the CSSHREC CSIRO Human Research Ethics Committee and the New South Wales (NSW) Department of Education research unit for providing approvals and useful feedback on the ethical and research components of the project. In particular, the authors would like to acknowledge the time and expertise of Myra Singh, a proud Wiradjuri woman. Myra is an Academic Coordinator for CSIRO’s Young Indigenous Women’s STEM Academy and her assistance with the analysis of qualitative data collected from participating Aboriginal students and Community members for the Deadly in Generation STEM (DiGS) program has helped inform the insights discussed. The Impact & Evaluation team acknowledges the Traditional Owners of the lands with whom this project is collaborating, their vibrant living cultures and knowledge systems and acknowledge the Countries on which the evaluation work took place. We pay our respects to Elders past and present and thank all community members who provide the leadership to ensure meaningful and effective engagement with Aboriginal and Torres Strait Islander communities. CSIRO acknowledges the contributions of former and current members of the Impact and Evaluation team who supported the program monitoring and evaluation methodologies, data collection, and analyses, particularly Kate Miller. We also acknowledge the significant contributions of former and current members of the Generation STEM program team who supported the program monitoring and evaluation methodologies, data collection, and analyses. We also thank the Consultative Council of Generation STEM, who provided oversight and strategic advice to the evaluation and data insights projects, particularly Ian Oppermann and David Wright. Contents Introduction.................................................................................................................................................3 Purpose of this report........................................................................................................................................................4Evaluation approach...........................................................................................................................................................6STEM Community Partnerships Program.....................................................................................9Methodology.......................................................................................................................................................................9Process...............................................................................................................................................................................10Impact and outcomes.......................................................................................................................................................16Discussion..........................................................................................................................................................................25Recommendations............................................................................................................................................................29Generation STEM Links.........................................................................................................................31Methodology.....................................................................................................................................................................31Process...............................................................................................................................................................................31Impact and outcomes.......................................................................................................................................................36Discussion..........................................................................................................................................................................39Recommendations............................................................................................................................................................39Deadly in Generation STEM................................................................................................................41Methodology.....................................................................................................................................................................41Process...............................................................................................................................................................................41Impact and outcomes.......................................................................................................................................................44Discussion..........................................................................................................................................................................45Recommendations............................................................................................................................................................45Conclusion..................................................................................................................................................49References.................................................................................................................................................50Appendices.................................................................................................................................................51Appendix A. Proposed school-level data analysis for STEM CPP..................................................................................51Appendix B. STEM CPP student survey data 2022..........................................................................................................53Appendix C. STEM CPP teacher survey data 2022..........................................................................................................60Appendix D. STEM CPP industry mentor survey data 2022...........................................................................................65Appendix E. Generation STEM Links student survey data 2022....................................................................................70Appendix F. Generation STEM Links supervisor survey data 2022...............................................................................77 Figures Figure 1. Generation STEM program model...........................................................................................................................3Figure 2. Generation STEM impact pathway..........................................................................................................................4Figure 3. Likelihood that industry contact would recommend STEM CPP to colleagues..................................................11Figure 4. Net percentage change in student interest and knowledge (2021 and 2022)...................................................16Figure 5. Net percentage change in awareness and intentions..........................................................................................18Figure 6. Student responses to ‘STEM subjects are important to my future study and career’ (2022)............................19Figure 7. Student self-reported confidence in skill areas (2022).........................................................................................20Figure 8. STEM CPP program intensity over academic year................................................................................................26Figure 9. STEM subject enrolments in Australia...................................................................................................................27Figure 10. Generation STEM Links student applicant demographics.................................................................................33Figure 11. Student applicant areas of interest versus industry project focus....................................................................34Figure 12. Generation STEM Links priority cohorts.............................................................................................................35Figure 13. Change in student confidence in STEM skills and knowledge..........................................................................37Figure 15. STEM Evidence Tool prototype............................................................................................................................48Figure 16. Breadth versus depth in STEM education programs..........................................................................................49 Tables Table 1. Summary of evaluation outcomes (does not include implementation outcomes)...............................................2Table 2. Generation STEM interim evaluation overview.......................................................................................................2Table 3. Key evaluation questions...........................................................................................................................................7Table 4. Evaluation rubric for outcome evaluation questions..............................................................................................7Table 5. STEM CPP monitoring and evaluation methods.......................................................................................................9Table 6. STEM CPP implementation.......................................................................................................................................10Table 7. What happened to students at different levels of STEM interest (2022)..............................................................17Table 8. Teacher perceptions of students’ awareness levels...............................................................................................19Table 9. Teachers’ perceptions of students’ skill improvements (2022).............................................................................20Table 10. Net percentage change before and after STEM CPP by gender..........................................................................23Table 11. Cost per participant................................................................................................................................................24Table 12. Effectiveness-cost ratio..........................................................................................................................................24Table 13. Student categories for strategic focus..................................................................................................................26Table 14. Year 12 subject enrolment trends in Australia.....................................................................................................27Table 15. Data collection methods for Generation STEM Links evaluation.......................................................................31Table 16. Generation STEM Links placement data...............................................................................................................33 Generation STEM evaluation report 2019–22 Acronyms ABN Australian Business Number CALD Culturally and linguistically diverse CESE Centre for Education Statistics and Evaluation CEO Chief Executive Officer CSIRO Commonwealth Scientific and Industrial Research Organisation DiGS Deadly in Generation STEM HSC High School Certificate ICSEA Index of Community Socio‑educational Advantage MOU Memorandum of Understanding NAPLAN National Assessment Program – Literacy and Numeracy NESA New South Wales Education Standards Authority NSW New South Wales PhD Doctor of Philosophy RDA Regional Development Association SES Socio-economic status SHAE Academy Moree Sports, Health, Arts and Education Academy SIEF Science and Industry Endowment Fund STEM Science, Technology, Engineering, and Mathematics STEM CPP STEM Community Partnerships Program TAFE Technical and Further Education TPL Teacher professional learning VET Vocational education and training WIL Work integrated learning Executive summary Generation STEM is a New South Wales (NSW) Government initiative to attract, support and retain NSW students into Science, Technology, Engineering, and Mathematics (STEM) education and career paths. Generation STEM is made possible through the NSW Government’s $25 million endowment to the Science and Industry Endowment Fund (SIEF). The initiative launched in 2018 and to date comprises four programs: STEM Community Partnerships Program (CPP), Deadly in Generation STEM (DiGS), Generation STEM Links, and Data Insights. A monitoring and evaluation process has been undertaken with the first three programs; a description of the progress to date and evaluation assessments are presented in this report. The key results of the implementation and outcome evaluation include: • strong implementation outcomes were observed, including significant scaling up of STEM CPP and formation of multiple industry partnerships, and successful first year implementation of Generation STEM Links and DiGS • increased teacher capacity, particularly in the area of inquiry-based learning for STEM CPP • increased student interest and knowledge of STEM, particularly among those with lower initial interest levels and female students • increased awareness of and intention to pursue STEM careers among student participants • engagement of students in hands-on, real world STEM activities • strengthened knowledge and understanding of culture and Indigenous knowledges • successful targeting of Generation STEM Links to students in lower socio-economic status (SES) areas • increased confidence in skills, including problem solving and working in teams. The cost per participant ratio and effectiveness-cost ratio for STEM CPP were reasonable and broadly in line with other STEM programs; the cost per participant ratios for DiGS was higher in comparison, reflecting the pilot phase of the program and a different type of program model (i.e., narrower and deeper focus on individual students). The cost per placement ratio for Generation STEM Links was also relatively higher, reflecting the pilot phase, the subsidy component, and that significant work goes into the student and industry placement process, placement facilitation, monitoring, and interventions as appropriate. Due to the stage of the initiative, there was insufficient evidence at this time to conclusively determine whether STEM CPP led to increased numbers of students taking STEM or the diversity and number of high potential students participating in STEM after Year 10. There was also insufficient evidence to determine outcomes for Generation STEM Links but initial results indicate the program is set up to achieve positive outcomes going forward. A summary of the assessment of outcomes for STEM CPP and Generation STEM Links is presented in Table 1. The Data Insights project progressed substantially, with initial results from the predictive analytics providing useful evidence and relatively accurate ability to predict whether students in the middle years of high school would pursue STEM in Year 12. Evidence X developing a prototype STEM Evidence Tool and the establishment of a co-design process. A number of challenges arose among STEM CPP, Generation STEM Links, and DiGS, including successfully targeting students disengaged from STEM, program fidelity, sustainability of outcomes, and establishing and maintaining service delivery partner relationships. The findings were based on a range of data sources. Table 2 outlines these sources along with program maturity levels and evaluation type for each program. Relevant programs were assessed against a set of evaluation questions using an evaluation rubric: insufficient evidence, objectives not met, met objectives, and exceeded objectives. Key recommendations for the three programs focused on refinement of the program model and targeting (STEM CPP), utilising industry mentors more effectively (STEM CPP), greater focus on vocational education and training (VET) (Generation STEM Links), investigation of culturally and linguistically diverse (CALD) students’ placement rates (Generation STEM Links), discussion with young people on design and delivery of camps/immersion days (DiGS), and ensuring emerging outcomes are reflected (DiGS). Table 1. Summary of evaluation outcomes (does not include implementation outcomes) PROGRAM QUESTION AREA ASSESSMENT STEM CPP Increased student interest, knowledge and understanding of STEM Met objectives Increased student awareness about STEM education and career pathways Met objectives Increased student transferable skills Met objectives Increased overall number of students participating in STEM education after Year 10 Insufficient evidence (data available later in 2023) Increased diversity and number of high potential students participating in STEM education after Year 10 Insufficient evidence (data available later in 2023) Cost per participant $1,665 (2019) $1,467 (2020) $805 (2021) $589 (2022) Generation STEM Links Increased students’ (a) technical and enterprise skills (b) awareness about potential STEM career pathways and (c) commitment to work in STEM Met objectives Increased the number and type of tertiary students working in STEM jobs Insufficient evidence (too early to tell) Increased the capacity of industry Insufficient evidence Cost per placement (July to December 2022) $8,377 Table 2. Generation STEM interim evaluation overview GENERATION STEM COMPONENT DATA SOURCES PROGRAM MATURITY LEVEL EVALUATION TYPE STEM Community Partnerships Program • Participant and stakeholder surveys and interviews • Administrative data • Case studies • Program staff feedback • Cost/outcome analysis Mature program (from 2019) Implementation and outcome evaluation Generation STEM Links • Administrative data • Participant surveys and interviews • Program team feedback New program, implemented from 2022 Implementation and early outcomes evaluation Deadly in Generation STEM • Administrative data • Student yarning session • Community stakeholder focus group • Program team feedback New program, implemented from 2022 Implementation evaluation Data insights • Administrative data • Program staff feedback In development, commenced in late 2021 Progress update Introduction Generation STEM is a NSW Government initiative to attract, support and retain NSW students into STEM and school, into further education and into employment. The NSW Government has made a ten year, $25 million endowment to the Science and Industry Endowment Fund (SIEF) to establish Generation STEM. Generation STEM takes a location-based approach with the program being delivered in regions in NSW where there is a current and future need for STEM skills. Generation STEM empowers young people with the relevant STEM skills to pursue a STEM career. The ambition is to build a strong community of STEM‑capable citizens to fuel local industry. CSIRO, under the steerage of SIEF and in consultation with the NSW Government, is working to design and deliver Generation STEM. Generation STEM comprises several programs and activities organised in three focus areas (see Figure 1). This report is organised around individual programs rather than the broader focus areas, primarily because each program is at a different level of maturity and it is difficult to make overall assessments of these general focus areas at this stage of the initiative. The final evaluation report in 2027 aims to have sufficient data in each focus area, due to greater program maturity and the progression of data requests. STEM CPP creates partnerships between local schools and industry, with the goal of highlighting local STEM careers and opportunities and providing avenues for students to develop their STEM skills in an engaging and rewarding way. STEM inquiry-based projects are developed by students with the guidance of industry mentors and teachers, where students also receive opportunities to build their STEM skills and awareness through a range of activities, including site visits, work experience, masterclasses and VET programs. Deadly in Generation STEM aims to increase engagement and retention of Aboriginal and/or Torres Strait Islander students in STEM educational pathways, STEM employment, and/or future education through culture and on Country. Currently run on Dharawal Country (Illawarra-Shoalhaven region) and on Kamilaroi Country (Moree), the program brings together Aboriginal and/or Torres Strait Islander students with CSIRO project officers, STEM mentors, cultural mentors, and community stakeholders to deliver workshops and run hands-on activities through culture and on Country. Generation STEM Links provides internships to help tertiary students gain workplace skills and transition into STEM jobs after graduation. It also aims to build a pool of STEM-capable professionals for the future of NSW STEM industries through its partnerships. Generation STEM Links is a hands-on internship program that pairs NSW students in their penultimate to final year of study in STEM degrees and qualifications with industry to allow both sides to learn, connect, and innovate. The program also seeks to provide additional recruitment pathways for businesses that may not have considered a tertiary student intern. Data Insights comprises two projects aiming to increase the evidence base and its use in improving STEM education outcomes. The first is a data analytics project to understand the factors (barriers and facilitating) leading to STEM education outcomes. The second is called Evidence X, which seeks to build an evidence platform/set of tools that will assist Generation STEM, and the sector more broadly, design and evaluate the success of programs. It is planned that the analytics project will eventually form part of Evidence X. Figure 1. Generation STEM program model Attract, support and retain NSW students in STEM Focus area 1 Building the STEM pipeline Increasing the overall quantity and diversity of students entering STEM educational pathways. Focus area 2 Transition to employment Supporting STEM tertiary students (from university and VET) in the transition to STEM employment. Increasing diversity Generation STEM aims to increase participation of students from groups underrepresented in the STEM pipeline: Aboriginal and/or Torres Strait Islander peoples, women, people from low socio-economic areas and people from regional and remote areas. Location priorities Generation STEM is established in areas where it can have the greatest impact – growing communities with growing STEM industries. Taking a community-driven approach to ensure high impact and relevance. Purpose of this report Monitoring and evaluating Generation STEM’s implementation and impact is an integral aspect of the Generation STEM strategy. CSIRO’s approach to planning, monitoring and evaluating impact is based on the concept that there must be a clear pathway leading from the impact back to the activities of the program. The Generation STEM Impact Pathway (Figure 2) is used to articulate these relationships. STEM CPP, Generation STEM Links, and DiGS have individual impact pathways that elucidate program‑specific impact chains. This impact model approach enables evidence-based changes to be made, if needed, providing the opportunity for continuous improvement. Impact pathway planning and monitoring provides a means to identify and articulate key project activities and objectives; as well as the opportunity to monitor progress towards desired goals, and implement required changes as identified through this tracking process to enhance the capacity to achieve key project objectives. Monitoring and evaluation are being undertaken over the life of Generation STEM and will include: • monitoring, evaluating, and reporting on the initiative and its activities • making recommendations for future action. The overall performance of Generation STEM will be assessed against the intended implementation goals and program outcomes across its 10 year span. Data are being collected throughout the 10 years in order to measure progress towards achieving these outcomes. The monitoring and evaluation framework described above will be used to assess the degree to which outcomes have been achieved. A key challenge of impact evaluation is clarifying the intended impacts, which includes consideration of the attribution and the alternative reality without the Generation STEM intervention (counterfactual). It is important to acknowledge that collecting evidence of the state of the counterfactual alongside monitoring will be important and is subject to the availability of data. It is acknowledged that it will be difficult to measure and to determine attribution of each outcome to Generation STEM’s activities. Figure 2. Generation STEM impact pathway Impact statement: Generation STEM is a 10-year initiative aimed at attracting, supporting, and retaining NSW students in STEM at school, and into further education and employment in NSW. Participation: Who we need to reach across the various parts of the pathway? SIEF, Generation STEM team, CEdO, delivery partner(s), tertiary institutions and employers Generation STEM team, CEdO, delivery partner(s), tertiary institutions, employers Generation STEM team, delivery partner(s), tertiary institutions, employers and students INPUTS What we invest ACTIVITIES What we do OUTPUTS Our deliverables Funding – CSIRO/ Industry Staff – Generation STEM team, CEdO M&E team, delivery partner(s) and industry Resources – participating businesses, tertiary institution and CSIRO (materials, equipment and facilities etc.) CSIRO corporate services (e.g., finance, HR, comms etc.) CSIRO research, reputation, national footprint Governance – NSW Department of Education, SIEF, Consultative Council CSIRO contract management Facilitate relationships between CSIRO, industry, and education institutions Recruit industry, education intuitions and students (with a specific location focus**) Engage government and other community stakeholders Develop resources and educational material Develop and collect data for monitoring and evaluation Communications and marketing STEM Community Partnerships Program (STEM CPP) STEM Links Deadly in Generation STEM (DIGS) Data Insights Program monitoring and evaluation, analysis, interim and final reports Events communications and marketing collateral, (including for targeted equity groups) Generation STEM programs Students/young people Industry Education sector Policy and STEM program providers Direct influence on outcomes Indirect influence on outcomes Minimal influence on outcomes Generation STEM team, CEdO, delivery partner(s), tertiary institutions, employers and students OUTCOMES The uptake, adoption or consumption of our work BENEFITS Eco, environ, soc Shorter term (direct) outcomes Longer term (indirect) outcomes Participants* have increased awareness of STEM, and its associated education pathways and careers Participants have greater exposure to STEM experiences relevant to their interests and/or everyday life Participants improve their enterprise and/or technical STEM skills NSW STEM industry develops stronger connections with local education institutions and community Educators are more aware of the different approaches/activities available to help engage students in STEM Generation STEM continues to improve based on evidence Sustainability and legacy outcomes TBC Participants have increased confidence about their abilities in STEM Participants have an increased interest in, and attitude towards STEM Participants have an increased aspiration to study and work in STEM Participants have an increased work readiness skills NSW STEM industry have increased opportunities to collaborate with local education institutions and students Educators have greater confidence and skills to use different approaches/ activities to help engage students in STEM Educators and policy makers better understand what factors influence young peoples’ STEM education and career pathways Sustainability and legacy outcomes TBC More participants pursue STEM education pathways More participants have the appropriate STEM skills for the NSW STEM jobs in demand NSW STEM industry better align student engagement opportunities with current and future STEM employment demands NSW STEM industry better align student engagement opportunities to attract underrepresented groups into STEM School and tertiary systems are better able to promote and support STEM Educators and policy makers design more effective STEM education interventions to increase STEM uptake among young people Sustainability and legacy outcomes TBC More people are STEM literate in their tertiary education and/or careers More people are employed in NSW- based STEM jobs** More NSW STEM workers remain employed in STEM jobs NSW STEM industry are able to hire more people with the technical skills in demand Economic – increased size of the STEM-skill workforce in NSW Social – increased awareness of STEM and its place in NSW society *Generation STEM participants include underrepresented groups in STEM, in particular: females, Aboriginal and/or Torres Strait Islander people, people in regional and/or remote areas, and people from low socioeconomic backgrounds. **A focus is placed on STEM jobs associated with Special Activation Precincts, other regions with upcoming industry development and infrastructure investments, or industries where there is a shortage of STEM professionals in NSW. Evaluation approach This evaluation is based on a set of key evaluation questions (see Table 3), impact pathways for each program and Generation STEM overall, and an evaluation rubric. Each program has been assessed against several evaluation questions using all available evidence to date, with a focus on more recent data for the outcome evaluation questions. As each program is at different stages of maturity, different types of evaluation questions have been covered in this report. For example, for STEM CPP, the most mature of the Generation STEM programs, both implementation and outcome questions were examined; while for Deadly in Generation STEM, which was implemented in 2022, only implementation questions were assessed. Assessments against the outcome evaluation questions employed an evaluation rubric (see Table 4) and were made using data from a multi-method approach, including self‑report surveys, interviews, focus groups, and administrative data. Where no pre‑determined metric or goals were identified for the program, the Impact and Evaluation team based the assessment on a reasonable standard based on existing research and evaluation literature. For example, increasing student interest in STEM was based partially on whether students’ self-reported increases in STEM interest were statistically significant and with moderate effect sizes1 . A process of triangulation aimed to position the findings within the broader context of the Generation STEM strategy to help guide the discussion of the findings and to inform key implications and recommendations. Triangulation of the results involved: • scanning key strategic and operational documents to identify relevant priorities; and • positioning the findings of the evaluation against the priorities of the program’s funder. 1 Effect sizes are a measure of how meaningful the relationship between two variables or the difference between groups is. Cohen (1988) classified effect sizes as small (0.2), medium (0.5), and large (>0.8). Table 3. Key evaluation questions PROGRAM(S) Process – implementation and monitoring STEM CPP Generation STEM Links DiGS How was the program implemented? Which aspects are working well? Which aspects can be improved? How? Outcome – impact and outcomes STEM CPP To what extent has STEM CPP increased students’: (a) interest, knowledge and understanding of STEM; (b) awareness about STEM education and career pathways; and (c) transferrable skills? To what extent has STEM CPP increased the (a) overall number of students participating in STEM education after Year 10; and (b) diversity and number of high-potential students participating in STEM education after Year 10? Generation STEM Links To what extent has Generation STEM Links increased students’: (a) technical and enterprise skills; (b) awareness about potential STEM career pathways; and (c) commitment to work in STEM? To what extent has Generation STEM Links increased the number and type of tertiary students working in STEM jobs? To what extent has Generation STEM Links increased the capacity of industry? What unexpected impacts has the program had? Economic – investment and outputs STEM CPP Generation STEM Links DiGS To what extent is the relationship between inputs and outputs been cost-effective and to expected standards? Table 4. Evaluation rubric for outcome evaluation questions OUTCOME EVALUATION RUBRIC Insufficient evidence Did not meet objective Met objective Exceeded objective There was not enough evidence to make an assessment for this question at the current time. The program did not meet the objectives set out in the Generation STEM strategy based on all available evidence. The program met the objectives set out in the Generation STEM strategy based on all available evidence. The program substantially exceeded the objectives set out in the Generation STEM strategy based on all available evidence. Note: Objectives may be quantifiable and based on pre-determined targets or goals or be based on overall qualitative assessments and comparisons with the existing evaluation and research literature. STEM Community Partnerships Program Methodology The evaluation of the STEM CPP program is based on the data collection methods outlined in Table 5. The CSIRO Human and Interdisciplinary Research Ethics Committee provided approval to undertake all monitoring and evaluation activities. The NSW Department of Education, relevant Catholic Dioceses, and individual independent schools also provided permissions to conduct these activities in schools. The suite of self‑report surveys, interviews, case studies, and program administrative data provides a rich and diverse range of data to assess program implementation and outcomes. The process of obtaining data indicating whether STEM subject enrolments have increased at STEM CPP schools compared to other schools is still in progress. Table 5. STEM CPP monitoring and evaluation methods METHODOLOGY SAMPLE NOTES Student self-report online general survey 2021 = 123 2022 = 220 Total = 343 Some surveys were distributed prior to 2021 but the number of responses were small and unrepresentative of the program model due to COVID-19 disruptions. Post event surveys 2021 = 165 2022 = 176 Total = 346 Post-event surveys were distributed after key events and activities, such as career expos and STEM Taster experiences. Teacher survey 2021 = 7 2022 = 44 Total = 52 Teachers involved in the program were offered an opportunity to respond to a voluntary, online survey seeking their feedback and perceptions of the program. In 2022, surveys with less than three question responses were excluded. Teacher interview 2021 = 3 2022 = 5 Total = 8 A small set of teachers (who indicated willingness to be interviewed in the teacher survey). Industry stakeholder survey 2021 = 4 2022 = 32 Total = 36 In 2022, surveys with less than three question responses were excluded. Industry stakeholder interview 2021 = 2 2022 = 6 Total = 8 A small set of industry stakeholders were interviewed. Council (local government) survey 2021 = 70 2022 = 9 Total = 79 In 2022, surveys with less than three question responses were excluded. Case studies 2021 = 1 2022 = 2 Total = 3 Case studies were undertaken of participating schools. Program administrative data 2019 to 2022 Details of locations, numbers of classes, events, etc. have been collected and collated by the Generation STEM program team. School-level data* Not available CSIRO and the NSW Government (Centre for Educational Statistics and Evaluation (CESE)) and NSW Education Standards Authority (NESA)) have been in discussions on signing an MOU to obtain school-level data to assess the impact of STEM CPP on school-level STEM subject selection. The proposed statistical methodology is outlined in Appendix A. Note: Report appendices include: STEM CPP student (Appendix B), teacher (Appendix C), and industry mentor (Appendix D) survey data, and Generation STEM Links student (Appendix E) and teacher (Appendix F) survey data. Process How is STEM CPP being implemented? Assessments of implementation effectiveness are based on program administrative data, feedback from the Generation STEM team, analysis of participant interviews and surveys, and observation by the Impact and Evaluation team. After a relatively slow roll-out and significant disruption due to COVID-19 up to and including 2020, the implementation and scaling up of the STEM CPP program has been comparatively swift in 2021 and 2022. As can be seen in Table 6, the pace of the roll out has increased substantially from 2021 to 2022 in terms of the numbers of students, schools, STEM professionals, industry partners, etc. The program was launched in 2018 but the gradual implementation and COIVD-19 disruptions mean the program model can only be considered to be implemented fully since 2021, which is the reasoning for focusing on the data from those two years (2021 and 2022). The program team has also expanded and been restructured to better meet the needs of the program; for example, by establishing an Industry Engagement position to focus on building and maintaining industry relationships and collaborations. STEM CPP also saw numerous operational activities and improvements that demonstrated the maturity of the program; for example, revising the initiative website, improved communications and promotions, stakeholder engagement and feedback opportunities, and refinement of the program model and elements (e.g., STEM Taster Program, Careers Expo, Gender Inclusive Classroom professional development). Table 6. STEM CPP implementation ELEMENT 2021 2022 PER CENT CHANGE Student participants 1,122 2,231 +98.8 Schools 47 77 +63.8 School retention rate (proportion of the schools from previous year participating in current year) 87% 91% +4.0 STEM professionals 29 60 +106.9 Virtual work experience opportunities 27 44 +63.0 Local Government Areas 7 12 +71.4 Industry partners (active) 14 61 +335.7 Delivery partnership pilot In 2022 the STEM CPP program focused on expanding the program beyond Western Sydney, with a successful expansion into the Central Coast region through a non‑financial collaborative partnership with a key community stakeholder. Exploratory discussions were undertaken in parallel in Wagga Wagga and Central West, with a formal delivery partnership being established in Central West. This partnership was the first subcontractor agreement established for delivery of STEM CPP. The delivery partner in Central West (a Regional Development Association or RDA) was identified organically through conversations between the CSIRO program leadership and other local stakeholders. It was identified that the delivery partner had strong relationships and expertise both in industry engagement and STEM education programs locally. Recognising the time involved in building trust and credibility with local stakeholders in an already congested space there was a strong rationale for partnering with an existing local organisation with established relationships rather CSIRO starting fresh with this process. Further, the STEM CPP model is designed so that Council is engaged as a central collaborator. Partnering with an RDA meant that they could assume both roles – program delivery and central point of collaboration across the region. Although this partnership was established with a strong rationale through existing connections, factors beyond the control of CSIRO meant that the partnership was not as fruitful as hoped in expanding the program into the Central West region (2 schools were actively involved in 2022). The contractual partnership arrangement was formally ended by CSIRO in late 2022. Despite this, there remains a positive relationship between CSIRO and the local delivery partner reflecting the strength of the relationship building to support effective implementation of the partnership approach. Insights amongst the program team have identified a number of areas where the approach could be strengthened in order to support greater success with future attempts to establish successful delivery partnerships. These include: • recognising the challenges of recruiting appropriately skilled Project Officers in regional areas and exploring different ways to attract candidates • test approaches to embedding local Project Officers into the CSIRO STEM CPP team to build capability and a greater sense of shared understanding and ownership of the program i.e., inducting the Project Officer through CSIRO • strengthen contractual arrangements to support accountability towards achieving partnership objectives • recognise the exploratory nature of this activity i.e., delivery partner model hasn’t been trialled in CSIRO Education and Outreach before, other programs run in regional areas successfully; however, these have CSIRO project officers on the ground • consider developing an approach to service delivery contracts that more formally and methodically assesses the potential fit of the model, provider, context, and timing • consider the visibility of CSIRO within a delivery partnership and how this can be maximised to leverage off trust and recognition within the region. How is STEM CPP facilitating NSW STEM industry connections and collaboration opportunities with local education institutions and students? Over 80 per cent of industry stakeholders reported that they were likely or very likely to recommend involvement in the STEM CPP program to industry colleagues (Figure 3). Amongst the survey sample (n = 32), the majority of respondents reported increased engagement with schools (73 per cent), with fewer indicating increased engagement between community members and industry (33 per cent respectively). Six interviews were undertaken in 2022 with industry stakeholders, including some with successful mentoring partnerships with schools, some who attended career expos, and some who had been matched with schools for a mentoring partnership without success. One representative lauded the information provided about involvement in the program: So, it gave us all the options on how we could participate. It talked about mentoring, it talked about the expo, it talked about the showcase, it talked about site tours, which we have scheduled as well during this year but then we had to reschedule due to COVID, so it’s been penciled in for next year already. So, it did give us a lot of options on how we could contribute to the program. Industry representative Figure 3. Likelihood that industry contact would recommend STEM CPP to colleagues The survey and interview results indicate that where industry was successfully engaged with students, the experience was overwhelmingly positive for the business and for the professionals themselves. For example: I went to the most recent display out at Campbelltown when they had their showcase, and I was just so inspired by these young kids who – as I was walking past their displays, they were almost tugging at me, saying, “Come and have a look at this. Come and have a look at this. Come and have a look at this.” And honestly, you feel so good. You love it. You know, I just – and they pull you out the front and you take a few photographs with the local dignitaries, and that’s fine, but the real stars are the kids, right? And you’re in a room full of – probably literally over 100 in that room, at the Campbelltown Arts Centre, and so, so proud of them, right? Industry mentor A new element of STEM CPP in 2022 was a partnership that was formed with TAFE NSW to deliver the STEM Taster Program to increase awareness about VET pathways into STEM careers. A total of 40 students took part in the program across two TAFE campuses. The experience offered hands-on, practical activities and interactions with STEM industry professionals, including in IT, manufacturing, aviation, and health. Students who responded to a voluntary post-activity survey (n = 16) indicated high levels of satisfaction, with 75 per cent either ‘very’ or ‘extremely’ satisfied with the program. The practical aspects of the experience were particularly useful as demonstrated by students: [The best part was] all the practical work (e.g., making things, exercises, seeing/using equipment, etc.). Student The subjects were interesting and the opportunities to look and do things that we normally wouldn’t be able to do in school. Student Student survey responses indicated increased levels of interest and positive intention around STEM as well, including 100 per cent of respondents saying they were more interested in learning about STEM (n = 12). Which aspects are working well? Which aspects can be improved? How? This section outlines a number of areas of consideration. The structure comprises areas for consideration that encompass both aspects that are working well and aspects that could be improved within those topics areas. Roll-out, reach, and diversity of schools STEM CPP has been effective in reaching a large number of schools and students in 2021 and 2022. The program is succeeding in engaging a set of schools (via school leadership), many individual teachers, and industry and local government partners to deliver a relatively cohesive program. The geographic reach of the program is also expanding beyond the initial Western Sydney focus area into Central Coast (7 schools) and Central West (2 schools). The process of reflection and refinement is also working well, with numerous program improvements made based on double-loop learning processes. For example, based on recommendations on a 2021 Insights Report by the Impact and Evaluation team, recommended actions are well-advanced, including those related to increasing gender diversity, mentor matching processes, and the effectiveness of certain program elements (e.g., Masterclass). The diversity of schools involved is also encouraging, with 43 per cent of active participating schools in 2022 having an Index of Community Socio-Educational Advantage (ICSEA) score below 1000 (34 out of 78). As a voluntary program, the program has been delivered to schools, and a small set of teachers2 within those schools who volunteer, or are sometimes asked or told, to take part. Many of these teachers are interested in and often already engaged in STEM, although a minority are told by their Head of Department or Principal to take part. This presents an issue that the most interested and/or engaged teachers in a school may be volunteering to be a part of the program, meaning that it’s possible that some teachers and students more in need of support may be missing out. 2 A small proportion of participants were careers advisors. Building capability amongst teachers Building the capability of teachers to effectively teach STEM and to identify and support students who are interested in STEM can have a ‘multiplier’ effect and is the principal way the program can achieve sustainable outcomes. This is because STEM-capable teachers will continue to support multiple cohorts of students for many years after involvement in the program. It’s also important because of the high proportion of STEM teachers who are teaching ‘out of field’ in Australia, with one report estimating a 19 per cent probability that mathematics would be taught by an out of field teacher (Shah, Richardson & Watt, 2020). Given the inquiry-project is the central feature of STEM CPP, it is important to consider how difficult teachers found it to embed the inquiry-based learning project into their regular teaching practice. In 2022 (n=36), most surveyed teachers found it ‘easy’ (47 per cent) or ‘very easy’ (14 per cent), but a significant minority found it ‘difficult’ (39 per cent) to integrate. This reflects the diversity of teacher participants in terms of levels of experience, confidence, and familiarity with inquiry and STEM-related pedagogies. More broadly, teachers were asked the degree to which STEM CPP events and activities had on their teaching of STEM. The proportion of respondents who felt each element had made ‘a significant’3 difference were: • 42 per cent: Delivering the inquiry-based learning project (n=36) • 42 per cent: Attending the teacher professional learning workshop (n=33) • 35 per cent: Engaging with industry mentors (n = 26) • 30 per cent: Attending a teacher networking event (n=20) It is encouraging that many teachers found the inquiry learning project and other elements of the program impactful in their teaching. Comparatively, the teacher networking event had less of a perceived impact and was attended by fewer respondents. Sustainability and fostering STEM culture within schools The sustainability of STEM CPP has multiple components, including schools continuing with the program while its running, and schools building a STEM culture that can effectively assist students pursue STEM beyond participation in the program. STEM school culture is defined as the overall STEM beliefs, values, practices, and resources in a school (White, Marshall & Alston, 2019). Future Generation STEM evaluation work will include explicit assessment of STEM school culture to understand whether programs are shifting culture, which would be a significant sustainability goal. Within the program lifespan, the majority of teachers (in 2022) felt that their schools would likely continue to participate in STEM CPP (n=40): 53 per cent (very likely), 35 per cent (likely), 10 per cent (possible), and 3 per cent (unlikely). Another feature of the sustainability of programs is the degree to which teacher participants share their knowledge and skills with their peers, and also the degree to which they encourage peers to get involved in the program. A ‘willingness to recommend’ question was included in the teacher survey to assess this propensity. Most teachers would recommend STEM CPP to other teachers (n=40): 48 per cent (very likely), 43 per cent (likely), and 10 per cent (possible). This demonstrates that the program is appealing to teachers and is providing perceived value. The majority of teachers who were interviewed as part of the evaluation indicated that their schools did not have an active industry mentor relationship; and therefore, these relationships were not able to be explored in depth at this time. However, the teacher survey data and a few of the teacher interviews indicated that when industry connections were facilitated, there was a perceived significant impact on students’ interest in their projects and STEM careers more broadly, indicating that this relationship may be valued sufficiently to continue beyond the program. 3 Based on a three point scale: Has made a significant difference, has made some difference, hasn’t really made a difference Key factors influencing program fidelity A number of factors affected the program’s fidelity, that is, the degree to which the program was delivered as intended. Some issues were largely operational; for example, some schools being unable to book excursions of most interest due to demand (e.g., Western Sydney Airport). Others were more central issues. For example, based on teacher interviews, it was apparent that early career teachers faced more barriers in maintaining the fidelity of the program as they were less likely to have the capacity and confidence to: • liaise with industry • incorporate inquiry-based learning into the classroom • support students to think creatively and explore ideas. Although this is not a surprising finding, it may behove the program team to consider differentiation of teacher support; for example, offering more or different assistance to early career teachers. The new content specialist position on the Generation STEM program team is an opportunity to address this issue. Finally, the way the program has been implemented means that schools can select from a menu of different activities they want to be involved in (e.g., mentor relationship, site visit(s), work experience, virtual work experience, showcase event, etc.), with only the inquiry project being a mandatory aspect of the program. Although this degree of program flexibility has been highlighted as a positive by some participants, it can also lead to large differences in how the program operates and the likelihood of achieving student, teacher, school, and industry outcomes. For example, an area of consideration is the difference in how lower ICSEA schools implement the program compared to higher ICSEA schools that have more resources and capacity. There is anecdotal evidence that some higher ICSEA schools may have the capacity to fully integrate STEM CPP into their curriculum and schedules, while some lower ICSEA schools may only expose students to specific aspects of the program, with much less integration and connection to the curriculum and therefore potentially less impact. The implication of this is the program team may need to differentiate their approaches and support depending on the capacity of the school. Leveraging peer-based learning opportunities Some teachers reported that opportunities for students to learn from other schools was valuable (e.g., to see what ideas other students came up with inspired them; meeting students from other schools; challenging stereotypes of kids who like STEM). Within schools, teachers reported that students learnt a lot from working together and focusing on what different strengths people could bring to a problem to create the solution, often resulting in working with students other than their friends in order to be successful with their inquiry project. Many teachers in the interviews reported that networking activities (in addition to teacher professional learning (TPL)) were valuable, although the survey data indicated lower relative valuing of these activities compared to other activities (see ). Related to this, for those teachers struggling with incorporating inquiry-based learning into classroom, TPL could be strengthened to spotlight schools that have had success and what that has involved. Another option is involving more experienced teachers in a ‘lessons learnt / what I’d do differently’ type of setting or buddying them up with less experienced teachers. Resources could be developed so that teachers have something to refer to in order to support translation of learnings from the TPL. Another consideration is building on existing peer support that is happening, particularly for the schools already established in delivering STEM CPP within their school (teachers mentoring and engaging new teachers into the program). Maximising industry interest for student benefit The current approach to mentorship is to focus on supporting and building capacity amongst teachers. Many teachers report that when they have had a strong partnership with an industry professional, this has been due to the response of students’ face-to-face opportunities to engage with real-life STEM professionals. These professionals have done some of the heavy lifting with students in terms of supporting the creative thinking, ideas, initial research phase, and in some schools, have come back to work with student groups directly. Teachers reported benefiting from other teachers (see above re peer‑based learning re what works) more than they reported learning from / being supported by mentors. Research has shown that mentors can provide a range of emotional and technical support to students, which can raise their self-efficacy, skills, confidence, and ability to navigate STEM pathways (Aitkens et al., 2020; Jin, 2021; Millar, Hobbs, Speldewinde & van Driel, 2022). Direct relationships with mentors, not mediated by a teacher for example, have shown to be effective; in addition, having mentors and mentees with shared beliefs, values and backgrounds works well.4 One teacher highlighted the benefit of the STEM professional expertise: What’s really valuable about our partnership with… [CSIRO] has been the expertise…and the real-world settings. The kids were very excited to be shown through a laboratory, for example. The students loved being taken to a real site and getting that. We do inquiry-based learning anyhow but having the access to expertise, and localised expertise as well, that’s what makes it really full of potential for the students. Teacher 4 Research also shows that other ‘what works’ elements to STEM mentoring: Mentors being involved at particularly important times, for example in years of subject selection in high school; Mentors that provide technical/professional (e.g., career resources, research) and socioemotional/psychological support (e.g., warm/friendly, good listeners); Fostering student confidence and self-efficacy; Facilitating the valuing of STEM among students; Having a focus on the growth and development of students; Mentor-mentee dynamics that are personal and reciprocal in nature (mutual exchange of energy and support); and Making STEM seem relatable and possible (especially for girls), for example by providing counter-narratives to stereotypes. Impact and outcomes Question 1a: To what extent has STEM CPP increased students’ interest, knowledge and understanding of STEM? Student and teacher participants were asked about any changes in relation to interest in learning about STEM in 2021 and 2022 (post-retrospective survey). As seen in Figure 4, in most areas of student interest and knowledge, there was a net percentage increase (the proportion of students who indicated a positive change minus the proportion who indicated a negative change). Overall, the 2021 results were stronger compared to 2022. Given the program delivery was largely similar between years, this may be due to a post-lockdown ‘bump’, with students conflating satisfaction with more normalcy in schooling with participation in the program; alternately, it could be due to differences in the samples. However, only some areas of interest and knowledge demonstrated statistically significant changes for 2021 (Expect to do well in STEM subjects, STEM important for society, STEM useful in everyday life) and for 2022 (STEM useful in everyday life). Figure 4. Net percentage change in student interest and knowledge (2021 and 2022) Surveyed teachers also indicated that they felt their students were more interested in STEM as a result of participating in STEM CPP. Specifically, in 2022 teachers identified different aspects of STEM CPP that they felt made their students ‘a bit’ or ‘much more’ interested in STEM: • 91 per cent: Attending the showcase event (n=35) • 89 per cent: Visiting a local STEM industry or worksite (n=19) • 89 per cent: Completing the inquiry-based learning project (n=38) • 82 per cent: Interacting with industry mentors (n=33) • 81 per cent: Completing the STEM work experience (n=16) Overall, according to students and teachers, STEM CPP is contributing to many students’ increased interest, knowledge, and understanding of STEM in many areas, particularly practical and contextual ones, such as illuminating to students how STEM is useful in everyday life and how STEM is important for society. One explanation for why STEM CPP is not resulting in changes across all areas is that students already had high levels of interest, knowledge, and understanding, and therefore the program was unlikely to improve those levels overall. This ceiling effect was exemplified in the 2022 student survey: 31 per cent of students were already at the highest self-reported level of interest in STEM before participating in STEM CPP; therefore, there was no potential for positive change in the post-retrospective survey. Table 6 demonstrates this issue: the biggest increases in interest were seen among students with lower (but not the lowest) levels of pre‑existing interest in STEM. The students with the lowest and highest levels of interest before STEM CPP were least likely to shift their perception. This result also relates to the issue of program targeting, as there are minimal benefits to investing in students who are already highly interested in STEM and likely to continue that pathway regardless of the program. Therefore, the program objectives are most strongly being met by the students with lower levels of pre‑existing interest (and other attributes). Question 1a assessment: To what extent has STEM CPP increased students’ interest, knowledge and understanding of STEM? ASSESSMENT Insufficient evidence Did not meet objective Met objectives Exceeded objective Table 7. What happened to students at different levels of STEM interest (2022) PERCENTAGE AFTER PARTICIPATING IN STEM CPP LEVEL OF INTEREST BEFORE PARTICIPATING IN STEM CPP INCREASED INTEREST SAME INTEREST DECREASED INTEREST 0 – lowest interest (n = 9) 0% 100% n/a 1 (n = 16) 50% 38% 13% 2 (n = 25) 56% 24% 20% 3 (n = 39) 23% 49% 28% 4 (n = 56) 27% 41% 32% 5 – highest interest (n=65) n/a 89% 11% All levels (n=210) 22% 58% 20% Note: changes of 50% or greater highlighted in green. Question 1b: To what extent has STEM CPP increased students’ awareness about STEM education and career pathways? Participating students were asked several questions about their awareness of STEM education and career pathways (see Figure 5), and their intentions for following those pathways. In both the 2021 and 2022 cohorts, there was a substantial net positive change before and after participating in the program, particularly for ‘awareness of the type of jobs I could work in’ with a 41 per cent positive increase in 2021, and 27 per cent in 2022. There were smaller, but still positive, net percentage changes in the area of ‘would like to have a job in STEM’: 11 per cent in 2021 and 12 per cent in 2022. There may have been a ceiling effect in this area as well, with 24 per cent of surveyed students in 2022 already at the highest self‑reported level of intention to want a STEM career. There was a small to no net change in the question about ‘STEM subjects are important to my future study and career’: 0 per cent change in 2021 and an 8 per cent net increase in 2022. In 2022, 31 per cent of students already thought STEM subjects were critically important to their future study and career, compared to only 4 per cent who thought it wasn’t important. As can be seen in Figure 6, the majority of students already thought STEM subjects were important before participating in STEM CPP. The distribution of responses is very negatively skewed, reflecting the pre-existing high levels of engagement of students in STEM. This distribution demonstrates one of the challenges the program has in changing perceptions and intentions (compared to a more ‘normal’ distribution that might be expected in a random sample of students), but also to the issues with the program targeting approach and implementation model. That is, all the students in a particular class receive a similar ‘intervention’ regardless of their pre-existing attitudes, although it is accepted that students would engage to different levels in the common activities. Teachers were asked about their perceptions of improvements, if any, that STEM CPP activities had on their students’ awareness of STEM education and career pathways. As can be seen in Table 8, a large majority of teachers in 2022 felt that students’ awareness had increased, including around a third that thought it had increased ‘significantly.’ Figure 5. Net percentage change in awareness and intentions Figure 6. Student responses to ‘STEM subjects are important to my future study and career’ (2022) Table 8. Teacher perceptions of students’ awareness levels AWARENESS OF STEM EDUCATION PATHWAYS AWARENESS OF POTENTIAL STEM CAREERS No improvement 3% 0% Some improvement 30% 31% Moderate improvement 35% 33% Significant improvement 33% 36% Total 100% 100% Question 1b assessment: To what extent has STEM CPP increased students’ awareness about STEM education and career pathways? ASSESSMENT Insufficient evidence Did not meet objective Met objectives Exceeded objective Question 1c: To what extent has STEM CPP increased students’ transferrable skills? The extent to which student participants increased their transferrable skills was based on self-report surveys and also teacher perceptions. The majority of teachers indicated (in 2022) that students had improved skills as a result of participating in STEM CPP, with the most ‘significant improvements’ in problem solving (54 per cent) and communicating ideas (49 per cent) (see Table 9). Students were asked whether their confidence in different skill areas had increased as a result of participating in STEM CPP. As can be seen in Figure 7, most students in 2022 reported increases in ‘working in a team with others’ (91 per cent), problem solving abilities (89 per cent), and ‘communicating ideas to others’ (87 per cent). Male and female students reported similar levels of confidence improvements. Table 9. Teachers’ perceptions of students’ skill improvements (2022) WORKING IN A TEAM WITH THEIR PEERS COMMUNICATING THEIR IDEAS EFFECTIVELY TO OTHERS USING CRITICAL THINKING TO REASON AND DRAW CONCLUSIONS CREATIVELY THINKING ABOUT WAYS TO SOLVE PROBLEMS No improvement 0% 3% 3% 3% Some improvement 16% 15% 18% 10% Moderate improvement 41% 33% 42% 33% Significant improvement 43% 49% 37% 54% Total 100% 100% 100% 100% Figure 7. Student self-reported confidence in skill areas (2022) A STEM teacher involved in STEM CPP exemplified many comments by other teachers on the transferable ‘soft’ skills that students were able to develop as a result of the program: They really learned; they were able to meet new people...they learned how to communicate with each other. They had to learn to collaborate with each other rather than just sticking to their friends. Like each time they did a different group activity, they had to mix and collaborate with other students even if they didn’t want to. They had to learn to collaborate with each other, how to build teamwork skills, how to solve problems and challenges and be able to work out how to…build different parts of models to save time. STEM teacher/coordinator Beyond skills, one coordinator in a low SES school also identified how the virtual work experience component of STEM CPP may have built a sense of STEM identity through interactions with the scientists: But the boys that took part in the earlier – like, the stage 1 phase...they’re not knowing yet what direction they’re taking and to be able to do that virtual work experience and also have relationship – obviously virtual relationship but they really liked the scientists who were guiding them through the experience. They liked conducting that more adult conversation. Didn’t have those same assessment stressors or anything. It was a really validating experience for them to have that. Coordinator low-SES, high-CALD school Question 1c assessment: To what extent has STEM CPP increased students’ transferrable skills? ASSESSMENT Insufficient evidence Did not meet objective Met objectives Exceeded objective Question 2a: To what extent has STEM CPP increased the overall number of students participating in STEM education after Year 10? STEM CPP works with students in Years 9 and 10 who are already in STEM-related classes (47 per cent in core STEM subject classes, and 53 per cent in elective or extra‑curricular classes). In addition, the program works through teachers at the class level rather than with individual students, making the tracking of students post involvement in the program challenging. However, a memorandum of understanding (MOU) is in development that seeks to obtain data from all schools in New South Wales that will allow a school-level analysis of STEM subject selection (see Appendix A). This analysis will need to exclude Year 10 students (because they have already made their subject selections before most of STEM CPP activities have taken place) and therefore there is a 2 to 3 year lag between participation (in Year 9) and subject selection data (in Years 11 and 12). There is also a lag time to collect and provide the data to CSIRO for analysis. For example, the Year 11 subject selection data for STEM CPP Year 9 participants in 2021 will likely not be available until mid‑2024. What can be said about increases in participation in STEM after year 10 currently is based on the theory of change of the program, which posits that increased interest, knowledge, and skills will lead to increased probability of engagement and retention in STEM. Teachers were asked in a 2022 online survey ‘Overall, what impact has STEM CPP had on your students’ likelihood of studying STEM after Year 10?’ Of the 47 responses, 79 per cent said ‘moderate impact’, 14 per cent said ‘slight impact’, 5 per cent said ‘no impact’, and 2 per cent said ‘significant impact’. This indicates that teachers felt that STEM CPP was likely to be a factor, but possibly not the deciding factor, of whether students continued on to STEM. This aligns with contemporary research that highlights a myriad of contributing influences on decisions to enter and/or stay in STEM education pathways, including individual, family, school, and cultural/society level factors (many of which are outside the control of STEM education providers) (Murphy et al., 2019; Edwards et al., 2023). The analysis presented in previous questions gives a partial picture to address this question, including evidence of overall increases in confidence, interest, and intention; however, it will not be possible to assess this question robustly until the school-level data from the NSW Department of Education is made available for the evaluation. Question 2a assessment: To what extent has STEM CPP increased students’ transferrable skills? ASSESSMENT Insufficient evidence Did not meet objective Met objectives Exceeded objective Question 2b: To what extent has STEM CPP increased the diversity and number of high‑potential students participating in STEM education after Year 10? At this point in the life of STEM CPP, there is insufficient evidence to assess whether the diversity and number of ‘high potential’ students has increased after Year 10. There has been some debate within the program about what ‘high potential’ means and, given its ambiguous meaning and rationale for why it was included as a goal for the program, this aspect of the question has been put on hold. To clarify, some staff believe that all students have potential for STEM (but perhaps not an interest in it), and that delineating between ‘high potential’ and ‘low potential’ is not productive. However, there is a rationale for considering under-represented cohorts as ‘high potential’, such as young women and students from regional/remote and/or low socio-economic areas, because these cohorts are of course equally capable of succeeding in STEM education and career pathways but have not been represented at equivalent levels to their counterparts for a range of reasons. Because it is difficult to track students after participation in the program and even more difficult to attribute the ongoing STEM engagement of former participants to the program compared to a range of other influences, the best proxy at this point is to assess student self-reported changes in attitudes and intentions that could reasonably lead to future STEM engagement. Table 10 shows the net changes in STEM interest and intention to pursue a STEM career among male/female students (for 2022). CALD status was not analysed at this time but may be presented in future reports. As can been seen, STEM CPP has resulted in positive self-reported net changes among young women in interest (2021 and 2022) and intention to work in STEM (2022). These changes were greater compared to male students (except for intention to have a STEM job in 2021). Table 10. Net percentage change before and after STEM CPP by gender 2021 FEMALES 2021 MALES 2022 FEMALES 2022 MALES Interest in learning about STEM +24% +14% +13% -9% Wanting to have a job working in STEM -2% +19% +19% +7% Question 2b assessment: To what extent has STEM CPP increased the diversity and number of high-potential students participating in STEM education after Year 10? ASSESSMENT Insufficient evidence Did not meet objective Met objectives Exceeded objective Economic To what extent has the relationship between inputs and outputs been cost‑effective and to expected standards? STEM CPP has taken several years to implement into a fully formed program, but it is now useful to consider cost to outputs. There are several ways to assess the cost to output/outcome ratios. The first is a simple cost per participant ratio. As can be seen in Table 11, in 2019 the total cost of delivering STEM CPP (labour and operational service delivery, not including management and evaluation costs) divided by the number of student participants was $1,665. This reduced each year to a current ratio of $589 in 2022, a reduction of 64 per cent. This is indicative of the program gaining economies of scale, that is, delivering to more schools with similar investment in staff and operational costs. It is likely not useful to compare these costs per participant figures with other programs because of the many differences in context, organisational accounting, and program type; instead, it is more useful to monitor them annually as the program grows in order to understand trends and potential economies of scale. It is much more challenging to assess cost-outcome ratios for STEM CPP, primarily because there is an absence of longer term outcome data available at this time. However, to provide some baselines and initial indications of cost versus outcome, a calculation has been made on the effect size of increases in self-reported student intention to have a job in STEM and interest in STEM (see Table 12 – for 2021 and 2022 only, when outcome data is available). If the central focus of STEM CPP is to increase the number of students in STEM education and career pathways, then increases in self‑reported interest and intention is a reasonable proxy for longer term outcomes measured directly. It is assumed that the sample of survey respondents is representative of the overall population of program participants. The measure of cost effectiveness is an ‘effectiveness-cost ratio’, which is calculated by dividing the effect size for different outcomes by the annual cost per participant. Some comparative data exists using this measure for raising student achievement in mathematics (Yeh, 2010) as a broad comparison. Of the 25 initiatives reviewed in this study, the median annual cost per participant was $1,006 United States dollars, and the median cost-effectiveness ratio was 0.000083. Although STEM CPP does not have a focus on raising achievement, it is clear that the effectiveness-cost ratio is relatively strong in the areas of raising interest (0.000409 and 0.000044) and intention to have a job in STEM (0.000315 and 0.000170). These metrics will continue to be monitored as the program continues to be delivered. Table 11. Cost per participant CALENDAR YEAR TOTAL SERVICE DELIVERY COST (ACTUAL) STUDENT PARTICIPANTS ANNUAL COST PER PARTICIPANT 2019 $487,953 293 $1,665 2020 $366,788 250 $1,467 2021 $902,903 1,122 $805 2022 $1,315,019 2,231 $589 Note: The service delivery cost is split between Generation STEM funding, and CSIRO contributions in the form of ‘overheads.’ From 2019 to 2022, the Generation STEM proportion of the funding was 79 per cent, 70 per cent, 66 per cent, and 66 per cent, respectively. Table 12. Effectiveness-cost ratio CALENDAR YEAR COST PER PARTICIPANT OUTCOME AREA AND EFFECT SIZE EFFECTIVENESS-COST RATIO 2021 $805 Interest 0.3296 Intention 0.2539 0.000409 0.000315 2022 $589 Interest 0.0257 Intention 0.1003 0.000044 0.000170 Note: effect sizes calculated using Wilcoxon Signed Rank Test. Discussion Industry partnerships and real-world STEM Research consistently finds that STEM education and learning is effectively experienced in a hands-on approach that is student-centric, facilitates inquiry-based learning, and provides clear links between STEM skills and knowledge and everyday environments (Morris et al., 2021). There is also emphasis on the enrichment to STEM education for students if industry and STEM mentors played a more active role in their learning as a way of bridging the gap between classroom STEM subjects and awareness of real-world STEM application (Morris et al., 2021). However, there is also the acknowledgement that bringing STEM into everyday life or in partnership with industry is not within reach of many teachers or schools, primarily due to lack of time or resources, lack of interaction with enrichment programs or lack of awareness from non‑specialist or non-science teachers teaching STEM subjects, especially in pre-year 10 classes (Hackling et al., 2014). Areas for future consideration STEM CPP has been successfully designed and implemented in a range of schools and has been productive in partnering and understanding the needs of local councils and industries. The program delivery is efficient and there is evidence of positive student, teacher, and industry outcomes, although the key question of whether the program is resulting in increased numbers of students in STEM remains largely unanswered at this point. At approximately mid-way through Generation STEM funding, there are a number of issues the evaluation has highlighted that warrant further investigation and/or decisions for the remainder of the program funding in order to maximise benefits and program effectiveness, including: • refining the program model to target and focus on students who are not interested in STEM but could be • differentiated teacher training, student activities, and industry partnerships to focus on under‑represented students • refine program goals to include retention of students in STEM in addition to increasing the number of students in STEM • prepare teachers to place value on the inquiry-based skills rather than the particular industry/discipline to maximise industry connection opportunities • provide case studies of successful industry partnerships to teachers and potential industry partners and highlight key factors contributing to this • consider the proximity between mentor locations and schools they are matched with. Distance is a major barrier to face-to-face incursions • undertake greater assessment of teacher interest and desire for industry mentorship to better gauge partnership readiness • focus resources on matching teachers with greatest interest/investment in achieving success from industry partnerships • support existing industry relationships to leverage peer referral opportunities within their companies/ organisations to engage potential mentors or school presenters with different STEM skills and experience • develop an opt in program offering for schools to elect for a STEM industry professional to visit the classroom (as opposed to schools visiting sites or an established industry mentorship) to enable face-to-face engagement between industry and students. Differentiation and targeting The current STEM CPP program model results in many interested schools and interested/engaged teachers signing up. In addition, many of the students that are taking part are pre-existing STEM students with high levels of interest in STEM already. To increase the number of students in STEM, programs must be clear about their focus. STEM CPP’s program model is operationally efficient but may not be maximising effectiveness in targeting resources to those students most in need or most amenable to support. Table 13 outlines a conceptual grouping of four cohorts of students that are either currently engaged (groups 1 and 2) or not engaged in STEM (groups 3 and 4). To achieve the goal of increasing the number of students in STEM, only the targeting of group 3 (not engaged in STEM but potential to be engaged) will result in an overall increase in STEM students in New South Wales compared to the counterfactual of doing nothing. However, STEM CPP spreads considerable effort and investment (but not exclusively) to groups 1 and 2. Although it is important to retain the students in group 2, this will not lead to overall increases; it will only prevent decreases. The STEM CPP could make adjustments to better target effort and resources through differentiation of resources and training and to take a more outcome, rather than operational, focus to scaling up. That is, focus on reaching and supporting group 2 and 3 students, and less on group 1. Another consideration may be to consider explicitly adding ‘retain’ students in STEM along with ‘increase’ students in STEM as a program goal in the impact pathway(s), although this will be even more challenging to measure accurately. Table 13. Student categories for strategic focus GROUP POTENTIAL STRATEGY PROGRAM INVESTMENT LEVEL WILL PROGRAM INCREASE NUMBER OF STUDENTS ON STEM PATHWAYS? 1. Students engaged in STEM, and likely to remain engaged Continue to offer general opportunities and promote access to existing events, activities, resources Low No 2. Students engaged in STEM, but at (preventable) risk of not remaining in STEM5 Provide targeted support and opportunities and increase capabilities, motivation, and continue to build STEM identity Medium No, but important to retain in STEM (prevent decrease) 3. Students not engaged in STEM, but potential to be engaged Provide targeted engagement activities and support and begin to build and verify STEM identity High Yes 4. Students not engaged in STEM, with little to no interest in pursuing STEM Focus on raising overall levels of STEM awareness and skills that will be useful in any career in the future (i.e., 21st Century skills) Low No Year levels targeted Another program design feature that may be worthy of consideration is that Year 10 students’ inclusion in the program means that they have already made subject selections for Year 11 before the majority of the STEM CPP program activities have occurred in that academic school year. It should be noted that this is more prevalent in schools in their first year of involvement in STEM CPP, as existing schools have more activities available in terms 1 and 2. The general level of program intensity over the academic year is shown in Figure 8, noting that some major activities such as the Careers Expo do occur in term 2. If the goal of STEM CPP is to influence STEM subject selection, then Year 10 may be somewhat too late to change attitudes and behaviours. In this regard, the program team is already considering broadening the program inclusion criteria to include Year 8 students. STEM CPP is also a ‘year-based’ program; that is, it works with cohorts of students for one year. It is acknowledged that some students may take part in multiple years but this only occurs through happenstance rather than an intentional multi-year strategy. This year-based focus makes it difficult to track students from an evaluation perspective, but also difficult to maintain strengths and work with individual students over time. Building and maintaining a student’s STEM identity takes time and continuous effort, and consideration may need to be given on how to maintain support for participants, especially students from under-represented cohorts, over multiple years. Figure 8. STEM CPP program intensity over academic year To reverse the trend in declining STEM subject enrolments (see Figure 10) and reach recent high proportions (e.g., 2012 in Mathematics), New South Wales would need an additional 2,886 students to take Mathematics, 1,499 to take Sciences, and 4,329 to take Technologies (based on 2021 figures) (see Table 14). Given there are around 500 secondary schools in New South Wales, this equates to around 3 (Sciences) to 9 (Technologies) students per school. The STEM CPP program may consider an approach that targets a smaller number of students, but provides more in-depth, longer-lasting support. In addition, the team could work with schools to set targets for STEM enrolments. 5 This excludes individuals that make an informed decision not to pursue STEM based on personal choice, rather than negative experiences, barriers, lack of information or opportunity, etc. Related to this, the program team have observed that there may be a shift in focus of students, with Year 9 students more focused on potential STEM jobs, and Year 10 students more focused on subject selection. This could have implications for tailoring STEM CPP more to different ages or year levels to capitalise on these variations in interest areas. Scale of problem and tailoring program To reverse the trend in declining high school STEM subject enrolments (see Figure 9) and reach recent high proportions (e.g., 2012 in Mathematics), New South Wales would need an additional 2,886 students to take Mathematics, 1,499 to take Sciences, and 4,329 to take Technologies (based on 2021 figures) (see Table 14). Given there are around 500 secondary schools in New South Wales, this equates to around 3 (Sciences) to 9 (Technologies) students per school. Although these numbers would not necessarily meet the growing demand for STEM, it does demonstrate the realistic scale of addressing the problem at the level of individual high schools. The STEM CPP program may consider an approach that targets a smaller number of students, but provides more in-depth, longer-lasting support. In addition, the team could work with schools to set targets for STEM enrolments. Related to this, the program team have observed that there may be a shift in focus of students, with Year 9 students more focused on potential STEM jobs, and Year 10 students more focused on subject selection. This could have implications for tailoring STEM CPP more to different ages or year levels to capitalise on these variations in interest areas. Figure 9. STEM subject enrolments in Australia Table 14. Year 12 subject enrolment trends in Australia MATHEMATICS SCIENCES TECHNOLOGIES Current rate (2021 in Australia) 67.7% 50.0% 28.0% Recent high rate (Australia) 72.9% (2012) 52.7% (2017) 35.8% (2012) Total Year 12 students (2021) (Australia) 179,023 179,023 179,023 Number of additional Year 12 students needed to achieve recent high rate (Australia) 9,309 4,834 13,964 New South Wales share (approximately 31%) 2,886 1,499 4,329 Average number of students per high school (~500 high schools) 6 3 9 Factors that STEM CPP can influence Regardless of student interest in STEM and intention to pursue STEM, there are a number of individual, school level, and systemic factors, such as subject weighting for ATAR success and parental background, that heavily influence student subject selection (Jeffries, Curtis & Conner, 2020; Palmer, Burke & Aubusson, 2017). The program’s inclusion of industry mentors, inquiry-based learning, showcase events, etc. covers a range of these factors. However, there may be an opportunity to further focus the program to address other factors. For example, there may be an opportunity to further inform and influence what students study at the tertiary level. In addition, in line with the principle of ‘you can’t be what you can’t see’ (Gladstone & Cimpian, 2021) it may be useful to examine the gender of industry mentors. The program team have reported that STEM CPP is actively encouraging representation from women during site visits and discussion panels and are actively seeking more female STEM mentors. In both 2021 and 2022, student survey data indicated strongly that girls’ confidence was lower than their counterparts in almost all STEM subjects, and that the program was able to lift these self-reported confidence levels; therefore, further refining the program to focus on this strength-based area may be useful. For example, having more materials and training on how to engage young women in STEM. Of course, with greater targeting and focus, there may be fewer economies of scale and less ability to grow the program geographically and to a greater number of schools. However, the ability to genuinely and sustainably build individual students’ interest, engagement, and identity, and bridging the gap between self-perception and actual ability, requires substantial focus and targeting to achieve longer-term outcomes. One strength of the program that has been particularly encouraging has been changing students’ perceptions of the relevance of STEM to everyday life, potentially as a result of demonstrating the practical applications of STEM above and beyond theory, and also the focus on the inquiry projects. In relation to sustainability, the current model is heavily reliant on the interest and engagement levels of individual teachers; it may be effective to consider more school-wide activities that encourage a change in culture at the school level. Finally, research shows that interest in subject matters begins in early childhood education, where student perception of knowledge-building and learning experiences begin to form in the classroom (Stephenson et al., 2021). This is also where the gap in STEM interest and confidence begins to develop between girls and boys, where a mixture of unconscious gender biases, culture, the social environment and inadequate school initiatives cause a lag in the participation and engagement of girls in STEM-related activities (Speldewinde & Campbell, 2021; Stephenson et al., 2021). This gap, when left undisturbed, continues to widen through primary school and high school, especially with the absence of targeted STEM programs designed exclusively for girls (McKinnon, 2020). STEM CPP may consider earlier engagement with students in lower high school or primary school. Recommendations The recommendations for STEM CPP comprise several primary ones that focus on the refinement of the program model and a number of secondary ones that focus on more operational areas. Primary recommendations • Continue to build on the commendable implementation and delivery of STEM CPP in terms of resource development, relationship building, and ability to raise interest and intention levels among student participants. • Consider refining the program model to target and focus on students who are not interested in STEM but could be. This could involve fewer students being involved, potentially offering more individualised rather than class-based programming. • Consider offering more differentiated teacher training, student activities, and industry partnerships to focus on the ‘group 3’ students. Secondary recommendations • Refine the program goals to include ‘retention’ of students in STEM in addition to ‘increasing’ the number of students in STEM. • Explore ways to utilise potential industry mentors when the first match doesn’t work out with the school. These mentors could be redirected into other activities to maximise positive engagement. • Prepare teachers to place value on the inquiry-based skills rather than the particular industry/discipline to maximise industry connection opportunities. • Provide case studies of successful industry partnerships to teachers and potential industry partners and highlight key factors contributing to this. • Consider the proximity between mentor locations and schools they are matched with. Distance is a major barrier to face-to-face incursions. • Undertake greater assessment of teacher interest and desire for industry mentorship to better gauge partnership readiness. • Focus resources on matching teachers with greatest interest/investment in achieving success from industry partnerships. • Support existing industry relationships to leverage peer referral opportunities within their companies/ organisations to engage potential mentors or school presenters with different STEM skills and experience. • Develop an opt in program offering for schools to elect for a STEM industry professional to visit the classroom (as opposed to schools visiting sites or an established industry mentorship) to enable face-to-face engagement between industry and students. Credit: Charles Sturt University Generation STEM Links Methodology The evaluation of the Generation STEM Links program is based on a mixed methods approach outlined in Table 15. Data were collected on a rolling basis throughout 2022 from student and industry participants via a brief voluntary online survey. This was distributed by email from the Impact and Evaluation team once both parties had met the exit requirements from the program. If indicated in their survey response, a voluntary semi-structured interview may be undertaken by the Impact and Evaluation team with students and industry supervisors. The CSSHREC provided approval to undertake all monitoring and evaluation activities. Process How was the program implemented? The Generation STEM Links program model involves attracting industry partners and matching them with a pool of students. The students are recruited via tertiary education providers in NSW, with an assessment of their suitability being undertaken, and offering industry partners a short-list of potential interns that have been sourced and vetted by the program. Throughout the program, industry partners and students are also provided support and mentoring, including check-in meetings, resolving any issues that arise among students and placement employers, and ensuring students are experiencing expected benefits from the program. Requirements of the program include: • 200 hours of internship must be completed • students must be formally employed by the business and paid at least the minimum wage of $25 per hour from the business (Generation STEM Links provides a $2,500 placement grant to the business on completion of the internship) • the business must provide proof of payment to CSIRO before the grant subsidy is paid. Table 15. Data collection methods for Generation STEM Links evaluation METHOD SAMPLE NOTES (IF APPLICABLE) Student self-report online general survey 11 8 students indicated their willingness to be contacted for an interview. 1 student indicated that they did not wish to participate in a destination survey in future years. Only two female students are included. An additional survey was partially completed (i.e., 11 complete surveys, 1 partially completed) Student semi-structured interview 2 Industry supervisor survey 10 Industry supervisor interview 3 Case studies 3* *Case studies were developed using the data collected in the respective student and industry interviews Program administrative data n/a Program design Program staff report that during the design phase of Generation STEM Links, it was identified that the main bottleneck for the program would be finding and onboarding industry partners to host interns. In addition, a key learning was that Generation STEM Links would need to be differentiated from unpaid ‘Work Integrated Learning (WIL)’ programs primarily delivered by universities and VET providers such as TAFE, which often provide limited matching of students and businesses and often offered little to no support throughout the placement. As a result, Generation STEM Links went with a fully facilitated model in which the program supports the company to refine its ‘on the job’ project brief, advertise to potential students, review applications, conduct screening interviews and present a shortlist to the business. In the Generation STEM Links program model, the business chooses the student and presents the employment contract. Once the student has been confirmed, there is a CSIRO-led onboarding meeting, a mid-term follow-up, and close-out meetings. There are also interventions where necessary when assistance is requested by business or student. The facilitated model is more resource intensive than WIL programs but can give businesses assurances that the process will minimise risk and that the significant resources it puts into the hosting will result in a well-matched, engaged student. The program team feel that this has become a strong value proposition when attracting new businesses and has had the follow-on benefit that many businesses have returned for subsequent internships. Repeat industry internships are more efficiently delivered as they are well prepared for specific projects through to onboarding and the internships themselves. Program guideline changes At the outset of the program implementation in 2021, only STEM businesses in Generation STEM’s location priorities were eligible to participate in the program. During the year, these guidelines were streamlined and simplified, including expanding the geographical scope of the program to organisations anywhere in NSW. In addition, the eligibility criteria were simplified to cover any organisation with an Australian Business Number (ABN) and a STEM need within their business. To ensure suitability of the placement opportunities for students, a further criterion was included that at least one employee must have the relevant STEM skills, experience, and qualifications to supervise a student placement. Program staff report that the geographical expansion of the program to all of NSW (i.e., eliminating complicated location boundary criteria) has had the greatest impact in increasing interest from businesses. Applications and coverage Students enter the program through two ways: through an application in response to an advertised opportunity (n = 117 in 2022) or through an application to be kept on file for future opportunities (n = 154 in 2022). As the program grew and more industry partners have been onboarded, the ‘on file’ students have been sourced for potential matches to new opportunities. Figure 10 shows the attributes of Generation STEM Links applicants by gender. Male applicants outnumbered female applicants by a factor of two. Substantial proportions of applicants were from lower socio-economic areas and were culturally diverse. Only one in ten applicants were provided a placement, demonstrating how competitive the program has been in its first year. It should be noted that less than half of the applicants applied for advertised industry positions, with the remainder lodging expressions of interest for future opportunities. Table 16 shows the number of placement locations, industry sectors, educational institutions and placement organisations involved in Generation STEM Links, demonstrating a diversity of geography and sector. In addition, most common industry sector for student placements was advanced manufacturing (n=12) and professional and financial services (n=4). Figure 10. Generation STEM Links student applicant demographics Note: Total eligible students applied n = 271. For each demographic category, the bar represents the percentage of all applicants in that category. For example, 43 per cent of applicants were culturally diverse, 12 per cent of applicants were female and culturally diverse, and 30 per cent of applicants were male and culturally diverse. The ‘given placement’ bars represent the proportion of applicants in that category that were placed: 11 per cent of all applicants were placed, 14 per cent of all female applicants were placed, and 10 per cent of all male applicants were placed. Table 16. Generation STEM Links placement data STAKEHOLDERS AND LOCATIONS PLACEMENT LOCATIONS (7) INDUSTRY SECTORS (7) EDUCATIONAL INSTITUTIONS (9) Central Sydney Western Sydney Illawarra-Shoalhaven Riverina-Murray Hunter New England Southern Sydney Advanced manufacturing (12 placements) Professional and financial services (4 placements) Agribusiness and food (3 placements) Mining (2 placements) Healthcare (1 placement) Sustainability (1 placement) Other (7 placements) Charles Sturt University Macquarie University TAFE Ultimo University of Newcastle University of New England University of New South Wales University of Technology Sydney University of Wollongong Western Sydney University Which aspects are working well? Which aspects can be improved? How? Approximately 54 per cent (n=141) of the student applications received for the Generation STEM Links program involved students studying engineering (see Figure 11). One third of the students involved in the placements in 2022 (n=10) are studying engineering. Engineering degrees require an internship as compulsory learning; therefore; these students are actively seeking internship programs to fulfil their academic requirements for graduation. Students report that of the internship programs on offer, Generation STEM Links has a strong value proposition offering the greatest opportunity to gain hands-on experience and develop transferable skills. In addition, engineering overall has a culture of WIL, which makes it easier to find industry partners looking to take on student interns. Figure 11. Student applicant areas of interest versus industry project focus Note: When applying, students are asked to indicate three preferred interest areas. Data represented is inclusive of all three areas (student identified interest areas n = 695; Industry project identified areas n=138). The other ones I looked at I was more so just applying for trying to get the internship. Wasn’t really anything that was direct. I was, “If I get it, I get it.” Might not enjoy it but it covers the hours to meet the requirement. But this is definitely [the] ideal case.” Student I put mechanical or mechatronics because that’s the two majors that I do. But other than that, I wasn’t too fussed on exactly what it was. I was just looking for an opportunity. I had other internships lined up as well, but I just found that with this one, especially with the backing of CSIRO and it being in the industry that was more interesting to me compared to the other one, the work just seems better, and I could learn more from the senior engineer here. Student Offering a paid internship opportunity is a significant incentivising component of the Generation STEM Links model, as many internships elsewhere in the sector are unpaid. The majority of applicants (79 per cent) lived in a lower socio-economic area (according to postcode indexation) which suggests that this program is addressing a known barrier for participation in voluntary internships. Although the program is attracting students from culturally and linguistically diverse backgrounds (43 per cent, n = 114), this is not being reflected as proportionately in the placements that have been undertaken to date (see Figure 12 below). Of the 29 placements, only 17 per cent (n = 5) of students identified as culturally and linguistically diverse. However, due to the two different applicant streams (specific applications and general pool), the proportions were higher when looking at students who specifically applied for the 29 placements: 117 students applied, 47 (or 40 per cent) were culturally diverse, and those 47 students only applied for 13 of the placements. Looking at it in this way, CALD students were self-selecting out of many placement opportunities. Nevertheless, this general issue suggests an opportunity to identify why these differences are reflected and apply continuous improvement processes to address this. One potential explanation is that many CALD student applicants may have been located in Western Sydney where there were fewer placement opportunities. In addition, the program team reports that there were some challenges identifying CALD students and clarifying the number of applicants, specifically that some CALD students are not self-identifying in their applications. Figure 12. Generation STEM Links priority cohorts Note: Total eligible applications n = 271; Total placements n =29. The data in the chart represents the proportion of applicants and placements per demographic category. For example, 29 per cent of all applicants were female, and 13 per cent of the students that receive placements were female. Impact and outcomes To what extent has Generation STEM Links increased students’ (a) technical and enterprise skills (b) awareness about potential STEM career pathways and (c) commitment to work in STEM? This program is in its infancy and there are only early indications and small amounts of data to assess the achievement of outcomes at this point. However, there are insights reflected in the data to suggest the effect the program may have as it continues to scale as planned. Although the sample size is too small to conduct tests of statistical significance, students reported a net positive percentage change in all domains (see Figure 14) measuring the impact the work placement had on students’ skills, understanding, and interest in pursuing STEM jobs. The greatest change was observed for students’ self‑reported core work-readiness skills (e.g., communication skills, working in a team, planning etc.) Self-report qualitative data from students indicated that skills, awareness, and commitment to STEM increased as a result of participating in the program. One student highlighted how their career options had widened due to the work placement: This placement has broadened my perceived career options. I am passionate about the environment and originally, I wanted to use my degree in the renewable energy sector. Having worked at [company] for six months, I am now considering my career options within the food and beverage industry. I would also be very interested in transitioning into the design of machines like the ones I have been working on in the plant. Student Another student reported increased confidence in both their skills and career choice: Now after the placement I felt much more confidence in what I do and my choice of choosing courses. I was once a Biochemistry major who decided to make a shift into the tech industry. So, through being exposed to the various technical challenges in the business, solving them and creating new products, it’s given me much more confidence knowing that what I do serve the company’s customer pool in a meaningful manner. Student One student expressed the benefit of seeing what a STEM job is like in real life: It has shown me the type of work an engineer would do. At the same time, it has shown me the possibility for career development in the future into work other than technical design work. Student While another student emphasised the application of skills as being particularly beneficial: It was refreshing to see the application of developed skills. Student The self-report survey (n=11) provided evidence that participating students overall felt a net positive change among all areas (see Figure 13), particularly confidence in core work-readiness skills (78 per cent net positive change in the sample), knowing about the reality of working in STEM (67 per cent), and professional working relationships (67 per cent). These practical skills and networking‑related changes emphasise the benefits of the real-world context of Generation STEM Links; areas that were more generic (e.g., knowing about STEM area of interest, confidence in tertiary studies) exhibited lower net changes. Figure 13. Change in student confidence in STEM skills and knowledge Assessment: To what extent has Generation STEM Links increased students’ (a) technical and enterprise skills (b) awareness about potential STEM career pathways and (c) commitment to work in STEM? ASSESSMENT Insufficient evidence Did not meet objective Met objectives Exceeded objective To what extent has Generation STEM Links increased the number and type of tertiary students working in STEM jobs? The program has not reached a sufficient scale to meaningfully assess this outcome. There are emerging indications that the program may offer an opportunity to broaden the recruitment pipeline of businesses from the traditional source of universities to include more VET students to meet demand. The Generation STEM Links program team is currently exploring ways to engage with the VET sector more deeply. Industry partners highlighted this potential: We are exploring more university and even TAFE placements in other areas [outside engineering] within our company. Industry partner But where we were having issues with getting the right people was actually in our manufacturing group, where we actually physically make [product] here in [location]. And I said to my colleague “A TAFE student would be more suitable – for the roles we had in mind – than a university degree student”, but we hadn’t sort of tried TAFE. Industry partner Assessment: To what extent has Generation STEM Links increased the number and type of tertiary students working in STEM jobs? ASSESSMENT Insufficient evidence Did not meet objective Met objectives Exceeded objective To what extent has Generation STEM Links increased the capacity of industry? The program appears to be meeting the needs of several of the host organisations involved, with 86 per cent (n = 12 out of 14) of students who have completed their placement by December 2022 being offered further employment by the industry partner (9 of the students accepted those offers, with the other 3 students being unable to accept due to location issues and a student deciding to change their focus). All of the businesses who have hosted at least one complete placement have offered further employment to an intern. Even more importantly now, it is so difficult to attract staff into the manufacturing area. Unfortunately, ‘manufacturing’ isn’t one of those buzz words that people look for a career in. Industry partner Although the program has not reached a sufficient scale to meaningfully assess this outcome, qualitative insights indicate the program offers the potential to increase the capacity of industry. Primarily, this appears to be through a highly facilitated recruitment pipeline that leverages the trust and recognition of CSIRO. The candidates we were offered were all good but the candidate we have chosen is an exceptional fit for our business. I doubt we would have been able to achieve this without the aid of the CSIRO. Industry partner Assessment: To what extent has Generation STEM Links increased the capacity of industry? ASSESSMENT Insufficient evidence Did not meet objective Met objectives Exceeded objective Economic To what extent has the relationship between inputs and outputs been cost‑effective and to expected standards? An initial (baseline) cost per placement ratio was calculated for Generation STEM Links for the last six months of the 2022 calendar year (July to December), after the bulk of establishment work had been done and the majority of service delivery effort was directed towards placements. The total costs to deliver the placements program, including operational, labour, and subsidy costs related to service delivery, was $242,9406 . The total number of completed placements in the second half of 2022 was 297 , resulting in a cost per placement of $8,377. It should be noted that placements involve significant effort to promote, select students, source industry partners, provide training and information, etc. Therefore, the cost per ‘placement’ is an outcome unit that encompasses multiple people, processes, and activities, and is not directly comparable to different types of programs, such as STEM CPP. For context, a recent study found that the average cost of hiring a candidate in Australia was $23,860 in 2021, with an average time to hire of 40 days (Morris-Reade, 2022). The cost per placement is a valuable measure because as the program scales up, processes are streamlined, and industry partnerships are solidified, it will be important to track anticipated ratio decreases. Discussion Initial insights from the program implementation in 2022 indicate the program application process is accessible and that there is demand for the students that the program targets. Many of the program applicants and the program placements were engineering focused, which suggests that the program is effectively reaching engineering students, offering an attractive value proposition to both student and industry stakeholders. It may be worth considering how other disciplines can be more represented. However, as employment outcomes in some non‑engineering areas are not overly strong. The program matching has ensured either relative or, as seen with women, a higher proportion reflected in the placement cohort (38 per cent) than the application cohort (29 per cent). In contrast, the program matching has resulted in less students from CALD backgrounds included in the placement cohort (17 per cent) than the application cohort (43 per cent). It is important that this difference is further explored so that potential barriers (e.g., language) can be understood and addressed, where possible. Self-report data from industry stakeholders indicates that the value proposition of the program for employers is access to a high quality recruitment pipeline. Despite some organisations preferring to assess the applications for their industry project themselves, the majority are assessed and interviewed initially by the CSIRO program team prior to a matching interview with the potential employer, which provides an opportunity for students to ask questions and for the industry partner to identify their specific needs. Industry reported that their participation was influenced by the financial subsidy and the reputation of CSIRO. It will be important to monitor industry participation data as the program continues to assess the sustainability of the program model. Recommendations • Investigate the benefits in increasing program focus on attracting and matching VET students with placements to meet industry demand in manufacturing.8 • Identify potential barriers or bias in the placement assessment process and final industry partner selections that may be contributing to the discrepancy between CALD student involvement in the placement pool compared to representation within the applicant pool. • Increase workflow automation in the program application and matching process to support scaling the program by reducing the resource intensiveness of these processes.9 • Continue to undertake monitoring of program expenditure to ensure the cost/outcome and cost/benefit ratios are within expected ranges, and to quantify impacts on industry. • Leverage the new dedicated Industry Engagement Manager to increase participation of different industry sectors involved in the program (e.g., biochemistry (see Figure 12)) to support expansion of placement opportunities. In addition, the program could target locations that are identified as important for reaching under-represented student cohorts. 6 Around 66.8% was provided by Generation STEM funding and 33.2% was provided by CSIRO in the form of an ‘overhead’ cost. 7 An additional 31 placements were in the pipeline in 2022, and if completed, will be counted in the 2023 calculation. 8 The program team report that working with the VET sector has been challenging due to the highly decentralised nature of TAFE NSW campuses, with no straightforward way of advertising or recruiting students across the whole sector or regionally. In addition, the capability of individual campuses and cohorts is difficult to ascertain. 9 The program team feel that improving program efficiencies was a significant focus in 2022 with iterative continuous improvement across the program based on real-time learning about processes such as how to best advertise jobs, streamlining application processes and questions, and onboarding industry partners. The team report that they are reasonably confident that the ‘big wins’ were made in 2022. Deadly in Generation STEM Methodology The Deadly in Generation STEM (DiGS) program was implemented for the first time in 2022 in two NSW locations – Kamilaroi Country (Moree) and Dharawal Country (Illawarra). It was decided to focus data collection activities for the purposes of evaluation with the participants and community stakeholders in Illawarra as the primary case study location for the program. This was due primarily to timing and resource challenges. As a result, participant insights reflect the local experience in Illawarra only, although it is likely that some findings would be generalisable. Due to the impacts of COVID-19 on delivery of DiGS in Illawarra in 2022, there were only a small number of students involved in the program by the end of the year (n = 7). The key stakeholders involved in delivering the program, including a CSIRO staff member, a STEM mentor, and a cultural mentor, recommended that that they lead the yarning with students, and an informal yarning session with community stakeholders, given the existing relationships and familiarity. It was felt that it would be better for the local Indigenous people who run the program to elicit feedback on the program with participants, rather than the non-Indigenous evaluation team who had no relationships with students for 2022. Data collection activities were undertaken in late 2022 and consisted of: • yarning session with students (where parental/guardian consent has been obtained) who had been involved in the program. This session was facilitated by a local cultural mentor recognised as a community leader, and a local Aboriginal STEM professional mentor, who were both guided by questions developed by the Impact and Evaluation team. Students were split into two groups, according to gender, with a facilitator each • yarning session with community members (e.g., parents, teachers, education workers, etc.) facilitated by the CSIRO Project Officer based in Illawarra • very brief online survey for adult community members accessible via QR code to Qualtrics survey platform as an additional method of sharing their feedback, especially if they may not have felt comfortable saying something in the group setting • discussion between Impact and Evaluation team and the Generation STEM leadership to explore the delivery partnership in Moree. Program implementation reflections gathered by Generation STEM leadership with key stakeholder in Moree. Process How was DiGS implemented? In both Moree and Illawarra, the DiGS program model was co-designed in collaboration with community stakeholders. The community identified their needs and the program elements they felt were important. Across the two locations, there were many commonalities that emerged from the co-design process although there were some differences in the models that were implemented as a result. In Moree, the model involved an in-school component which was delivered via the Inquiry for Indigenous Science Students (I²S²) program, overseen by CSIRO. Originally, both locations intended to undertake a camp; however, this was not feasible, due to risk of disruptions relating to COVID-19. Instead, both locations adapted to deliver four themed Immersion Days spread over a number of months. Kamilaroi Country (Moree) It is useful to understand and briefly document the process in Moree10 . Originally, both locations had a dedicated Project Officer working with a key community stakeholder acting in a central collaborator capacity (a similar role as the Councils in the STEM CPP project). The community engagement process had identified the importance of involving the Moree Sports, Health, Arts and Education (SHAE) Academy due to their connections with the community and with young people, including the familiarity of the location. CSIRO and the SHAE Academy entered into a collaboration agreement, with the CSIRO Project Officer having responsibility for delivering the project and working closely with the Academy. The first of the four Immersion Days with students were successfully delivered in Moree in collaboration with the Academy prior to the departure of the Project Officer (who moved to CSIRO’s Young Indigenous Women’s STEM Academy team). The CSIRO team did look to find a suitable replacement candidate in Moree; however, no candidates applied for the position. To address this delivery capacity gap, CSIRO initiated delivery partner conversations with SHAE Academy, noting that CSIRO would need to maintain oversight of the STEM expertise for the in-school component. Implementation of this delivery partnership was somewhat challenging because of a lack of capacity in SHAE Academy, as was formalising an agreement between partners. The interim solution was to extend the collaboration agreement with some funding included to deliver the remaining Immersion Days. All remaining Immersion Days were held, however both CSIRO and SHAE Academy observed that the quality of outputs and outcomes were not as a strong as the first day, when the CSIRO Project Officer was involved. Despite some of the barriers with establishing and implementation of the formal delivery partnership, a strong relationship has been maintained between CSIRO and SHAE Academy. Staff from the Academy have emphasised the ongoing collaborative involvement of a central community stakeholder as critical to the success of a program such as DiGS to provide stability and continuity. There is an ongoing shared interest in continuing to work together to deliver a STEM engagement program for Aboriginal young people on Kamilaroi Country. Dharawal Country (Illawarra) Delivery of the program in 2022 involved a close collaborative partnership between a locally recognised cultural mentor, a CSIRO Project Officer, and a local STEM mentor. Originally, a partnership with the University of Wollongong was intended as an element of the implementation of the DiGS program in Illawarra. Due to local stakeholder relationships and interdependencies, this partnership was not able to be established. In contrast to Moree, the program implementation in Illawarra was supported by a dedicated Project Officer throughout 2022. In addition, there was a clearly recognised cultural mentor in the community to lead and facilitate the Indigenous knowledge framework underpinning the program. What were the particular elements of the program and context that made a difference? Based on the yarning session, observational data, and program team feedback: Common themes from both locations • Hands on activities were perceived to be effective to engage students, such as weaving, making medicine tea, throwing boomerangs, catching fish. • Local knowledge holders were important to engage with the students. • Strengthening knowledge and understanding of culture and history leads to increased pride and confidence in identity. 10 Evaluation reports are often the only public-facing source of information about program processes that can provide valuable learning opportunities for future programs. Kamilaroi Country (Moree) • Hosting at SHAE Academy, which is a culturally safe and secure organisation and site. • The program works best when CSIRO can do the groundwork to coordinate local knowledge holder’s involvement once introductions and connections have been made through the Academy. • The structure of four separate days made it difficult to engage with Moree Secondary School – the camp in 2023 will be a preferable structure because it will allow engagement with a wider range of schools. • Potentially more differentiation to students with different levels of interest. For example, the majority of students were highly engaged in the hands-on activities, but there was a high degree of variability observed in engagement levels during other activities. • Starting and ending days with cultural knowledge and Dreaming stories worked well. • Not using ‘STEM’ acronym and/or language but rather encouraging students to walk in the footsteps of their ancestors as the First Scientists, Bakers, Engineers and Mathematicians – encouraging cultural pride and their responsibility to care for Country. Dharawal Country (Illawarra) • Meeting students from other schools and forming connections with other Aboriginal young people they otherwise might not have met. Several students mentioned this: We pretty much don’t – say like us boys, we rarely ever get together. Like I’ve never been – besides footy, I’ve never met never met them in my life. Probably wouldn’t have talked to them either too… now I know where they’re from, they’re Indigenous. I treat them like my cousins and my family. Student And if you’re ever stuck on something or you need help with like an Indigenous thing, I could call [Student] or I can call [Student] and they’ll pretty much help me if I need to – because I know they probably would know, or give me a sense of idea what I should need to know to learn it. Student Being outside and on Country and grounding STEM in an Indigenous framework made it seem less boring and more relevant to students. For example: Yeah. I thought we were just going to be like sitting in a room doing boring stuff. Student Understanding how to use technology to care for Country, for example, drones to monitor ocean patterns or observe koalas. Impact and outcomes What outcomes are emerging from the initial implementation of DiGS? Students involved in the yarning session held on Dharawal Country (Illawarra) reported an increased understanding of Indigenous STEM knowledges as a result of their participation. They discussed how this knowledge made them feel, reflecting a greater sense of belonging and/or wellbeing. This was highlighted by several students: It shows how advanced we were…It shows how intelligent and caring our ancestors were. Student Through the program, I’ve definitely got a bigger sense of pride in my culture. Especially day one, just how [cultural mentor] was telling us how important and impactful our people were definitely inspired something that makes me think, damn, we are important people. It’s really an amazing culture to be a part of. So definitely a bigger sense of pride in my Aboriginality. And also, a fair bit more understanding in it. So, I know more about who I am through that now. I know where my people are from. That sort of stuff. I know more people like me through the program. Student Some of the things [cultural mentor] showed us, showed that we’re not nothing. Some of the techniques they use, like back in…ages away, definitely impact like how things are now. Student Discussion amongst the community stakeholders highlighted that some students have an increased confidence and interest in STEM, which is evident at home and at school. For example: Personally, for my child, he’s gone from like a [class five] and he’s made – I think this has opened up his mind and he’s moved to the top ten…The whole school is talking about how – they just don’t know where he comes from. And I think this has opened up the importance and just made it more relatable. Parent Students reported an increased connection to local community and Country, including increased cultural awareness. A few students emphasised this: Yeah. The way you connect things from our culture and then in modern day time. Student The biggest thing we learnt about was definitely local culture and what our people have used their whole lives. How that impacts modern day society. Student Students involved in the yarning discussed how they had engaged in knowledge sharing and capacity building across the community. Even that, just do that, refreshes everyone and gets them talking and as soon as like all the parents came, we went with our parents and we went back, and we [told our parents what we heard.] Most of us probably misheard or didn’t hear what Uncle said or what you said, or the other people said. It’s that – like other people are hearing what you maybe didn’t hear. Or you heard what they didn’t hear. And you just teach each other. Student Finally, a cost per participant ratio was calculated to establish a baseline for future years. The service delivery costs for 2022 were $344,030 and the total student participants who completed the program were 27, resulting in a cost per participant of $12,742. This ratio would be expected to decrease as the first year of program delivery is resource intensive, for example, setting up partnerships and developing processes and resources. In addition, a number of participants withdrew from the program throughout the year. Discussion The available data suggests that the piloted model for the DiGS program has been effective in achieving some of the shorter-term outcomes reflected in the Impact Pathway for individual student participants. In particular, participating in the program appears to have increased understanding of Indigenous STEM knowledges amongst students and increased connection to local community and Country. The students’ reflections suggest that this understanding may have extended to others in their families and community, through opportunities for shared reflection and connection that the Immersion Days incorporated. There was a rich discussion between the students involved in the male student group regarding the benefits of the camp model versus the interconnected Immersion days. The students demonstrated a lot of thought and consideration regarding these factors, suggesting that further exploration with the students would be beneficial to continue to co-design the DiGS program. Although both male and female students were involved in the qualitative data collection, the depth of discussion was noticeably greater amongst the male students. The implementation of the program in Illawarra primarily involved male knowledge holders and STEM role models. Although no attribution can be made with certainty, similar gender considerations as have been applied in the STEM CPP program may be relevant for future delivery of DiGS. Insights from community stakeholders was limited; however, early indications point to the program model being well received, with ongoing interest in its availability within both regions. For ongoing implementation of the project and subsequent evaluation, consideration is needed as to how to respond to feasibility factors to support embedding Indigenous experts into the Impact and Evaluation team, especially for data collection activities. Although the students were split into male and female groups for the student focus group discussions, there is richer data evident amongst the male participants. Both facilitators were local Aboriginal cultural and STEM mentors, however both were men. It is worth considering whether a female led discussion may have yielded greater insights from the female students. Recommendations What can be strengthened or adapted to reflect initial community feedback? • Undertake further discussion with young people regarding the appropriateness of camps and/or Immersion days to explore the considerations identified by the participants in Illawarra. • Involve young people as champions and ambassadors of the program to engage potential participants.11 • Consider adapting impact pathway to include emerging outcomes of importance or undertaking further co-design to reflect on how this is valued, such as transferable skills versus knowledge sharing and teaching. Specific to Kamilaroi Country (Moree) • SHAE Academy to take over organisation of consent and information forms so that they can communicate with individual families and coordinate this process through their relationships and proven place within the community.12 Although this approach would increase the cultural safety for families, if this approach was adopted, any processes would need to be reviewed by the CSIRO Privacy team prior to use. • CSIRO to work with SHAE Academy Chief Executive Officer (CEO) as closely as possible to shape the localised approach to recruitment of new Project Officer for 2023 as the recruitment approach in 2022 is likely to have been a barrier for potential candidates. Learning and reflections gathered among Generation STEM team and SHAE Academy implemented into planning for 2023 and built into sustainability considerations. • Risk engagement of students within Group 3 (see Table 13) in Moree if activities aren’t well planned and executed, as young people are already highly disengaged. 11 The program team reports this is being planned for Illawara in 2023. 12 The program team reports this happened in the later immersion days in 2022. Specific to Dharawal Country (Illawarra) • Plan Immersion Days or future days on camp with students’ experience of moving through the day(s) in mind (including behavioural and cultural considerations). I felt like day two needed to be split into two. Just like another day, because there was just that much activity going on, it was kind of hard to navigate. And like [another student] said, everyone got a bit restless by the end. Student Ensure communications about the program highlight the student reported benefits (see Impact and Outcomes above) and address barriers (perceived and systemic). …with my daughter, she was worried that it was all about maths. And she doesn’t like maths. “I can’t do it. I can’t go to that program….” Parent Also, maybe like the education level and some of the kids feeling like that’s just for the nerdy kids. So that stigma. And I guess the other thing, the money side of things as well. Like, “Oh, I’d never be able to afford to pay for a uni degree” or “I can’t. How am I meant to get there? I haven’t got the money.” All that sort of stuff. Community member Things we could improve, the shirts. They’re really ugly. Student Ensure effective dissemination of information with stakeholders to contribute to increased participation (e.g., let schools already engaged in the program know that they can send more students). Reach students earlier within the educational setting, for example in Year 7 before school attendance may start to drop in Years 9 and 10.13 But the start of school’s really – that’s really important once they finish primary school as well, because a lot of the time kids can go off on a path that may not be the one that they were in when they were in primary school. Because there’s all those different relationships and friendship groups. And then that has a big impact on where they go from there. It’s not about the education, it’s about the friendship groups a lot. Community member 13 The program team reports that a primary school pilot is being trialled in 2023 with the Inquiry for Indigenous Science Students teacher professional learning. Data insights The governance body overseeing the Generation STEM initiative (Consultative Council), which includes members from industry and the NSW Government, in addition to the Trustee of SIEF, have expressed an interest in building a better evidence base for ‘what works’ in STEM education and also a platform to utilise this evidence to understand the degree to which Generation STEM programs are effective. To address this need, two data insights projects have been established and, although still in their nascent stages, produced some initial findings up to early 2023. Data analytics A project to apply predictive analytics to an education dataset to understand the factors that influence STEM education decisions and outcomes in high school was established in late 2021. Human research ethics approval was obtained from the CSIRO Human Research Ethics Committee and a collaboration agreement was signed with a Catholic education diocese in New South Wales to undertake the analytics using available data from their schools. Two Doctor of Philosophy (PhD) students14 from the Australia National University with expertise in artificial intelligence and predictive analytics were engaged by CSIRO to undertake the analyses. Initial analyses have been completed with the final analyses to be concluded by April 2023 and presented in a separate report or research article. The dataset comprised 51,851 students from 73 schools, although there were significant amounts of missing data (primarily from students entering and leaving the Diocese’s system). More detailed analyses will be outlined in a separate report, but initial exploratory results indicate: • Higher academic performance in STEM subjects up to year 10 is positively correlated with becoming an High School Certificate (HSC) STEM student. • Female students tend to perform better in HSC STEM subjects, even though a higher proportion of HSC STEM students are male. • Students with parents with a qualification of bachelor’s degree and above are more likely to select STEM for HSC and to perform better in those subjects than other students. • Most students do not ‘change course’, that is, students that don’t take STEM subjects up to year 10 do not usually go on to do HSC STEM subjects. • HSC STEM subjects are difficult; most students who did HSC STEM subjects did academically better in their non‑STEM HSC subjects. As of February 2023, the analytics portion of this project had only very initial results using machine learning models. The objective was to predict whether a student followed an HSC STEM stream (defined as taking either 50 or 80 per cent or more STEM subjects in year 12) based on available independent variables, such as previous years’ academic performance, personal attributes, family attributes, and school details. A total of 12 machine learning models were tested resulting in accuracy levels between 66 per cent and 72 per cent for the 50 per cent definition, and up to 85 per cent for the 80 per cent definition. That is, based on a student’s pre-Year 12 data, the models could predict whether the student would be a STEM HSC student 66 to 85 per cent of the time. The best performing model for the 50 per cent definition was the ‘Lass-Logistic regression with important variables’ (0.7251) followed by ‘Random forest’ (0.7191). In general, anything greater than 70 per cent can be considered good model performance; however, further refinements will be made to attempt greater accuracy levels. The implementation learnings from the analytics project to date include: • Processes to obtain datasets are relatively lengthy and multi-faceted, but the close relationship and support from staff at the Diocese made it feasible. • Missing data are problematic but not unsurmountable; the data that were ‘missing’ comprised a small portion of data that were absent due to non-entry or other quality control reasons, and a large portion were ‘gaps’ as a result from students moving in and out of the Diocese’s education system. • The top-level results of the analyses confirmed much of what is known about STEM education outcomes; however, drilling down further into analyses of particular groups and locations may provide new insights. 14 Vimu Inguruwattage and Chathura Nagoda Gamage Evidence X A second project was established to help build an evidence base for STEM education interventions; that is, better understand what interventions work and why and achieve an increased ability to assess an intervention’s effectiveness. The first step in this project was to build a prototype of an online, interactive evidence tool that would allow users (program designers, program funders, and program evaluators) to understand the current state of evidence and inform their decision-making (see screen shot in Figure 15). The prototype involved classifying a set of research articles by a number of different variables including the category of outcome (attitude, behaviour, skill, knowledge). Each ‘bubble’ summarises a research study linking an intervention (e.g., a program) or a factor (e.g., a personal attribute) with STEM-education related outcomes and provides a link to the original research. Users can see the evidence linkage and what the strength of the evidence is (weak, promising, strong). This prototype was designed using R Shiny and was developed through a research collaboration project with CSIRO’s Information Management and Technology unit. There are some technical limitations with the software that was used, and therefore different options will be explored as the project progresses. The next stage in the process involves a co-design phase led by an external consultancy to bring together key stakeholders in the STEM education sector to design a platform of evidence/set of tools. This phase will be completed by the end of 2023. Figure 15. STEM Evidence Tool prototype Conclusion The goals of Generation STEM are complex and challenging to achieve. As can be seen in Figure 3, the impact pathway for Generation STEM identifies increasing the number of participants pursuing STEM education pathways and ultimately the number of people employed in NSW‑based STEM jobs and the STEM literacy of all people. These goals aim to address the ‘leaky’ STEM ‘pipeline’, and there are no easy fixes because the influences on student behaviours and decisions are so diverse and multi-layered. Among these myriad of factors, having a STEM identity is of central importance to attracting and retaining students in STEM; that is, increasing the effectiveness of teaching STEM will have little impact if students don’t believe STEM is a place for them. Despite a relatively slow roll- out, Generation STEM is beginning to achieve a number of short-term implementation objectives and outcomes, such as student’s self-reported interest and intention to pursue STEM. The place-based focus of the initiative has seen strong relationships developed with communities and relevant industries in Western Sydney, Illawarra, Moree, and a number of other locales. STEM CPP is reaching over 2,000 students per year, and although Generation STEM Links and DiGS are much smaller in breadth, they perhaps have more potential for depth in terms of potential outcomes among individual students and schools. Figure 16 outlines a model (adapted from Farazi, Gopalakrishnan, & Perez-Luno, 2019 by the first author of this report) comprising four program model types of different depths (impact) and breadth (size of cohort). Generation STEM Links and DiGS aim to be ‘gorge’ programs, working with fewer students and investing more per student, to achieve substantial individual outcomes over time. Currently, it is a question whether STEM CPP is a lagoon or gorge program; the solution may be a lagoon program that works with individual schools to become gorge-like in their outcomes, particularly from a sustainability perspective. Growing the breadth to more schools, locations, and industry partners is casting a wide net, when perhaps a more targeted one may be more effective to address the complexities of STEM identity formation and verification among students and building a STEM school culture that can be sustained beyond the life of the program. Regardless of this over-arching question, all three programs have been successfully designed and implemented and are well-placed to achieve their goals by 2027. Figure 16. Breadth versus depth in STEM education programs Large cohort (breadth) Low intensity (depth) High intensity (depth) Small cohort (breadth) Lagoon programs Impact Budget size: $$ Examples: campaigns, events, citizen science, marketing, awards Ocean programs Impact Budget size: $$$$$ Examples: education system Gorge programs Impact Budget size: $$ Examples: work experience, longer-term support and mentoring Pond programs Impact Budget size: $ Examples: Targeted online resources, one-off teacher professional learning References Atkins, K., Dougan, B.M., Dromgold-Sermen, M.S. et al. (2020). “Looking at Myself in the Future”: How mentoring shapes scientific identity for STEM students from underrepresented groups. International Journal of STEM Education, 7(42). https://doi.org/10.1186/s40594-020-00242-3 Cohen J. (1988). Statistical Power Analysis for the Behavioral Sciences. New York, NY: Routledge Academic Edwards, D., Buckley, S., Chiavaroli, N., Rothman, S. & McMillan, J. (2023). The STEM pipeline: pathways and influences on participation and achievement of equity groups. Journal of Higher Education Policy and Management. https://doi.org/10.1080/1360080X.2023.2180169 Farazi, M.S., Gopalakrishnan, S., & Perez-Luno, A. (2019). Depth and breadth of knowledge and the governance of technology alliances. Journal of Engineering and Technology Management, 54, 28–40. https://doi.org/10.1016/j. jengtecman.2019.08.002 Gladstone, J.R. & Cimpian, A. (2021). Which role models are effective for which students? A systematic review and four recommendations for maximizing the effectiveness of role models in STEM. International Journal of STEM Education, 8(59). https://doi.org/10.1186/s40594-021-00315-x Hackling, M., Murcia, K., West, J., & Anderson, K. (2014). Optimising STEM education in WA schools. Edith Cowan University Research Online. Retrieved from https://ro.ecu. edu.au/cgi/viewcontent.cgi?article=7940&context=ecuworkspost2013 Han, J. Kelley, T. & Knowles, G. (2021). Factors influencing student STEM learning: Self-efficacy and outcome expectancy, 21st Century skills, and career awareness. Journal for STEM Education Research, 4, 117–137. https://doi.org/10.1007/s41979-021-00053-3 Jeffries, D., Curtis, D.D. & Conner, L.N. (2020). Student factors influencing STEM subject choice in Year 12: a Structural equation model using PISA/LSAY data. International Journal of Science and Mathematics Education, 18, 441–461. https://doi.org/10.1007/s10763-019-09972-5 Jin, Q. (2021). Supporting Indigenous students in science and STEM education: A systematic review. Education Sciences, 11(9), 555. https://doi.org/10.3390/educsci11090555 McKinnon. (2022). The absence of evidence of the effectiveness of Australian gender equity in STEM initiatives. The Australian Journal of Social Issues, 57(1), 202–214. https://doi.org/10.1002/ajs4.142 Millar, V., Hobbs, L., Speldewinde, C. & van Driel, J. (2022), Stakeholder perceptions of mentoring in developing girls’ STEM identities: “you do not have to be the textbook scientist with a white coat”, International Journal of Mentoring and Coaching in Education, 11(4), 398–413. https://doi.org/10.1108/IJMCE-11-2021-0100 Morris, J., Slater, E., Boston, J., Fitzgerald, M., & Lummis, G. (2021). Teachers in conversation with industry scientists: Implications for STEM education. International Journal of Innovation in Science and Mathematics Education, 29(1), 46–57. https://doi.org/10.30722/IJISME.29.01.004 Morris-Reade, R. (2022). The cost of hiring new workers doubles to more than $23,000. IT Brief, 17 March 2022. Retrieved from https://itbrief.com.au/story/the-cost-of- hiring-new-workers-doubles-to-more-than-23-000 Murphy, S., MacDonald, A., Wang, C.A., & Danaia, L. (2019). Towards an understanding of STEM engagement: A review of the literature on motivation and academic emotions. Canadian Journal of Science, Mathematics and Technology Education, 19, 304–320. https://doi.org/10.1007/s42330-019-00054-w Palmer, T.A., Burke, P.F. & Aubusson, P. (2017). Why school students choose and reject science: A study of the factors that students consider when selecting subjects. International Journal of Science Education, 39(6), 645–662. http://dx.doi.org/10.1080/09500693.2017.1299949 Shah, C., Richardson, P., & Watt, H. (2020): Teaching ‘out of field’ in STEM subjects in Australia: Evidence from PISA 2015, GLO Discussion Paper, No. 511. Global Labor Organization (GLO), Essen. Retrieved from https://www.econstor.eu/ bitstream/10419/215639/1/GLO-DP-0511.pdf Speldewinde, C. & Campbell, C. (2021). Bush kinders: enabling girls’ STEM identities in early childhood. Journal of Adventure Education and Outdoor Learning, 1(1), 1–16. https://doi.org/10.1080/14729679.2021.2011337 Stephenson, T., Fleer, M., & Fragkiadaki, G. (2021). Increasing girls’ STEM engagement in early childhood: Conditions created by the Conceptual Playworld model. Research in Science Education, 52(4), 1243–1260. https://doi.org/10.1007/ s11165-021-10003-z White, C., Marshall, J.C., & Alston, D. (2019). Empirically supporting school STEM culture—The creation and validation of the STEM Culture Assessment Tool (STEM-CAT). School Science and Mathematics, 119, 299–311. Yeh, S. (2010). The cost effectiveness of 22 approaches for raising student achievement. Journal of Education Finance, 36(1), 38–75. Appendices Appendix A. Proposed school-level data analysis for STEM CPP Evaluation question: Does STEM CPP contribute to higher uptake of STEM elective subjects by students in intervention schools? Overall analysis model – difference-in-difference regression analysis Analysis 1 Intervention group: STEM CPP schools in 2019 with Year 9 students (see list below) Comparison group: all other schools in NSW with Year 9 students Controls: school ICSEA, school Australian Statistical Geography Standard remoteness area, school size (enrolments 2021), per cent Aboriginal and/or Torres Strait Islander, per cent language background other than English, sector (government, Catholic, independent), single or mixed gender, National Assessment Program – Literacy and Numeracy (NAPLAN) Year 9 numeracy Outcome measures: (1) percentage of Year 11 students completing any STEM subject electives (2) percentage of Year 11 students completing 2 or more STEM subject electives Analysis 2 As above, except Intervention group schools are from 2021 (see list below), and the comparison group excludes a number of schools involved in the program that have dropped out or joined after 2021. OUTCOME GROUPS Percentage of Year 11 students completing any STEM subject elective Analysis 1: Year 11 STEM subject enrolments in 2019 STEM CPP schools and Year 11 STEM subject enrolments in all other schools Analysis 2: Year 11 STEM subject enrolments in 2021 STEM CPP schools and Year 11 STEM subject enrolments in all other schools (with some exclusions) Percentage of Year 11 students completing 2 or more STEM subject electives Analysis 1: Year 11 STEM subject enrolments in 2019 STEM CPP schools and Year 11 STEM subject enrolments in all other schools Analysis 2: Year 11 STEM subject enrolments in 2021 STEM CPP schools and Year 11 STEM subject enrolments in all other schools (with some exclusions) COEFFICIENT CALCULATION INTERPRETATION β0 B Baseline average β1 D-B Time trend in comparison group β2 A-B Difference between two groups pre-STEM CPP β3 (C-A)-(D-B) Difference in changes over time Regression equation: Y= β0 + β1*[Time] + β2*[Intervention] + β3*[Time*Intervention] + β4*[Covariates]+ε Appendix B. STEM CPP student survey data 2022 WHAT YEAR ARE THEY CURRENTLY IN NUMBER PROPORTION Year 9 123 56% Year 10 97 44% (n=220 for all questions including blank responses) GENDER NUMBER PROPORTION Female 113 51% Male 102 46% Other (non-binary, intersex, indeterminate, etc.) 4 2% Prefer not to say 1 0% ABORIGINAL AND/OR TORRES STRAIT ISLANDER NUMBER PROPORTION Yes 7 3% No 209 95% Prefer not to say 4 2% CULTURALLY AND LINGUISTICALLY DIVERSE NUMBER PROPORTION Yes 107 49% No 107 49% Prefer not to say 6 3% Student post-school destination INTENTION TO STUDY AT TAFE OR UNIVERSITY NUMBER PROPORTION No 2 1% Unsure / don’t know yet 33 15% TAFE, but unsure what to study 6 3% TAFE, and know what to study 3 1% University, but unsure what to study 95 43% University, and know what to study 81 37% Factors influencing student subject selection WHAT THEIR PARENTS AND FAMILY THINK NUMBER PROPORTION No impact 55 25% Slight impact 109 50% Big impact 48 22% Unsure 8 4% WHAT THEIR FRIENDS WILL BE STUDYING NUMBER PROPORTION No impact 126 57% Slight impact 67 30% Big impact 13 6% Unsure 14 6% HOW INTERESTING THEY FIND THE SUBJECT NUMBER PROPORTION No impact 100 6 3% Slight impact 36 16% Big impact 173 79% Unsure 5 2% HOW WELL THEY THINK THEY’LL DO IN THE SUBJECT NUMBER PROPORTION No impact 11 5% Slight impact 71 32% Big impact 125 57% Unsure 13 6% RELEVANCE OF THE SUBJECT TO THEIR DAILY LIFE NUMBER PROPORTION No impact 33 15% Slight impact 90 41% Big impact 81 37% Unsure 16 7% RELEVANCE OF THE SUBJECT TO WHAT THEY WANT TO DO FOR A JOB NUMBER PROPORTION No impact 8 4% Slight impact 31 14% Big impact 171 78% Unsure 10 5% PRE-REQUISITES FOR TAFE OR UNIVERSITY NUMBER PROPORTION No impact 20 9% Slight impact 57 26% Big impact 123 56% Unsure 20 9% WHICH TEACHER WILL BE TEACHING THE SUBJECT NUMBER PROPORTION No impact 62 28% Slight impact 97 44% Big impact 44 20% Unsure 17 8% Student confidence studying STEM subjects SCIENCE NUMBER PROPORTION Not very confident 23 11% Somewhat confident 71 32% Confident 123 56% N/A 2 1% MATHEMATICS NUMBER PROPORTION Not very confident 24 11% Somewhat confident 68 31% Confident 125 57% N/A 2 1% TECHNOLOGY SUBJECTS NUMBER PROPORTION Not very confident 21 10% Somewhat confident 72 33% Confident 120 55% N/A 6 3% iSTEM NUMBER PROPORTION Not very confident 27 12% Somewhat confident 80 37% Confident 81 37% N/A 31 14% STEM-RELATED VET SUBJECTS NUMBER PROPORTION Not very confident 30 14% Somewhat confident 86 39% Confident 51 23% N/A 52 24% Student intention to study STEM subjects in Years 11 and 12 GENERAL MATHEMATICS NUMBER PROPORTION No 54 25% Yes, potentially 41 19% Yes, definitely15 111 51% Not applicable/unsure 13 6% ADVANCED OR EXTENSION MATHEMATICS NUMBER PROPORTION No 48 22% Yes, potentially 42 19% Yes, definitely 121 55% Not applicable/unsure 8 4% SCIENCE SUBJECTS NUMBER PROPORTION No 42 19% Yes, potentially 36 16% Yes, definitely 135 62% Not applicable/unsure 6 3% TECHNOLOGY SUBJECTS NUMBER PROPORTION No 66 30% Yes, potentially 50 23% Yes, definitely 86 39% Not applicable/unsure 17 8% STEM-RELATED VET SUBJECTS NUMBER PROPORTION No 95 43% Yes, potentially 51 23% Yes, definitely 31 14% Not applicable/unsure 42 19% STEM-RELATED SCHOOL APPRENTICESHIPS OR TRAINEESHIPS NUMBER PROPORTION No 103 47% Yes, potentially 54 25% Yes, definitely 28 13% Not applicable/unsure 34 16% 15 This question option combines ‘yes’ and ‘yes, definitely’ from the Year 9 and Year 10 surveys Change in students’ interest and attitudes towards STEM I AM INTERESTED IN LEARNING ABOUT STEM NUMBER PROPORTION Negative change 43 20% No change 121 58% Positive change 46 22%* Effect sizes – All: 0.035; Females: 0.106; Males: 0.027 I THINK STEM IS USEFUL IN EVERYDAY LIFE NUMBER PROPORTION Negative change 24 11% No change 113 54% Positive change 73 35%* Effect sizes – All: 0.195; Females: 0.300; Males: 0.101 I FIND STEM SUBJECTS EXCITING NUMBER PROPORTION Negative change 46 22% No change 123 59% Positive change 41 20%* Effect sizes – All: 0.022; Females: 0.073; Males: 0.031 I THINK STEM SUBJECTS ARE IMPORTANT TO MY FUTURE STUDY AND CAREER NUMBER PROPORTION Negative change 42 20% No change 110 52% Positive change 58 28%* Effect sizes – All: 0.044; Females: 0.062; Males: 0.021 I FIND STEM SUBJECTS DIFFICULT NUMBER PROPORTION Negative change 47 22% No change 104 50% Positive change 59 28%* Effect sizes – All: 0.051; Females: 0.048; Males: 0.046 I THINK STEM IS IMPORTANT FOR SOCIETY NUMBER PROPORTION Negative change 31 15% No change 117 56% Positive change 62 30%* Effect sizes – All: 0.115; Females: 0.146; Males: 0.093 I AM AWARE OF THE TYPE OF STEM JOBS I CAN WORK IN NUMBER PROPORTION Negative change 28 13% No change 97 46% Positive change 85 40%* Effect sizes – All: 0.237; Females: 0.270; Males: 0.211 I EXPECT TO DO WELL IN STEM SUBJECTS NUMBER PROPORTION Negative change 32 15% No change 123 59% Positive change 55 26%* Effect sizes – All: 0.099; Females: 0.135; Males: 0.073 I WOULD LIKE TO HAVE A JOB WORKING IN STEM NUMBER PROPORTION Negative change 27 13% No change 130 62% Positive change 53 25%* Effect sizes – All: 0.073; Females: 0.101; Males: 0.051 Impact of STEM CPP activities on students’ interest in STEM COMPLETING INQUIRY PROJECT NUMBER PROPORTION Less interested 5 2% No change in interest 32 16% A bit more interested 83 40% Much more interested 70 34% Not applicable/unsure 15 7% INTERACTING WITH INDUSTRY MENTOR NUMBER PROPORTION Less interested 7 3% No change in interest 36 18% A bit more interested 58 28% Much more interested 69 34% Not applicable/unsure 35 17% WATCHING MASTERCLASS VIDEO NUMBER PROPORTION Less interested 11 5% No change in interest 52 25% A bit more interested 38 19% Much more interested 39 19% Not applicable/unsure 65 32% ATTENDING BUSINESS SITE/WORKPLACE NUMBER PROPORTION Less interested 5 2% No change in interest 31 15% A bit more interested 50 24% Much more interested 65 32% Not applicable/unsure 54 26% PARTICIPATING IN THE SHOWCASE NUMBER PROPORTION Less interested 8 4% No change in interest 30 15% A bit more interested 54 26% Much more interested 70 34% Not applicable/unsure 43 21% PARTICIPATING IN WORK EXPERIENCE NUMBER PROPORTION Less interested 2 2% No change in interest 12 14% A bit more interested 24 28% Much more interested 21 24% Not applicable/unsure 28 32% ATTENDING A TAFE TESTER EVENT NUMBER PROPORTION Less interested 0 0% No change in interest 0 0% A bit more interested 6 40% Much more interested 9 60% Not applicable/unsure 0 0% Appendix C. STEM CPP teacher survey data 2022 STEM SUBJECTS TAUGHT* NUMBER PROPORTION Sciences 34 72% Mathematics 5 11% Technology and applied studies 8 17% iSTEM 9 19% Other STEM subjects (including VET in school) 4 9% STEM-based extra-curricular activities (e.g., school clubs) 15 32% Other 4 9% (n=47) *Teachers were able to choose multiple choices for this question SCHOOL FOCUS ON PROMOTING AND SUPPORTING STUDENT ENGAGEMENT IN STEM EDUCATION AND ACTIVITIES NUMBER PROPORTION Moderate 32 70% Strong focus 10 22% Very strong focus 4 9% (n=46) IMPACT OF STEM CPP ON STUDENTS’ LIKELIHOOD OF STUDYING STEM AFTER YEAR 10 NUMBER PROPORTION No impact 2 4% Slight Impact (e.g., a few students) 6 13% Moderate impact (e.g., some students) 34 74% Significant impact (e.g., a lot of students) 1 2% Unsure 3 7% (n=46) STEM CPP activities available at the school and their impact on student interest in STEM COMPLETING THE INQUIRY-BASED LEARNING PROJECT NUMBER PROPORTION Has not changed their interest 2 5% Has made them less interested in STEM 2 5% Has made them a bit more interested in STEM 21 55% Has made them much more interested in STEM 13 34% (n=38) INTERACTING WITH INDUSTRY MENTORS NUMBER PROPORTION Has not changed their interest 6 18% Has made them less interested in STEM 0 0% Has made them a bit more interested in STEM 18 55% Has made them much more interested in STEM 9 27% (n=33) ATTENDING THE SHOWCASE EVENT NUMBER PROPORTION Has not changed their interest 2 6% Has made them less interested in STEM 1 3% Has made them a bit more interested in STEM 13 37% Has made them much more interested in STEM 19 54% (n=35) VISITING A LOCAL STEM INDUSTRY OR WORKSITE NUMBER PROPORTION Has not changed their interest 2 11% Has made them less interested in STEM 0 0% Has made them a bit more interested in STEM 8 42% Has made them much more interested in STEM 9 47% (n=19) COMPLETING STEM WORK EXPERIENCE NUMBER PROPORTION Has not changed their interest 3 19% Has made them less interested in STEM 0 0% Has made them a bit more interested in STEM 7 44% Has made them much more interested in STEM 6 38% (n=16) Impact of STEM CPP activities on students’ skills and awareness of STEM WORKING IN A TEAM WITH THEIR PEERS NUMBER PROPORTION No improvement 0 0% Some improvement 6 16% Moderate improvement 15 41% Significant improvement 16 43% (n=37) COMMUNICATING THEIR IDEAS EFFECTIVELY TO OTHERS NUMBER PROPORTION No improvement 1 3% Some improvement 6 15% Moderate improvement 13 33% Significant improvement 19 49% (n=39) USING CRITICAL THINKING TO REASON AND DRAW CONCLUSIONS NUMBER PROPORTION No improvement 1 3% Some improvement 7 18% Moderate improvement 16 42% Significant improvement 14 37% (n=38) CREATIVELY THINKING ABOUT WAYS TO SOLVE PROBLEMS NUMBER PROPORTION No improvement 1 3% Some improvement 4 10% Moderate improvement 13 33% Significant improvement 21 54% (n=39) AWARENESS OF STEM EDUCATION PATHWAYS NUMBER PROPORTION No improvement 1 3% Some improvement 12 30% Moderate improvement 14 35% Significant improvement 13 33% (n=40) AWARENESS OF POTENTIAL STEM CAREERS NUMBER PROPORTION No improvement 0 0% Some improvement 12 31% Moderate improvement 13 33% Significant improvement 14 36% (n=39) Impact of STEM CPP activities that teachers have participated in DELIVERING THE INQUIRY-BASED LEARNING PROJECT NUMBER PROPORTION Has not really made a difference 4 11% Has made some difference 17 47% Has made a significant difference 15 42% (n=36) ATTENDING THE TEACHER PROFESSIONAL LEARNING WORKSHOP NUMBER PROPORTION Has not really made a difference 4 12% Has made some difference 15 45% Has made a significant difference 14 42% (n=33) ATTENDING A TEACHER NETWORKING EVENT NUMBER PROPORTION Has not really made a difference 3 15% Has made some difference 11 55% Has made a significant difference 6 30% (n=20) ENGAGING WITH INDUSTRY MENTORS NUMBER PROPORTION Has not really made a difference 3 12% Has made some difference 14 54% Has made a significant difference 9 35% (n=26) DIFFICULTY OF EMBEDDING THE INQUIRY-BASED LEARNING PROJECT INTO NORMAL TEACHING PRACTICE NUMBER PROPORTION Easy 17 43% Very easy 5 13% Difficult 14 35% Very difficult 0 0% Unsure 4 10% (n=40) Recommending STEM CPP LIKELIHOOD OF SCHOOL CONTINUING WITH STEM CPP NUMBER PROPORTION Likely 14 35% Very likely 21 53% Possible 4 10% Unlikely 1 3% (n=40) LIKELIHOOD OF RECOMMENDING STEM CPP TO OTHER TEACHERS NUMBER PROPORTION Likely 17 43% Very likely 19 48% Possible 4 10% Unlikely 0 0% (n=40) Appendix D. STEM CPP industry mentor survey data 2022 INDUSTRY/SECTOR OF ORGANISATION NUMBER PROPORTION Manufacturing and Advanced manufacturing 9 26% Education 5 14% Government 5 14% Water treatment 2 6% Aviation 2 6% Construction 2 6% Engineering 2 6% Astronomy 1 3% Telecommunications 1 3% Automotive 1 3% Community 1 3% Waste management 1 3% Health 1 3% Information technology 1 3% Entertainment rides 1 3% (n=35) OVERALL EXPERIENCE WITH STEM CPP NUMBER PROPORTION Far short of your expectations 4 13% Short of your expectations 4 13% Met your expectations 15 47% Exceeded your expectations 6 19% Greatly exceeded your expectations 3 9% (n=32) DIFFICULTY OF BECOMING AN INDUSTRY MENTOR WITH STEM CPP NUMBER PROPORTION Very easy 11 34% Easy 15 47% Difficult 2 6% Very difficult 0 0% Unsure 4 13% (n=32) Activities undertaken as industry mentor with STEM CPP MENTORING TEACHERS AND/OR STUDENTS NUMBER PROPORTION Yes 15 56% No 12 44% (n=27) PRESENTING AT A CAREERS DAY NUMBER PROPORTION Yes 8 31% No 18 69% (n=26) CONDUCTING AN INDUSTRY SITE VISIT NUMBER PROPORTION Yes 8 31% No 18 69% (n=26) ATTENDING THE VIRTUAL SHOWCASE NUMBER PROPORTION Yes 11 41% No 16 59% (n=27) OTHER NUMBER PROPORTION Yes 11 52% No 10 48% (n=21) Confidence in supporting teachers and/or students through activities MENTORING TEACHERS AND/OR STUDENTS NUMBER PROPORTION Somewhat confident 4 27% Confident 11 73% Not applicable/unsure 0 0% (n=15) PRESENTING AT A CAREERS DAY NUMBER PROPORTION Somewhat confident 0 0% Confident 8 100% Not applicable/unsure 0 0% (n=8) CONDUCTING AN INDUSTRY SITE VISIT NUMBER PROPORTION Somewhat confident 0 0% Confident 7 100% Not applicable/unsure 0 0% (n=7) ATTENDING THE VIRTUAL SHOWCASE NUMBER PROPORTION Somewhat confident 1 9% Confident 9 82% Not applicable/unsure 1 9% (n=11) OTHER NUMBER PROPORTION Somewhat confident 2 20% Confident 8 80% Not applicable/unsure 0 0% (n=10) Industry mentor support from STEM CPP I RECEIVED ENOUGH SUPPORT TO BE AN EFFECTIVE INDUSTRY MENTOR WITH STEM CPP NUMBER PROPORTION Agree 18 58% Strongly agree 5 16% Disagree 4 13% Strongly disagree 2 6% Unsure 2 6% (n=31) Change in engagement of industry mentor’s organisation with other stakeholders SCHOOLS NUMBER PROPORTION It has remained the same 17 55% It has increased 11 35% It has increased significantly 1 3% Not applicable/unsure 2 6% (n=31) INDUSTRY NUMBER PROPORTION It has remained the same 21 68% It has increased 6 19% It has increased significantly 0 0% Not applicable/unsure 4 13% (n=31) GOVERNMENT REPRESENTATIVES NUMBER PROPORTION It has remained the same 22 71% It has increased 4 13% It has increased significantly 0 0% Not applicable/unsure 5 16% (n=31) LOCAL COUNCIL NUMBER PROPORTION It has remained the same 23 74% It has increased 4 13% It has increased significantly 0 0% Not applicable/unsure 4 13% (n=31) COMMUNITY MEMBERS NUMBER PROPORTION It has remained the same 21 70% It has increased 5 17% It has increased significantly 1 3% Not applicable/unsure 3 10% (n=30) Recommending STEM CPP LIKELIHOOD OF CONTINUING AS INDUSTRY MENTOR NUMBER PROPORTION Likely 10 32% Very likely 16 52% Possible 3 10% Unlikely 1 3% Unsure 1 3% (n=31) LIKELIHOOD OF RECOMMENDING STEM CPP TO OTHER COLLEAGUES NUMBER PROPORTION Likely 9 29% Very likely 13 42% Possible 6 19% Unlikely 3 10% Unsure 0 0% (n=31) Appendix E. Generation STEM Links student survey data 2022 GENDER NUMBER PROPORTION Female 3 25% Male 9 75% (n=12) ABORIGINAL AND/OR TORRES STRAIT ISLANDER NUMBER PROPORTION Yes 0 0% No 12 100% DISABILITY NUMBER PROPORTION Yes 1 8% No 11 92% CULTURALLY AND LINGUISTICALLY DIVERSE NUMBER PROPORTION Yes 3 25% No 9 75% Work placement allocation INDUSTRY NUMBER PROPORTION Professional and financial services 3 25% Mining 2 17% Sustainability 1 8% Advanced manufacturing 1 8% Agribusiness and food 1 8% Other 4 33% HOURS WORKED IN A WEEK NUMBER PROPORTION 10 hours or less 1 8% Between 10 and 20 hours 2 17% Between 20 and 35 hours 4 33% More than 35 hours 5 42% Students’ experience with their work placement I FOUND THE APPLICATION PROCESS EASY NUMBER PROPORTION Agree 2 15% Strongly agree 9 69% Disagree 1 8% Unsure/hard to say 1 8% I RECEIVED TIMELY INFORMATION ABOUT THE APPLICATION PROCESS NUMBER PROPORTION Agree 4 31% Strongly agree 7 54% Disagree 1 8% Unsure/hard to say 1 8% I WAS SATISFIED WITH THE INTERVIEW AND MATCHING PROCESS NUMBER PROPORTION Agree 3 23% Strongly agree 7 54% Disagree 0 0% Unsure/hard to say 3 23% Student experience throughout the work placement OVERALL EXPERIENCE WITH THE WORK PLACEMENT NUMBER PROPORTION Short of my expectations 1 8% Met my expectations 4 33% Exceeded my expectations 7 58% I HAD ADEQUATE ACCESS AND SUPPORT FROM MY SUPERVISOR NUMBER PROPORTION Agree 3 25% Strongly agree 9 75% Disagree 0 0% Unsure/hard to say 0 0% I WAS GIVEN MEANINGFUL WORK, TASKS AND ACTIVITIES NUMBER PROPORTION Agree 2 17% Strongly agree 7 58% Disagree 2 17% Unsure/hard to say 1 8% I WAS ABLE TO APPLY MY THEORETICAL KNOWLEDGE IN A WORK‑BASED SETTING NUMBER PROPORTION Agree 7 58% Strongly agree 5 42% Disagree 0 0% Unsure/hard to say 0 0% I FELT I WAS GIVEN APPROPRIATE LEVELS OF RESPONSIBILITY NUMBER PROPORTION Agree 5 42% Strongly agree 6 50% Disagree 0 0% Unsure/hard to say 1 8% I WAS GIVEN WORK OPPORTUNITIES THAT WERE RELEVANT TO MY FIELD OF STUDY NUMBER PROPORTION Agree 7 58% Strongly agree 5 42% Disagree 0 0% Unsure/hard to say 0 0% THE WORK I DID CONTRIBUTED TO THE WIDER TEAM NUMBER PROPORTION Agree 3 25% Strongly agree 7 58% Disagree 0 0% Unsure/hard to say 2 17% Student confidence in their STEM skills and knowledge AFTER their work placement I FEEL CONFIDENT ABOUT MY TERTIARY STUDIES NUMBER PROPORTION Strongly agree (5) 3 33% Agree (4) 2 12% Neither agree nor disagree (3) 3 33% Disagree (2) 1 11% Strongly disagree (1) 0 0% Unsure/hard to say (0) 0 0% I AM CONFIDENT ABOUT MY CORE WORK-READINESS SKILLS (E.G., COMMUNICATION SKILLS, WORKING IN A TEAM, PLANNING ETC.) NUMBER PROPORTION Strongly agree (5) 3 33% Agree (4) 4 44% Neither agree nor disagree (3) 2 22% Disagree (2) 0 0% Strongly disagree (1) 0 0% Unsure/hard to say (0) 0 0% I AM CONFIDENT ABOUT MY TECHNICAL STEM SKILLS NUMBER PROPORTION Strongly agree (5) 3 33% Agree (4) 3 33% Neither agree nor disagree (3) 2 22% Disagree (2) 0 0% Strongly disagree (1) 1 11% Unsure/hard to say (0) 0 0% I KNOW ABOUT MY STEM AREA OF INTEREST NUMBER PROPORTION Strongly agree (5) 3 33% Agree (4) 2 22% Neither agree nor disagree (3) 2 22% Disagree (2) 2 22% Strongly disagree (1) 0 0% Unsure/hard to say (0) 0 0% I KNOW ABOUT THE REALITY OF WORKING IN STEM NUMBER PROPORTION Strongly agree (5) 4 44% Agree (4) 2 22% Neither agree nor disagree (3) 3 33% Disagree (2) 0 0% Strongly disagree (1) 0 0% Unsure/hard to say (0) 0 0% I WANT TO PURSUE WORKING IN STEM NUMBER PROPORTION Strongly agree (5) 4 44% Agree (4) 2 22% Neither agree nor disagree (3) 3 33% Disagree (2) 0 0% Strongly disagree (1) 0 0% Unsure/hard to say (0) 0 0% I HAVE PROFESSIONAL WORKING RELATIONSHIPS NUMBER PROPORTION Strongly agree (5) 2 22% Agree (4) 3 33% Neither agree nor disagree (3) 4 44% Disagree (2) 0 0% Strongly disagree (1) 0 0% Unsure/hard to say (0) 0 0% Student confidence in their STEM skills and knowledge BEFORE their work placement I FEEL CONFIDENT ABOUT MY TERTIARY STUDIES NUMBER PROPORTION Strongly agree (5) 2 22% Agree (4) 3 33% Neither agree nor disagree (3) 1 11% Disagree (2) 2 22% Strongly disagree (1) 1 11% Unsure/hard to say (0) 0 0% I AM CONFIDENT ABOUT MY CORE WORK-READINESS SKILLS (E.G., COMMUNICATION SKILLS, WORKING IN A TEAM, PLANNING ETC.) NUMBER PROPORTION Strongly agree (5) 1 11% Agree (4) 2 22% Neither agree nor disagree (3) 2 22% Disagree (2) 3 33% Strongly disagree (1) 1 11% Unsure/hard to say (0) 0 0% I AM CONFIDENT ABOUT MY TECHNICAL STEM SKILLS NUMBER PROPORTION Strongly agree (5) 0 0% Agree (4) 2 22% Neither agree nor disagree (3) 4 44% Disagree (2) 3 33% Strongly disagree (1) 0 0% Unsure/hard to say (0) 0 0% I KNOW ABOUT MY STEM AREA OF INTEREST NUMBER PROPORTION Strongly agree (5) 1 11% Agree (4) 2 22% Neither agree nor disagree (3) 4 44% Disagree (2) 2 22% Strongly disagree (1) 0 0% Unsure/hard to say (0) 0 0% I KNOW ABOUT THE REALITY OF WORKING IN STEM NUMBER PROPORTION Strongly agree (5) 1 13% Agree (4) 0 0% Neither agree nor disagree (3) 2 25% Disagree (2) 4 50% Strongly disagree (1) 1 13% Unsure/hard to say (0) 0 0% I WANT TO PURSUE WORKING IN STEM NUMBER PROPORTION Strongly agree (5) 3 33% Agree (4) 2 22% Neither agree nor disagree (3) 3 33% Disagree (2) 1 11% Strongly disagree (1) 0 0% Unsure/hard to say (0) 0 0% I HAVE PROFESSIONAL WORKING RELATIONSHIPS NUMBER PROPORTION Strongly agree (5) 0 0% Agree (4) 1 13% Neither agree nor disagree (3) 2 25% Disagree (2) 1 13% Strongly disagree (1) 1 13% Unsure/hard to say (0) 3 38% Value of work placement interactions for students PARTICIPATING IN A WORKPLACE INDUCTION SESSION NUMBER PROPORTION Slightly valuable 3 25% Moderately valuable 5 42% Very valuable 2 17% Not applicable/this did not happen 2 17% PARTICIPATING IN ONSITE TRAINING NUMBER PROPORTION Slightly valuable 1 8% Moderately valuable 4 33% Very valuable 6 50% Not applicable/this did not happen 1 8% INFORMAL MENTORING FROM YOUR SUPERVISOR NUMBER PROPORTION Slightly valuable 0 0% Moderately valuable 1 8% Very valuable 11 92% Not applicable/this did not happen 0 0% RECEIVING CONSTRUCTIVE FEEDBACK ABOUT YOUR PERFORMANCE NUMBER PROPORTION Slightly valuable 0 % Moderately valuable 1 8% Very valuable 11 92% Not applicable/this did not happen 0 % INTERACTIONS WITH OTHER TEAM MEMBERS NUMBER PROPORTION Slightly valuable 2 17% Moderately valuable 1 8% Very valuable 9 75% Not applicable/this did not happen 0 0% SUPPORT FROM THE CSIRO PROGRAM TEAM DURING THE PLACEMENT NUMBER PROPORTION Slightly valuable 3 25% Moderately valuable 5 42% Very valuable 4 33% Not applicable/this did not happen 0 0% Recommending Generation STEM Links LIKELIHOOD OF RECOMMENDING GENERATION STEM LINKS TO OTHER TERTIARY STUDENTS STUDYING A STEM QUALIFICATION NUMBER PROPORTION Likely 1 8% Extremely likely 11 92% Unlikely 0 0% Appendix F. Generation STEM Links supervisor survey data 2022 INDUSTRY OF BUSINESS NUMBER PROPORTION Manufacturing 4 44% Information technology 3 33% Mining 1 11% Software solutions 1 11% DIFFICULTY OF BECOMING AN INDUSTRY SUPERVISOR WITH STEM LINKS NUMBER PROPORTION Easy 5 63% Very easy 3 38% SATISFACTION WITH STUDENT PERFORMANCE NUMBER PROPORTION ion Moderately satisfied 4 50% Very satisfied 1 38% Extremely satisfied 3 13% OVERALL EXPERIENCE WITH STEM LINKS STEM Links NUMBER PROPORTION Short of your expectations 2 25% Met your expectations 3 38% Exceeded your expectations 1 13% Greatly exceeded your expectations 2 25% Change in student’s skills and awareness of STEM WORKING COLLABORATIVELY IN A TEAM WITH THEIR COLLEAGUES NUMBER PROPORTION No improvement 0 0% Some improvement 2 25% Moderate improvement 2 25% Significant improvement 3 38% Unsure/hard to say 1 13% COMMUNICATING THEIR IDEAS EFFECTIVELY TO OTHERS NUMBER PROPORTION No improvement 1 13% Some improvement 1 13% Moderate improvement 3 38% Significant improvement 2 25% Unsure/hard to say 1 13% USING THEIR TECHNICAL STEM SKILLS TO COMPLETE TASKS NUMBER PROPORTION No improvement 1 13% Some improvement 0 0% Moderate improvement 3 38% Significant improvement 4 50% Unsure/hard to say 0 0% USING CRITICAL THINKING TO REASON AND DRAW CONCLUSIONS NUMBER PROPORTION No improvement 0 0% Some improvement 2 25% Moderate improvement 1 13% Significant improvement 5 63% Unsure/hard to say 0 0% CREATIVELY THINKING ABOUT WAYS TO SOLVE PROBLEMS NUMBER PROPORTION No improvement 1 13% Some improvement 2 25% Moderate improvement 1 13% Significant improvement 4 50% Unsure/hard to say 0 0% AWARENESS OF STEM EDUCATION PATHWAYS NUMBER PROPORTION No improvement 0 0% Some improvement 0 0% Moderate improvement 3 38% Significant improvement 0 0% Unsure/hard to say 5 63% AWARENESS OF POTENTIAL STEM CAREERS NUMBER PROPORTION No improvement 0 0% Some improvement 0 0% Moderate improvement 3 38% Significant improvement 0 0% Unsure/hard to say 5 63% Supervising early career students PREVIOUS EXPERIENCE WITH SUPERVISING EARLY CAREER STUDENTS NUMBER PROPORTION I sometimes supervised early career students 2 25% I regularly supervised early career students 5 63% I have never supervised early career students 1 13% IMPACT OF GENERATION STEM LINKS ON CONFIDENCE IN SUPERVISING EARLY CAREER STUDENTS NUMBER PROPORTION No impact 1 13% Slight impact 2 25% Moderate impact 2 25% Significant impact 3 38% Impact of Generation STEM Links on business engagement CHANGE IN BUSINESS ENGAGEMENT WITH THE TERTIARY SECTOR NUMBER PROPORTION It has improved somewhat 4 50% It has remained the same 1 13% It has improved significantly 2 25% Unsure/hard to say 1 13% Recommending Generation STEM Links LIKELIHOOD OF CONTINUING AS A BUSINESS WITH GENERATION STEM LINKS NUMBER PROPORTION Possible 2 25% Likely 3 38% Extremely likely 3 38% Unlikely 0 0% LIKELIHOOD OF RECOMMENDING GENERATION STEM LINKS TO OTHER BUSINESSES AND/OR COLLEAGUES NUMBER PROPORTION Possible 2 25% Likely 2 25% Extremely likely 4 50% Unlikely 0 0% Contact us 1300 363 400 +61 3 9545 2176 csiro.au/contact csiro.au For further information Education and Outreach Christopher Banks +61 7 3833 5999 christopher.banks@csiro.au csiro.au/education As Australia’s national science agency, CSIRO is solving the greatest challenges through innovative science and technology. CSIRO. Creating a better future for everyone.