June 2021 Inquiry for Indigenous Science Students (I2S2) Case study evaluation report Indigenous STEM Education Project Contents Acknowledgements.................................................................................................................................3Acknowledgement of Country...........................................................................................................................................3Other acknowledgements..................................................................................................................................................3Acronyms......................................................................................................................................................6Executive summary...................................................................................................................................7Background and methodology..........................................................................................................................................7Findings...............................................................................................................................................................................8Challenges and success factors..........................................................................................................................................9Discussion and recommendations....................................................................................................................................9Introduction................................................................................................................................................10History of the program.....................................................................................................................................................10Program design.................................................................................................................................................................10Evaluation.....................................................................................................................................................12Scope and purpose...........................................................................................................................................................12Research reflection and position....................................................................................................................................12Methodology.....................................................................................................................................................................13Limitations.........................................................................................................................................................................14Findings........................................................................................................................................................16Increased student engagement and academic results (Outcome 1)............................................................................17Increased student aspiration, sense of value/worth, and school belonging (Outcome 2).........................................33Increased teacher capacity in both inquiry and Aboriginal and Torres Strait Islander context (Outcome 3)...........37Increased community and parental/carer engagement and schools have increased cultural competency delivering Indigenous contextualised inquiries in partnership with families and community (Outcome 4)............42Increased number of Aboriginal and/or Torres Strait Islander (and non-Indigenous) students pursuing STEM pathways, including in Years 10 to 12, university, and alternatives (Outcome 5).............................................47Identification of ‘best practice’ in high expectations science inquiry education and teacher professional learning, and adoption of this ‘best practice’ by states and territories (Outcome 6).................................................51Schools supporting other STEM programs (e.g., ASSETS, CREST Awards, PRIME Futures) (Outcome 7)...................53School culture of high expectations – also benefitting other subject areas (Outcome 8).........................................54Discussion...................................................................................................................................................59Student engagement and academic results...................................................................................................................59Student sense of value and school belonging................................................................................................................59Teacher capacity...............................................................................................................................................................60Engagement and partnerships........................................................................................................................................60Student STEM pathways...................................................................................................................................................61Best practice and adoption by jurisdictions...................................................................................................................61Schools supporting other STEM programs.....................................................................................................................61High expectations.............................................................................................................................................................61 Recommendations..................................................................................................................................63References..................................................................................................................................................65Appendix 1: Impact pathway ............................................................................................................70Appendix 2: I2S2 inquiry topics.........................................................................................................72Appendix 3: Inquiry-based and Indigenous STEM Programs............................................73Appendix 4: Interview and focus group questions.................................................................75Appendix 5: Example inquiry rubric (Year 6: Let’s Stick It Together)............................76Appendix 6: Jurisdictional data analysis......................................................................................77 1 The wording of the outcomes in this report has been changed slightly as compared to the Impact Pathways in order to increase clarity. 2 PRIME Futures concluded in late 2019. 3 Delivered by the University of Melbourne 4 ‘Aboriginal or Torres Strait Islander’ refers to the two distinct First Nations groups of Australia, including their cultures and knowledges. ‘Aboriginal and/or Torres Strait Islander’ refers to First Nations peoples, and acknowledges that some people identify as both Aboriginal and Torres Strait Islander. 5 At the time the case study was conducted, there were eight inquiries available. In early 2020, an additional two inquiries (‘In the Mix’ and ‘Perfect Pitch’) were added, bringing the total to 10 inquiries. 6 A flyer with an overview of this module states that “The cultural considerations module provided participants with foundational knowledge and understanding of key cultural considerations to help [teachers] recognise the diversity of Aboriginal and Torres Strait Islander cultures and identities. Participants explore key topics and relate them to their own personal perspective and worldview which is imperative when beginning to construct understanding of Aboriginal and Torres Strait Islander ecological knowledge required to effectively deliver the I2S2 inquires.” 7 Taken from an advertised CSIRO Position Description for an ‘I2S2 Coordinator’ in 2018. The titles of these positions were changed to ‘Education Advisers’ when the program shifted to an online model of delivery. 8 A collaboration hub where teachers could engage and invite professional dialogue with other schools and teachers across Australia was originally planned but was not pursued. 9 The original Impact Pathway included attendance as part of this outcome. As discussed in the Third Evaluation Report (Cherry, Banks, Mudhan, & McNeilly, 2019), attendance is not a realistic outcome of I2S2 and was removed from the Impact Pathway. This decision was supported by Coordinators and teachers, and substantial evidence in the research literature. One Coordinator explained that having attendance as a key performance indicator is not an effective measure because: “There are too many outside variables that are impacting on student attendance...A student is sick for two weeks and their attendance drops by 20 per cent. I just think there are so many factors outside of our program that either way I don't think it would be fair to say that our program is increasing attendance or that student attendance is decreasing because of our program” and “The only way I can see attendance being useful is if teachers were looking at specific students who they know don't attend because they're not engaging and looked almost on a case study basis on whether or not the student's attendance is improvement”. 10 Originally these were two separate outcomes but the analyses revealed they were closely linked and were therefore combined into a single outcome area. 11 No minimum percentage was identified but the evaluation team’s judgement was used to select schools with sufficient numbers of Aboriginal and/or Torres Strait Islander students so their voices were represented and heard. 12 Superseded by the AIATSIS Code of Ethics for Aboriginal and Torres Strait Islander Research in October 2020. 13 This approach of providing only year level data was standard practice for this jurisdiction and not specific to the I2S2 data request. 14Teachers from one jurisdiction did not take part in this element of monitoring the program because of the decision by CSIRO not to pursue research approval in that jurisdiction. The decision was made because the specific requirements of that jurisdiction were deemed to be too onerous given the time and resource constraints of the evaluation and program teams. 15 Although some evaluation rubrics contain five or more categories or levels, it was decided that these scales can sometimes misrepresent the level of accuracy possible in an outcome evaluation. The three levels selected for this evaluation were deemed a more realistic categorisation for the achievement of outcomes, and reflect the innovative nature of the Indigenous STEM Education Project. In addition, the ‘emerging’ level has been employed to align with the strengths-based approach of the evaluation. 16 ‘Student aspiration’, particularly to follow STEM education and career pathways, is covered in Outcome 5 rather than in this section to avoid repetition. 17 I2S2 inquiries adhere to the Hackling inquiry cycle (Hackling, 2005). 18 Due to the similarity between two outcomes (a. Increased community and parental engagement and b. Schools have increased cultural competency delivering Indigenous contextualised inquiries in partnership with families and community), they have been combined for the purposes of analyses. 19 ‘Aspiration’ to follow a STEM education or career pathway is covered in this outcome rather than in Outcome 2. 20 Specific names have been removed to ensure schools are not identifiable. 21 Not directly attributable to I2S2. 22 Some teachers felt that the delivery of inquiry resources from CSIRO was at times inconsistent (prior to the end of 2018), as at times there were delays and issues in sourcing resources for the inquiries. This meant that units were sometimes delivered inconsistently across the teaching year or not at all for certain cohorts. After the end of 2018, teachers sourced their own resources. 23 No case study schools had a formal incentive or recognition program in place; however, some schools had informal acknowledgements. Impact Pathway Statement Inquiry for Indigenous Science Students CSIRO, BHPB • $5.90m • 30 year relationship between CSIRO and BHPB in science education • CSIRO experience in science inquiry education – especially CREST • Indigenous leadership INPUTS What we invest ACTIVITIES What we do • Recruitment of schools to participate in program and building community relationships Assumptions • Indigenous leadership is critical to program success including development of Indigenous inquiry. Indigenous assistant teachers know the cultural context so are important partners. They may need capacity building in western science context. • Using multi-modal ways to demonstrate and improve success in science will encourage students to improve their literacy and encourage schools and teachers to raise their expectations. • To improve pathways to university we need to work through middle school and into year 10. • Both VET and university pathways should be supported, tailored to individual student skills and aspirations. Inquiry pedagogy is consistent with Indigenous pedagogy. External factors • The Aboriginal and Torres Strait Islander cross curriculum priority is an important support for the program’s focus on Indigenous context. • The level of non-Indigenous parental support for Indigenous content in schools is untested. There is a lack of curriculum demonstrating Indigenous scientific inquiry skills. • Most teachers of Indigenous students are non- Indigenous so role modelling of high expectation STEM programs by non-Indigenous teachers is important. • Family support for education achievement varies. • There are systemic pressures that channel Indigenous students to VET. • Policy imperatives with literacy and numeracy can result in science pedagogy having lower priority. Participation I2S2 Team, Technical Experts • Recruit and train team to develop and implement: • Hands on scientific inquiries with indigenous context linked to Australian curriculum • Teachers support resources – e.g. multimodal delivery and assessment, wiki space, scaffolding • Teacher Professional Development (TPD) package – science inquiry skills and Indigenous context • Development of program monitoring processes OUTPUTS Our deliverables Development of agreements with schools and partnerships with community orgs Year 5-9 Indigenous contextualised inquiry and support resources developed OUTCOMES The uptake, adoption or consumption of or work TPD package delivered to participating teachers Delivery of inquiries Ongoing TPD IMPACTS Benefits to economy, environment and society Increased student engagement, attendance and results Increased student aspiration, sense of value and school belonging Increased teacher capacity in both inquiry and indigenous context Increased community, parental engagement Increased number of Indigenous (and non- Indigenous) students pursuing STEM pathways – Yr 10-12, university and alternatives Schools are culturally competent in delivering Indigenous contextualised inquiries in partnership with families and communities Best practice in high expectation science inquiry education programs and TPD identified; jurisdictions adopt Schools supporting other STEM programs (e.g. ASSETS, Crest and Awards, PRIME Futures) Indigenous knowledge and culture valued: complementarily to western science demonstrated Greater under-standing and care of environment Social cohesion/ reconciliation More, higher quality and greater workforce diversity of STEM professionals Increased innovation and workplace productivity Monitoring data I2S2 coordinators, Teachers, Students, Elders, Family, Community Universities, Community Jurisdictions, Schools I2S2 team, Dept Officials, Principals, Teachers School culture of high expectation – also benefitting other subject areas