Opening remarks by Educational Quality & Assessment Programme (EQAP) Director Dr Michelle Belisle
Good morning and a warm welcome to all of you as we begin our very first Pacific Islands Literacy and Numeracy Assessment, or PILNA, data workshop. While we at EQAP work with data on a regular basis, and have had the opportunity to dig into the PILNA data in the analysis of the results and in crafting the regional and national reports, this workshop marks several firsts and milestones in the PILNA program.
Although we have just completed the third PILNA cycle with the PILNA 2018 reporting and dissemination activities, this is the first time we have a full set of contextual data to go along with the cognitive results. PILNA 2018 is also the first time that we have been able to link all the data elements – literacy with numeracy, and cognitive data with teacher, student and school leader survey responses. The magnitude and richness of the dataset that has been compiled is really quite overwhelming when you stop to think about it: 40,000 students, almost 1000 schools, and over 2000 teachers. Looking at the data points available across all of these groups, we have somewhere close to 7 million data points from the instruments and several thousand more that have been calculated as a result of the analysis of the results.
This week also marks the first time we have brought people together to explore the PILNA data, beyond what has been shared through the PILNA reports. As you can likely imagine, the first question we asked ourselves was “Where do we start?” We decided to start with the questions that were commonly being asked whenever we talked about PILNA with various stakeholder groups.
The first PILNA administration in 2012 sought to answer some very key questions, posed by the Forum Education Ministers, namely, how are Pacific Island students doing when it comes to literacy and numeracy? Reading, writing and numeracy assessment tools were developed and 14 countries participated in the PILNA assessment that year. The results were analysed using classical test theory and reported back to countries on the basis of percentage of students below, at and above expected levels. When it was designed prior to the 2012 administration, PILNA was envisioned as being a one-time assessment to respond to the ministers’ questions.
The results of the first PILNA were presented back to the Forum Education Ministers at their 2014 meeting, two years after the assessment had taken place. The message was pretty stark – students in the Pacific were not doing well in literacy with less than half of students reaching minimum expected levels. Numeracy was better, but still not very good overall. This reporting of the results triggered a number of responses that have brought us to where we are today.
Data has a funny way of generating more questions than it answers.
Data has a funny way of generating more questions than it answers. While providing some general answers about how well boys and girls were doing, as well as students in urban and rural and government and non-government schools, the PILNA 2012 results prompted ministers to ask whether the investments being made across the region in literacy and numeracy were having any effect – and that prompted a second administration of PILNA in 2015 to try to see what changes were being noted as a result of all the investments. The PILNA 2012 data also brought about other questions from the ministers – were the national samples really representative of the countries involved? What about the different languages spoken by students? How valid and reliable were the results? Do the instruments represent the expectations we have of students after 4 and 6 years of formal education? These are but a few of the questions that were swirling around following PILNA 2012.
To carry out a second cycle of PILNA in 2015 that would try to address the key questions of what changes have occurred since the first PILNA, a new design was adopted. A steering group made up of the heads of education systems from each of the PILNA countries was brought together to provide input and advice on critical decisions about what a second PILNA would look like and that group met again to provide input into how the results would be reported. Technical expertise was sought from the Australian Council for Educational Research (ACER) to strengthen aspects of the assessment including sampling, IRT analysis, and reporting scales as well as logistical support in managing the scope of the project. Funding support from the New Zealand Ministry of Foreign Affairs and Trade made it possible to ensure that scoring sessions were held in each country and that when results were released in July of 2016, EQAP officers were able to visit each country to share the results with ministry officials and other stakeholder groups.
Still, even though each ministry or department of education received their 2015 data files along with their reports, little attention was given to supporting countries in exploring the raw data from PILNA 2015. To be fair, the data available was limited in what it might provide beyond what was in the reports themselves. In 2015, coding was piloted as a process to capture data about student work but it was not fully implemented until 2018. Similarly, contextual questionnaires were trialled in 2015 in a few schools in each country but were not fully implemented until 2018. As was the case following PILNA 2012, the PILNA 2015 results provided some answers about how Pacific Island students were faring in terms of literacy and numeracy and, when compared to PILNA 2012 results, provided some hint as to whether there were gains being noted.
Again, the results of PILNA 2015, particularly with the pilot coding and questionnaires providing hints of what might be possible, education leaders had more questions they wanted answered. When data showed that girls outperformed boys, questions were raised about what that meant, why it was happening and what could be done to improve achievement for boys. When the data showed urban schools outperforming non-urban schools, the same kinds of questions came up, along with questions about teacher qualifications and experience, school resources and language of instruction.
In addition to developing a second round of PILNA, the support provided in 2015 and 2016 allowed for the development of a long-term regional assessment program that could continue to provide information about literacy and numeracy as well as the contexts in which students were learning to the countries and to the region overall. As a result, we have the 2018 PILNA results and are soon to embark on a fourth cycle of PILNA to be administered in 2021.
But what does all of that have to do with roughly 7 million pieces of data and what you are all here for this week? In the materials provided ahead of this workshop, you will have seen the three questionnaires that were part of the PILNA 2018 administration. Additionally, data was collected about student responses, beyond correct or incorrect, for every item on the literacy and numeracy assessments. While the reports that have been compiled for each country address the main points and key themes coming out of the analysis of the data, the data sets themselves provide an opportunity for each country to look more deeply into their own results to gain a deeper understanding of those results. With so many possibilities though, we are back to the question of where do we start?
That is where you come into the process. Each of you has come here this week with specific priorities and questions that your ministry or department of education wants to explore further. To do that, we will be working together and making use of a number of tools and processes to help you address the questions that you have brought with you, but possibly more importantly, to provide a way for you to continue to explore your data when you return home at the end of the week.
Over the course of this week you will be invited to explore the data using Excel based tools developed by our EQAP officers and ACER specialists. We will talk as a whole group about the types of data available, and the limitations of the data as well. You will have an opportunity to formulate your own questions and work with participants as well as the data specialists to refine the questions you brought with you. As the week progresses we will take you through some ways to present what you have extracted from the datasets in ways that will support decision-makers in your ministries with good quality information that is well supported by evidence.
While the tools we will be working with are Excel-based, we know that there are some in the room who would like to get into more technical analysis and by keeping the number of participants small and having a wide range of technical staff here with us this week, we hope to be able to individualize the program for the week for each of you.
As I noted when I started speaking, this workshop is a first for us, and we are really excited about the possibilities. At the same time, we know that first times can be full of surprises and we will be learning alongside you as we use these data tools for the first time. We are looking forward to a week filled with opportunities to learn from one another and to work together in developing ways to make use of the PILNA 2018 data now and into the future.