Brown Bag Seminar | Can non-cognitive interventions improve academic outcomes? The first CGR brown-bag presents experimental evidence on this question

The Centre for Globalisation Research will launch on May 14th its ‘Brown-bag’ seminar series, which seeks to discuss the new research of CGR members. The workshop will be inaugurated by Prof Pedro Martins who will present his latest CGR working paper: “(How) Do Non-Cognitive Skills Programs Improve Adolescent School Achievement? Experimental Evidence”.

The question of how to improve the academic achievement of the youth lays at the core of most public initiatives on education. Fortunately, we can go beyond our own personal preferences and received knowledge to assess the success and better implementation of these initiatives. There is an important trove of academic evidence evaluating educational programs and interventions that deliver important lessons, such that early years’ investment goes a long way to reduce the differences in cognitive development or that reduced class sizes might be beneficial. With his latest working paper, Prof Martins adds to this literature, not only assessing the overall impact of non-cognitive interventions but also offering practical insights about how this type of interventions can deliver better results.

Prof Martins evaluates the program “Mediators for school success” – shorthanded EPIS – which aims to reduce early school leaving and promote student achievement and is delivered by the organisation of the same name, “Entrepreneurs for Social Inclusion”.  EPIS screens and selects 7th and 8th year grade pupils (around 13-15 years old) that are most at risk of failing or dropping out. Students selected to participate in the program are then monitored by a mediator that assigns a tailored set of individual and small-group intervention to help the students improve their motivation, self-control, problem-solving skills, or study competences, among others. The role of the mediator is an essential part of the program, that seeks to promote a stable relationship between them and the students.  Mediators implement the program’s interventions, meet the students in a relatively frequent basis, and keep in contact with relatives and teachers of the participating students.

The research exploits the EPIS longitudinal records not only to evaluate the program, but to assess who is most benefited by it and how EPIS’ promotes success. In particular, the study draws on data from a randomised controlled trial at the school level that followed 53 schools during the 2014/15 and 2015/16 academic years, observing a total of 2,959 students, 648 of which were assigned to a control group.

The study finds that program assignment lead to an increase of 4.8 percentage points over the two years and that program delivery has an effect of 9.9 percentage points. Given the baseline probability of 45%, the effects above can be translated into relative increases in the probability of progression of 11% and 22%, respectively. Figure 1, below, summarizes the causal impact of EPIS in the fully specified model in the probabilities of progression (i.e., non-retention) across the two years and considering other relevant outcomes too.

Figure 1: Impact of EPIS in progression and related outcomes

As we can observe, and Prof Martins points in his research, participating in the EPIS program not only increases significantly the chances of student academic progression but also has a positive impact on Portuguese, Mathematics, and English grades. However, the fact that EPIS has a positive impact does not mean that all students are impacted equally, as we can observe in Figure 2 EPIS tends to have greater impact on girls and older students.

Figure 2: Impact of EPIS across different subgroups of students

In this respect, the study finds that the mediator is of great importance to EPIS’ success. As the study is also interested in understanding in greater detail the reasons for the program success, Prof Martins explores the role of the different characteristics of the mediators on the students’ success. While Figure 1 outlined what was the impact of EPIS and Figure 2 to whom this impact benefited most, Figure 3 outlines how this benefit is achieved.

Figure 3: EPIS impact interaction with mediator characteristics

As we can observe from Figure 3, most of the individual traits of the mediator bear an insignificant impact in terms of EPIS’ overall results. Nonetheless, there is one important exception: Same gender mediators increase considerably the chances of their students’ progression. As Prof Martins explains, this finding is consistent with a small, emerging literature that discusses the role of the teacher’s gender in pupil’s academic achievement.

In his latest CGR working paper, Prof Martins shows the potential of careful empirical analysis to evaluate how and why public or private interventions succeed or fail. In this case, the study also furthers our understanding of how non-cognitive skills impact upon students’ academic achievement. As Prof Martins concludes, his study not only supports the importance of initiatives that enhance soft skills but also the importance of proper design, implementation, and monitoring of these programs.

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s