by Valentina Salvatelli
Studying science is very hard. Having a great tutor to help and motivate you, makes a real difference in the development of strong scientific and problem-solving skills. Unfortunately, the majority of students does not have access to any specialised and tailored STEM education. This often translates into a missed opportunity to access university.
Progression of students to university by UK region. Red regions only 0-20% students go on to study at university, orange = 20-40, yellow = 40-60%, light blue 60-80% and dark blue 80-100%.
The online educational platform IsaacPhysics.org was created two years ago with the aim to address this issue. IsaacPhysics.org is a highly interactive system where the students can find thousands of questions on different topics related to Maths and Physics at all levels of difficulty. Some features include live feedback on their answers and progressive hints to help them complete their tasks. So far so good, if it wasn't for the pre-composed set of challenges that drastically limits the level of customisation.
In my ASI project I have worked to transform the platform into a digital private teacher, a personalised system that can adapt to the needs of each student. The first step was to analyse the big amount of unstructured data (click-streaming records) collected from the platform (about 6M of attempted questions from about 25000 active students) and the database containing the content of the platform (about 6000 questions).
Personalisation can be achieved in different ways. I decided to start with setting the right level of difficulty for each questions to improve users' engagement rate; the more the students come back to the platform, the more likely they are to benefit from this learning opportunity. Plus, the number of students using the platform is proportional to the degree of personalisation.
What I found is that the engagement of students is connected to the ratio between questions type (easy, moderate, difficult) that the students face on the platform. This is not at all surprising: if questions are too easy the students lose their interest and leave the platform. On the other hand, if questions are generally too difficult the student will give up shortly, presuming that the challenge is far too difficult for their skills.
The next step was to find a small group of very active users and identify the right level of difficulty. As it turned out, this relates to a precise range: one difficult question every three moderate questions (see image below). The aim was then to predict the level of difficulty for every user in advance so that we would be able to adjust the balance between difficult, moderate and easy questions and increase students' engagement rate.
How does a difficult question look like? If a teacher could sit close to the student, it would be easy to gauge. So, I sat down with experienced teachers and translated their real-world experience into an algorithm that automatically labels questions based on how many attempts the student make and how many hints they view. It was then easy to predict the difficulty level merging the information on the individual student and the collective behaviour of other students.
Armed with all the labels and the features that I had extracted from the platform data, I built a number of predictive models with different machine learning algorithms. XGBoost, that is the most used algorithm for winning Kaggle competitions and is just a very sophisticated combination of decision trees, has been also the winner of my personal competition: being able to reach an AUC (area under the ROC curve) of 0.83 and an accuracy of 76%.
IsaacPhysics are now able to integrate my model on their platform and run A/B testing. Soon, students will get access to personalised game boards that are optimised for their engagement level and their learning goals. This represents a concrete step for providing a personalised STEM education and laying the foundations for the future of education.
Valentina Salvatelli took part in the ASI Fellowship January 2017. Prior to the Fellowship she completed a PhD in AstroPhysics at the University of Rome La Sapienza.