In this installment of the AI in Education: here and now series, we speak to Anthony Manny, Business Analytics Lead at The Universities and Admissions Centre (UAC), Australia, to find out how they are using AI to support and enhance the university admissions process.
Could you tell us a bit about UAC?
The Universities Admissions Centre (UAC) processes applications for admission to most undergraduate courses at participating institutions (mainly located in New South Wales (NSW) and the Australian Capital Territory (ACT)).
We also:
- process applications for admission to many postgraduate courses
- calculate and provide the Australian Tertiary Admission Rank (ATAR) to New South Wales students who have achieved the Higher School Certificate (HSC)
- process applications for Educational Access Schemes (EAS)
- process applications for some Equity Scholarships (ES)
- process applications for some Schools Recommendation Schemes (SRS).
UAC is committed to providing excellent service to its applicants as well as to tertiary institutions, schools, parents, government education departments, statutory authorities and other bodies.
In collaboration with our participating institutions, we continuously explore ways of improving our service and the way in which we deliver it, and promote equity of access to tertiary education.
UAC is a private company that operates within Australia and by virtue of its governing rules and its service to the community is a not-for-profit entity under section 50-10 of the Income Tax Assessment Act 1997 (Cth).
Could you provide an overview of how UAC has been using machine learning to help students make decisions about what courses they take?
We have developed two complementary tools for students to help them make future study decisions for the transition from school to university (Subject Compass and Course Compass – accessible via the UAC website). From a student perspective, year 10 students can use Subject Compass to identify the best HSC courses to take (equivalent to the qualifications students in UK sixth forms and colleges might work towards). Course Compass does the converse of this. Based on what HSC subjects a student has studied, it identifies the courses that students with similar study patterns to them have received offers to.
Course Compass provides university course suggestions based on ATAR and Year 12 performance as reported by our applicants. Course Compass is a machine learning model based on the most recent five years of Year 12 student data and is updated each year with the latest data.
Subject Compass recommends Higher School Certificate courses to New South Wales Year 10 students based on their interests in future university courses, their career aspirations, skills and personal interests. This system is rule-based, as opposed to machine-learning based, but we use natural language processing techniques to detect relationships between occupations and university courses.
What problems can this use of machine learning help to solve?
Choosing the right university degree is an important but difficult decision for many Year 12 students. A wrong choice may result in the student either not receiving the desired offer or enrolling in a course that is not suitable in relation to the student’s strengths and interests. Additionally, it will be a waste of both the university’s and the student’s time and financial resources if the student drops their course. Course Compass is developed to help students in choosing the most suitable university courses based on historical offer data. This tool tries to find the offers received for the majority of applicants who are similar to the user.
Why are machine learning-based systems suitable for addressing these problems?
Course Compass provides students course suggestions based on their ATAR and Year 12 performance. The recommendations are generated based on many dimensions’ input which vary depending on whether the user knows their results or not. Prior to results being known the suggestions are based on HSC subjects studied and an ATAR estimate and include actual ATAR and subject results once known. It would be difficult to design handcrafted rules, based on human knowledge, to analyse all inputs at the same time.
Moreover, the system is trained based on a huge amount of data, and the data patterns change through time (There are around 55,000 ATAR eligible HSC students per year with around 19,000 subject combinations – around 40,000 apply to UAC). Machine learning models can recalibrate with changing data patterns, whereas a rule-based system would have to be updated manually if the underlying data pattern changes.
What impact is the course compass having, and how are you measuring its impacts?
The Course Compass model was evaluated during its initial development. The evaluation was based on the 2015 Year 12 student data and their first-year university GPA data, which consisted of 11,242 student records. The experiments were conducted in three separate settings: comparing top 1, top 2, or top 3 recommendations from the Course Compass model and the actual courses the students took. The outcome measures of interest were average GPA and attrition rate. The students who studied in the same fields of study as recommended by Course Compass were found to outperform students who studied fields of study not recommended by Course Compass. The advantage of following Course Compass recommendations was on average approximately 3% to 6% GPA and attrition rate across the three different experiment settings.
Are there any risks to this use of machine learning? If so, how have these been addressed?
Machine learning models are trained based on historical data. The dataset should be big enough to represent historical patterns, otherwise the model’s result will not be accurate. Also, as the model is trained on data that is available, any model may have biases based on whether a representative cohort is captured (or not captured) in that data.
The users should understand machine learning models are developed based on statistical approaches. It provides recommendations for courses where the user will likely receive an offer. However, there is always a possibility that the user would not receive an offer, or they might receive offers for other courses.
The machine learning model’s suggestions are just like the suggestions from users’ parents, teachers, friends. The difference is that the suggestions from machine learning models are generated based on historical data, while human suggestions are generated from human knowledge. Any type of suggestions may contain bias. The users should still make their own decisions based on the suggestions from all different channels. The language used in the model is broad (recommending Fields of study rather than a specific degree) and is also careful to not be too definite in saying you should do this, rather it gives multiple suggestions and suggests that users should “use these fields of study as a starting point for narrowing your uni options” and that it will provide “the top fields of study offered to past applicants with your academic profile”.
What advice would you have to education institutions who are thinking about using AI in their settings?
Data is the foundation for any AI product, so good quality data should be accumulated first.
The fundamental part of an AI project is to convert a business case into a machine learning problem, and then apply appropriate models to solve the problem. It is important to understand the limitations of the answer from a machine learning model using data, context of the problem, and algorithm applied. Machine learning searches for patterns in the historical data and this may not always provide a useful prediction. It may perpetuate a pattern that is not optimal and might perpetuate biases that are undesirable in the current environment.
There should be some customisation on the training settings based on the actual problem and dataset. Applying a generic machine learning model to a given task would not produce the optimal outcome.
The data for educational institutions are primarily collected from students and applicants. These institutions need to be cognisant of data privacy issues and other data usage limitations before undertaking AI projects.
Find out more by visiting our Artificial Intelligence page to view publications and resources, join us for events and discover what AI has to offer through our range of interactive online demos.
For regular updates from the team sign up to our mailing list.
Get in touch with the team directly at AI@jisc.ac.uk
