National research evaluation initiatives and incentive schemes are often faced with a difficult dilemma: selecting between simplistic quantitative indicators and time-consuming peer/expert review, sometimes supported by bibliometrics. In an effort to provide a third alternative, researchers from the UK sought to determine whether machine learning could be an accurate predictor of article quality using multiple bibliometric and metadata inputs.
The team of researchers, including Mike Thelwall, Kayvan Kousha, Paul Wilson, Meiko Makita, Mahshid Abdoli, Emma Stuart, Jonathan Levitt, Petr Knoth, and Matteo Cancellieri conducted a study to predict article quality scores of 84,966 submissions to the UK Research Excellence Framework 2021. Each submission in their investigation matched a Scopus record from the period between 2014-18 and included a substantial abstract.
The study tested 32 machine learning algorithms, with the Random Forest Classifier and Extreme Gradient Boosting Classifier algorithms ultimately providing the highest accuracy scores. The results were promising for medical and physical sciences Units of Assessment (UoAs), reaching an accuracy rating up to 42% above the baseline of 72%. However, accuracies above the baseline for the social science, mathematics, engineering, arts, and humanities UoAs were much lower or close to zero. Additionally, while increasing accuracy with an active learning strategy and by selecting articles with higher prediction probabilities improved accuracy, there was a substantial reduction in the total number of scores predicted.
The study, published in the journal Quantitative Science Studies, provides an invaluable insight into the potential of machine learning in predicting article quality scores. It is hoped that further research into this application of machine learning will help to optimize existing bibliometric and metadata inputs to reach a higher level of accuracy for predicting article quality scores.
