Implementing Efficient Memory-Based Collaborative Filtering Recommendation Systems: Methods for Improving Scalability in Training Phase

Abstract

Memory-based collaborative filtering recommendation systems introduce an item to a target user if that item has been liked by users similar to the target user. Therefore, during the training phase, the system needs to compute the similarity of each pair of users. However, the cost for this task becomes infeasible as the number of users increases. To enhance the system’s scalability, it is necessary to cluster users and only compute the similarity between users within each cluster. For this user clustering process, we aim to propose two methods for initializing user clustering instead of random initialization as in previous studies in the field of recommendation systems. Each method is used in two different contexts: with only ratings and with both ratings and item genres. Furthermore, we also present a parallel processing design on Hadoop for computing the Jaccard similarity measure to further reduce the training time of the system.