Figure 4: Recall comparison on in-matrix and out-of-matrix prediction tasks by fixing the number of recommended articles at M = 100. Error bars are too small to show. This shows how the precision parameter λv affects the performance of CTR. The expected recall of random recommendation is about 3%. CF can not do out-of-matrix prediction.
The gap between CF and LDA is interesting—other users provide a better assessment of preferences than content alone. Out-of-matrix prediction is a harder problem, as shown by the relatively lower recall. In this task, CTR performs slightly better than LDA. Matrix factorization cannot perform out-of-matrix prediction. (Note also that LDA performs almost the same on both in-matrix and out-of-matrix predictions. This is expected because, in both settings, it makes its recommendations almost entirely based on content.) Overall, CTR is the best model. In Figure 4 we study the effect of the precision parameter λv. When λv is small in CTR, the per-item latent vector vj can diverge significantly from the topic proportions θj . Here, CTR behaves more like matrix factorization where no content is considered. When λv increases, CTR is penalized for vj diverging from the topic proportions; this brings the content into the recommendations. When λv is too large, vj is nearly the same as θj and, consequently, CTR behaves more like LDA. We next study the relationship, across models, between recommendation performance and properties of the users and articles. For this study we set the number of recommended articles M = 100 and the precision λv = 100. Figure 5 shows how the performance varies as a function of the number of articles in a user’s library; Figure 6 shows how the performance varies as a function of the number of users that like an article. As we see from Figure 5, for both in-matrix and out-of-matrix prediction, users with more articles tend to have less variance in their predictions. Users with few articles tend to have a diversity in the predictions, whose recall values vary around the extreme values of 0 and 1. In addition, we see that recall for users with more articles have a decreasing trend. This is reasonable because when a user has more articles then there will be more infrequent ones. As we see next, these articles are harder to predict. From Figure 6, on in-matrix prediction for CF, CTR and LDA articles with high frequencies tend to have high recalls for in-matrix prediction and their predictions have less variance. This is because these articles have more collaborative information than infrequent ones, and, furthermore, CF and CTR make use of this information. Topic −0.02 0.00 0.02 0.04 1 2 3 4 5 6 7 8 9 10 var theta theta correction topic 1: estimate, estimates, likelihood, maximum, estimated, missing, distances topic 10: parameters, Bayesian, inference, optimal, procedure, prior, assumptions