REVIEW 4 major objections 4 minor 2 cited by
Exploring the Effect of Context-Awareness and Popularity Calibration on Popularity Bias in POI Recommendations
T0 review · 4 major / 4 minor · reviewed 2026-08-06 · deepseek-v4-flash
Pith's one-line read This paper establishes that combining the context-aware model LORE with calibrated popularity re-ranking produces POI recommendation lists that match each user's head-mid-tail popularity profile while retaining accuracy.
desk verdict A competent, genuinely novel empirical study: first to bring calibrated popularity to POI recommenders; the headline tail-coverage claim for LORE+CP is real but conditional on the top-150 candidate pool. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The carrying mechanism is Calibrated Popularity (CP), a re-ranking procedure that starts from a base list of top-150 items and greedily builds a top-10 list by maximizing $(1-\lambda)\cdot\mathrm{Rel}(L_u) - \lambda\cdot\mathrm{JSD}(P,Q(L_u))$, where $\mathrm{Rel}$ is the base model's relevance score, $\mathrm{JSD}$ is Jensen-Shannon divergence between the user's historical popularity distribution $P$ and the list's distribution $Q$, and $\lambda$ is tuned per user group. Two variants are used: $\mathrm{CP}_H$ maximizes the harmonic mean of accuracy and calibration, and $\mathrm{CP}_{\mathcal{I}}$ sets $\lambda=1$ to minimize divergence directly. The base models are LORE, which combines sequential, social, and geographical influences and supplies tail items to the candidate pool, and USG, which blends user- and friend-based collaborative filtering with geographical influence, alongside the BPR baseline. The mechanism explains the paper's main result because CP can only reorder what the base model retrieves, so LORE's tail-rich top-150 pool is what lets calibration reproduce the full head-mid-tail distribution.
What would settle it
Re-run the same experiment with a larger candidate pool, for example the top 500 items per user, or with a base model whose top-150 recall of tail items is matched across BPR, LORE, and USG; if LORE plus CP then no longer has a unique advantage in reproducing the tail distribution, the paper's central comparison is an artifact of pool composition rather than of the calibration mechanism itself.
Extended reading notes
Core claim
Stated on the paper's own terms: for POI recommendation, the effect of context-awareness on popularity bias depends on the model, so context-awareness should not be treated as a single debiasing strategy; and calibrated popularity (CP) re-ranking aligns list popularity with the user's historical check-in popularity distribution, but only within what the base model's candidate pool contains. Applying CP to BPR's top-150 lists lowers popularity bias and approximates user profiles for head and mid items, yet tail items remain scarce because BPR's pool lacks them. LORE alone overcorrects, recommending items less popular than the user profile. When CP re-ranks LORE's top-150 list, the resulting recommendations show the closest match to user popularity profiles of all methods and combinations tested, including a faithful share of tail items, while increasing accuracy relative to LORE's uncalibrated base output. The paper therefore claims that the combination of context-awareness and calibration yields popularity-calibrated recommendations: accurate lists that closely mirror each user's popularity profile.
Load-bearing premise
The main result depends on the candidate pool: CP re-ranks only the top 150 items a base model retrieves, and the paper notes that BPR's pool contains too few tail items, so the finding that LORE plus CP best reflects the tail distribution is conditional on each model's recall of tail items and on 150 being the chosen pool size.
Editorial extensions
If this is right
- Practitioners aiming to reduce popularity bias should pair calibrated re-ranking with a context-aware model that surfaces tail items, rather than apply calibration to a standard collaborative-filtering baseline.
- Context-aware models cannot be treated as a uniform debiasing solution, since LORE and USG move popularity bias in opposite directions depending on the dataset.
- The choice of $\lambda$ is the main accuracy-bias lever: the calibration-oriented variant reduces bias more but costs accuracy, while the harmonic-mean variant preserves accuracy with little distributional change.
- Combining LORE with CP is the only studied configuration that reflects tail items faithfully, so it is the recommended setup when mitigating popularity bias is the primary goal; if accuracy is primary, BPR or USG with accuracy-oriented CP is preferable.
Reading between the lines
- Beyond the paper, a testable extension is to increase the candidate pool size or measure each base model's tail recall; if BPR's tail alignment improves accordingly, the LORE advantage is a recall effect of the base model rather than an intrinsic property of calibration.
- The group-specific $\lambda$ grid search invites a per-user version of calibration, which could reveal whether users within the same popularity group need different accuracy-bias trade-offs.
- If deployed, popularity-calibrated lists imply a provider-side consequence: niche POIs gain exposure only for users whose check-in histories already include tail items, so a multistakeholder objective might deliberately exceed the user's historical tail share to benefit niche providers.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper empirically compares two context-aware POI recommendation models (LORE and USG) and calibrated popularity (CP) re-ranking against a BPR baseline, using four real-world datasets (Foursquare, Yelp, Brightkite, Gowalla) of which only Foursquare and Yelp are reported in the main text. Users are split into LowPop, MedPop, and HighPop groups, and items into T/M/H popularity classes. RQ1 evaluates the individual effects of context-awareness and CP on accuracy and popularity bias; RQ2 evaluates their combinations. The main findings are that context-aware models have divergent effects, CP reduces popularity bias with varying accuracy trade-offs, and combining LORE with CP yields the closest alignment with users' popularity profiles, including T-item distribution, making it 'the only method that accurately reflects the distribution of T-items.' The paper also reports statistical significance tests and provides code and data on GitHub.
Significance. If the claims hold, the paper provides a useful empirical benchmark for popularity-bias mitigation in POI recommendation and offers practical guidance on choosing between context-aware and calibration-based strategies. The manuscript has clear strengths: it uses real-world datasets, reports significance-tested results, distinguishes two CP variants, and is transparent about its code and data availability. However, the central 'balance' and 'only method' claims are weakened by circular tuning of the CP_H variant and by a candidate-pool confound at m=150, and the cross-dataset generalization to all four datasets is not verifiable from the manuscript itself. These issues do not invalidate the work but require substantive revision before the conclusions can be considered robust.
major comments (4)
- [§2.3, Eq. (2) and §3 (Tables 2–3)] The CP_H variant selects λ by maximizing the harmonic mean of nDCG and (1−JSD), which are the same quantities used to report the accuracy–calibration trade-off in Tables 2–3 and Figure 1. The finding that CP_H 'balances' accuracy and popularity alignment is therefore partly a consequence of the optimization target rather than an independent empirical discovery. Additionally, the methodological description is inconsistent: Section 2.3 says λ is found by grid search, but Eq. (2) gives a closed-form expression, and that expression is not the harmonic mean as written (it lacks the factor 2). Please clarify how λ is actually chosen (e.g., validation-set grid search vs. the closed form), and report results for a λ selection criterion that does not include nDCG and JSD—for example, a fixed λ sweep or the CP_I variant—so that the balance claim can be tested against an independent criterion.
- [§2.4 and §4] The central RQ2 conclusion that 'the combination of LORE and CP is the only method that accurately reflects the distribution of T-items' is conditional on the base-list size m=150. CP re-ranks only the top-150 candidates produced by the base model, and it cannot introduce T-items that are absent from that pool. The paper itself acknowledges in Section 4 that BPR's top-150 recommendations lack T-items, so BPR+CP fails for reasons unrelated to calibration quality. The comparison is therefore confounded with base-model tail recall at cutoff 150. Please test sensitivity to m (e.g., m=50, 150, 500, 1000) and show whether LORE+CP remains the only method that reflects T-item distributions, or explicitly restrict the claim to the chosen pool size.
- [§2.2, Table 1, and §4] The abstract and Section 4 make statements about 'all four datasets' (for example, 'In all four datasets, the combination of BPR and CP_H hardly improves...'), but the manuscript reports results only for Foursquare and Yelp; Brightkite and Gowalla results are only available on GitHub. As a result, the cross-dataset generalization cannot be verified from the manuscript. Please include at least the All-group rows for Brightkite and Gowalla in an appendix or supplementary file, or explicitly restrict the generalization statements to the two datasets actually analyzed in the paper.
- [§2.3] USG is presented as a context-aware model that 'harnesses User preference, Social influence, and Geographical influence,' but the experiments disable the social component ('We do not use data on social relations'). This is a substantial modification of the original model, and it makes the RQ1 conclusion that 'context-aware models cannot be considered a uniform solution' depend on a version of USG that is not fully context-aware. Please either evaluate USG with social relations on datasets that provide them (e.g., Brightkite or Gowalla) or reframe the claims as being about the specific variants studied, e.g., 'USG without social relations.'
minor comments (4)
- [Table 3] For LORE in the Foursquare dataset, the CP_H and CP_I columns are identical for every user group, and several Yelp rows also show identical or nearly identical values; please explain whether the grid search selected λ=1 for these cases or whether the optimization converges to the same solution for both variants.
- [§2.3] The paper states that λ is optimized via grid search but does not report the grid range, step size, or the validation split used for this selection; adding these details is necessary for reproducibility.
- [Tables 2 and 3] The significance testing is described only as a t-test with Bonferroni correction; please specify whether the tests are paired per user, how many users are in each group, and whether the comparisons are against the BPR or Base condition as stated in the table headers.
- [Figure 1] The subfigures are small and the group labels are difficult to read; consider enlarging the figure or splitting it into separate panels per user group.
Circularity Check
No significant circularity: the paper is an empirical evaluation whose central claims are test-set measurements, not conclusions forced by construction or by self-citation.
full rationale
This paper is an empirical comparison study rather than a derivation, so most circularity patterns do not apply. The central RQ2 claim that LORE+CP is 'the only method that accurately reflects the distribution of T-items' is supported by the test-set results in Table 3 and Figure 1, which are measurements of the produced lists, not equations that force the outcome. Calibrated popularity (CP) is defined in Section 2.3 as re-ranking to optimize a weighted combination of relevance and Jensen-Shannon divergence (Eq. 1), so it is by design intended to improve distributional similarity; however, the paper does not present this as a prediction. The actual research contribution is the quantitative trade-off with accuracy and the comparison across models, which is not guaranteed by construction. For example, LORE+CP improves nDCG substantially in Foursquare while also changing popularity, and this trade-off could have gone the other way. The CP_H variant tunes lambda per user group on validation data by maximizing the harmonic mean of nDCG and (1-JSD) (Eq. 2), so the reported 'balance' for CP_H is partly a consequence of the selection criterion; but this is a transparent hyperparameter choice, not a hidden fitted input. Moreover, the main 'only method' conclusion about T-items holds for both CP_H and CP_I, and the LORE columns in Table 3 are identical for the two variants, making the conclusion independent of the lambda tuning. The paper's own admission that BPR+CP suffers from 'a lack of T-items in the top-150 recommendations produced by BPR that form the basis for the re-ranking' is a validity threat about candidate-pool composition and recall, not a circular reduction of the claim to its inputs. Self-citations such as [16,17,24,30] are contextual or methodological and are not load-bearing for the empirical findings; the base models come from external frameworks CAPRI [32] and RecBole [37]. No equation in the paper is equivalent by construction to its headline result, so there is no significant circularity.
Assumptions & free parameters
free parameters (3)
- lambda (CP_H) =
per user group, optimized on validation via grid search or Eq. (2)
- lambda (CP_I) =
1 (fixed)
- base list size m =
150
assumptions (5)
- domain assumption ARP and PopLift are valid and sufficient measures of popularity bias in POI recommendations
- domain assumption A user's historical check-in popularity distribution (normalized frequencies) captures her popularity preference
- domain assumption User and item percentiles (20/60/20) define meaningful LowPop/MedPop/HighPop and T/M/H groups
- ad hoc to paper LORE and USG without social relations remain representative context-aware POI models
- ad hoc to paper The top-150 base list is a sufficient pool for calibration to recover the user's popularity distribution
Cite this review
Pith. "Pith review of Exploring the Effect of Context-Awareness and Popularity Calibration on Popularity Bias in POI Recommendations." pith.science (2026). https://pith.science/paper/YALWCIKC
@misc{pith2026250703503,
author = {Pith},
title = {Pith review of: Exploring the Effect of Context-Awareness and Popularity Calibration on Popularity Bias in POI Recommendations},
year = {2026},
howpublished = {\url{https://pith.science/paper/YALWCIKC}},
note = {Machine review of arXiv:2507.03503}
}
read the original abstract
Point-of-interest (POI) recommender systems help users discover relevant locations, but their effectiveness is often compromised by popularity bias, which disadvantages less popular, yet potentially meaningful places. This paper addresses this challenge by evaluating the effectiveness of context-aware models and calibrated popularity techniques as strategies for mitigating popularity bias. Using four real-world POI datasets (Brightkite, Foursquare, Gowalla, and Yelp), we analyze the individual and combined effects of these approaches on recommendation accuracy and popularity bias. Our results reveal that context-aware models cannot be considered a uniform solution, as the models studied exhibit divergent impacts on accuracy and bias. In contrast, calibration techniques can effectively align recommendation popularity with user preferences, provided there is a careful balance between accuracy and bias mitigation. Notably, the combination of calibration and context-awareness yields recommendations that balance accuracy and close alignment with the users' popularity profiles, i.e., popularity calibration.
Figures
Forward citations
Cited by 2 Pith papers
-
A Multistakeholder Approach to Value-Driven Co-Design of Recommender System Evaluation Metrics in Digital Archives
A qualitative study translates stakeholder values from digital archives focus groups into a four-stage research funnel and eight proposed recommender evaluation metric directions.
-
Multistakeholder Fairness in Tourism: What can Algorithms learn from Tourism Management?
A comparative literature review shows tourism management and computer science define multistakeholder fairness differently, and argues algorithmic design should adopt qualitative, participatory methods from tourism research.
Reference graph
Works this paper leans on
-
[1]
Himan Abdollahpouri and Robin Burke. 2019. Multi-stakeholder recommenda- tion and its connection to multi-sided fairness.arXiv preprint arXiv:1907.13158 (2019)
arXiv 2019
-
[2]
Himan Abdollahpouri and Masoud Mansoury. 2020. Multi-sided exposure bias in recommendation.arXiv preprint arXiv:2006.15772(2020)
arXiv 2020
-
[3]
Himan Abdollahpouri, Masoud Mansoury, Robin Burke, and Bamshad Mobasher
-
[4]
Himan Abdollahpouri, Masoud Mansoury, Robin Burke, Bamshad Mobasher, and Edward Malthouse. 2021. User-centered evaluation of popularity bias in recommender systems. InProceedings of the 29th ACM conference on user modeling, adaptation and personalization. 119–129
2021
-
[5]
Ashmi Banerjee, Paromita Banik, and Wolfgang Wörndl. 2023. A review on indi- vidual and multistakeholder fairness in tourism recommender systems.Frontiers in big Data6 (2023), 1168692
2023
-
[6]
Ashmi Banerjee, Tunar Mahmudov, Emil Adler, Fitri Nur Aisyah, and Wolfgang Wörndl. 2025. Modeling sustainable city trips: integrating CO 2 e emissions, popularity, and seasonality into tourism recommender systems.Information Technology & Tourism(2025), 1–38
work page 2025
-
[7]
Martin Becker, Philipp Singer, Florian Lemmerich, Andreas Hotho, Denis Helic, and Markus Strohmaier. 2015. Photowalking the City: Comparing Hypothe- ses About Urban Photo Trails on Flickr. InSocial Informatics, Tie-Yan Liu, Christie Napa Scollon, and Wenwu Zhu (Eds.). Springer International Publishing, Cham, 227–244
work page 2015
-
[8]
Robin Burke, Gediminas Adomavicius, Toine Bogers, Tommaso Di Noia, Dominik Kowald, Julia Neidhardt, Özlem Özgöbek, Maria Soledad Pera, Nava Tintarev, and Jürgen Ziegler. 2025. De-centering the (Traditional) User: Multistakeholder Evaluation of Recommender Systems.arXiv preprint arXiv:2501.05170(2025)
arXiv 2025
Show all 40 references
-
[9]
Robin Burke, Gediminas Adomavicius, Toine Bogers, Tommaso Di Noia, Dominik Kowald, Julia Neidhardt, Özlem Özgöbek, Maria Soledad Pera, and Jürgen Ziegler
-
[10]
Jiawei Chen, Hande Dong, Xiang Wang, Fuli Feng, Meng Wang, and Xiangnan He. 2023. Bias and debias in recommender system: A survey and future directions. ACM Transactions on Information Systems41, 3 (2023), 1–39
2023
-
[11]
Yashar Deldjoo, Alejandro Bellogin, and Tommaso Di Noia. 2021. Explaining rec- ommender systems fairness and accuracy through the lens of data characteristics. Information processing & management58, 5 (2021), 102662
2021
-
[12]
Nada Ghanem, Stephan Leitner, and Dietmar Jannach. 2022. Balancing con- sumer and business value of recommender systems: A simulation-based analysis. Electronic Commerce Research and Applications55 (2022), 101195
2022
-
[13]
Aryan Jadon and Avinash Patil. 2024. A comprehensive survey of evaluation tech- niques for recommendation systems. InInternational Conference on Computation of Artificial Intelligence & Machine Learning. Springer, 281–304
2024
-
[14]
Kalervo Järvelin and Jaana Kekäläinen. 2002. Cumulated gain-based evaluation of IR techniques.ACM Transactions on Information Systems (TOIS)20, 4 (2002), 422–446
2002
-
[15]
Anastasiia Klimashevskaia, Mehdi Elahi, Dietmar Jannach, Lars Skjærven, Astrid Tessem, and Christoph Trattner. 2023. Evaluating the effects of calibrated popu- larity bias mitigation: a field study. InProceedings of the 17th ACM Conference on Recommender Systems. 1084–1089
2023
-
[16]
Dominik Kowald and Emanuel Lacic. 2022. Popularity bias in collaborative filtering-based multimedia recommender systems. InInternational Workshop on Algorithmic Bias in Search and Recommendation. Springer, 1–11
2022
-
[17]
Dominik Kowald, Markus Schedl, and Elisabeth Lex. 2020. The unfairness of popularity bias in music recommendation: A reproducibility study. InAdvances in Information Retrieval: 42nd European Conference on IR Research, ECIR 2020, Lisbon, Portugal, April 14–17, 2020, Proceedings...
2020
-
[18]
Emanuel Lacic, Leon Fadljevic, Franz Weissenboeck, Stefanie Lindstaedt, and Dominik Kowald. 2022. What drives readership? an online study on user inter- face types and popularity bias mitigation in news article recommendations. In European Conference on Information Retrieval. ...
2022
-
[19]
Emanuel Lacic, Dominik Kowald, Denis Parra, Martin Kahr, and Christoph Trat- tner. 2014. Towards a scalable social recommender engine for online market- places: The case of apache solr. InProceedings of the 23rd International Conference on World Wide Web. 817–822
2014
-
[20]
Emanuel Lacic, Dominik Kowald, Paul Christian Seitlinger, Christoph Trattner, and Denis Parra. 2014. Recommending Items in Social Tagging Systems Using Tag and Time Information. InIn Proceedings of the 1st Social Personalization Workshop co-located with the 25th ACM Conference...
2014
-
[21]
Oleg Lesota, Stefan Brandl, Matthias Wenzel, Alessandro B Melchiorre, Elisabeth Lex, Navid Rekabsaz, and Markus Schedl. 2022. Exploring Cross-group Discrep- ancies in Calibrated Popularity for Accuracy/Fairness Trade-off Optimization.. InMORS@ RecSys
2022
-
[22]
David Massimo and Francesco Ricci. 2021. Popularity, novelty and relevance in point of interest recommendation: an experimental analysis.Information Technology & Tourism23, 4 (2021), 473–508
2021
-
[23]
Pavel Merinov, David Massimo, and Francesco Ricci. 2022. Sustainability Driven Recommender Systems.CEUR Workshop Proceedings, Vol.3177, 22(June 2022)
2022
-
[24]
Peter Müllner, Elisabeth Lex, Markus Schedl, and Dominik Kowald. 2024. The impact of differential privacy on recommendation accuracy and popularity bias. InEuropean Conference on Information Retrieval. Springer, 466–482
2024
-
[25]
Hossein A Rahmani, Yashar Deldjoo, and Tommaso Di Noia. 2022. The role of context fusion on accuracy, beyond-accuracy, and fairness of point-of-interest recommendation systems.Expert Systems with Applications205 (2022), 117700
2022
-
[26]
Hossein A Rahmani, Yashar Deldjoo, Ali Tourani, and Mohammadmehdi Naghi- aei. 2022. The unfairness of active users and popularity bias in point-of-interest recommendation. InInternational Workshop on Algorithmic Bias in Search and Recommendation. Springer, 56–68
2022
-
[27]
Steffen Rendle, Christoph Freudenthaler, Zeno Gantner, and Lars Schmidt- Thieme. 2012. BPR: Bayesian personalized ranking from implicit feedback.arXiv preprint arXiv:1205.2618(2012)
2012 arXiv
-
[28]
Pablo Sánchez and Alejandro Bellogín. 2022. Point-of-interest recommender systems based on location-based social networks: a survey from an experimental perspective.ACM Computing Surveys (CSUR)54, 11s (2022), 1–37
2022
-
[29]
Joy Lal Sarkar, Abhishek Majumder, Chhabi Rani Panigrahi, Sudipta Roy, and Bibudhendu Pati. 2023. Tourism recommendation system: A survey and future research directions.Multimedia tools and applications82, 6 (2023), 8983–9027
2023
-
[30]
Harald Semmelrock, Tony Ross-Hellauer, Simone Kopeinik, Dieter Theiler, Armin Haberl, Stefan Thalmann, and Dominik Kowald. 2025. Reproducibil- ity in machine-learning-based research: Overview, barriers, and drivers.AI Magazine46, 2 (2025), e70002
2025
-
[31]
Harald Steck. 2018. Calibrated recommendations. InProceedings of the 12th ACM conference on recommender systems. 154–162
2018
-
[32]
Ali Tourani, Hossein A Rahmani, Mohammadmehdi Naghiaei, and Yashar Deld- joo. 2024. CAPRI: Context-aware point-of-interest recommendation framework. Software Impacts19 (2024), 100606
2024
-
[33]
Robin Ungruh, Karlijn Dinnissen, Anja Volk, Maria Soledad Pera, and Hanna Hauptmann. 2024. Putting Popularity Bias Mitigation to the Test: A User-Centric Evaluation in Music Recommenders. InProceedings of the 18th ACM Conference on Recommender Systems. 169–178
2024
-
[34]
Yining Wang, Liwei Wang, Yuanzhi Li, Di He, and Tie-Yan Liu. 2013. A theoretical analysis of NDCG type ranking measures. InConference on learning theory. PMLR, 25–54
2013
-
[35]
Mao Ye, Peifeng Yin, Wang-Chien Lee, and Dik-Lun Lee. 2011. Exploiting geo- graphical influence for collaborative point-of-interest recommendation. InPro- ceedings of the 34th international ACM SIGIR conference on Research and develop- ment in Information Retrieval. 325–334
2011
-
[36]
Jia-Dong Zhang, Chi-Yin Chow, and Yanhua Li. 2014. Lore: Exploiting sequential influence for location recommendations. InProceedings of the 22nd ACM SIGSPA- TIAL international conference on advances in geographic information systems. 103–112
2014
-
[37]
Wayne Xin Zhao, Shanlei Mu, Yupeng Hou, Zihan Lin, Yushuo Chen, Xingyu Pan, Kaiyuan Li, Yujie Lu, Hui Wang, Changxin Tian, et al. 2021. Recbole: Towards a unified, comprehensive and efficient framework for recommendation algorithms. Inproceedings of the 30th acm international ...
2021
-
[38]
Xu Zhou, Zhuoran Wang, Xuejie Liu, Yanheng Liu, and Geng Sun. 2024. An improved context-aware weighted matrix factorization algorithm for point of interest recommendation in LBSN.Information Systems122 (2024), 102366. 6
2024
-
[2019]
The unfairness of popularity bias in recommendation.arXiv preprint arXiv:1907.13286(2019)
2019 arXiv
-
[2024]
Dagstuhl Seminar on Evaluation Perspectives of Recommender Systems: Multistakeholder and Multimethod Evaluation.Dagstuhl Report on Evaluation Perspectives of Recommender Systems: Driving Research and Education(2024)
2024
Reviewed August 6, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.