{"id":"c4e80474-021b-45e3-adee-450c7b7483e2","arxiv_id":"2505.04143","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"Across ten years of ICPC data, Codeforces online ratings predicted World Finals rankings (tau = 0.596) better than any superregional contest, and Northern Eurasia had the strongest regional-to-superregional consistency (tau = 0.666).","lead":"This study ranks how consistently competitive programming teams perform across different levels of ICPC contests, from regional rounds to the World Finals. It finds that online contest ratings predict World Finals results better than any single regional championship, which could inform how programming contests are designed.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Codeforces-vs-World-Finals tau is computed globally across regions, while superregional-vs-World-Finals tau is within-region; the headline comparison may be inflated by between-region skill differences.","rationale":"The paper is transparent and reproducible in intent: it releases data and scripts, uses a simple and interpretable rank-correlation method, and explicitly acknowledges missing-data and selection limitations in Sections 5.6 and 5.7. The raw within-region correlations are plausible and useful as descriptive statistics. The most load-bearing weakness is not missingness alone, but the fact that the headline comparison of Codeforces to ICPC tiers mixes a global correlation with within-cluster correlations. Superregional-to-World-Finals tau is inherently within-region because superregional ranks across regions are not comparable, while Codeforces-to-World-Finals tau as reported pools all regions. The paper's own CF-vs-superregional rows are substantially lower than the global CF-vs-WF row, consistent with a large between-region contribution. A straightforward rerun of the public code, restricted to within-region pairs, would settle whether the Codeforces advantage is real or an artifact. The reader's weakest assumption about missing data and selection is real and related, but the comparability of the correlation domains is more direct and currently unaddressed. I therefore keep the CONDITIONAL verdict, with the condition that the within-region CF-vs-WF analysis be reported and the headline revised accordingly.","tokens_in":12347,"tokens_out":8411,"duration_ms":97704,"concrete_test":"Recompute the Table 5 World Finals row as a within-superregion, within-year analysis: for each superregion and each year 2021-2024, compute Kendall's tau between Codeforces rating and World Finals rank among teams from that region in that year, then take the pair-weighted average across region-years. Compare this within-region Codeforces average to (a) Table 5's reported 0.596 and (b) Table 4's weighted superregional-to-WF average of 0.407. Also recompute Table 4 using only 2021-2024 teams that have Codeforces ratings, so both analyses are on identical sets of teams and years. If the within-region CF-wF average remains close to 0.596 and exceeds the superregional average, the headline claim survives; if it drops to roughly 0.4 or below, the claimed Codeforces advantage is an artifact of pooling across regions.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The central empirical claim in the abstract and Section 3.2—that Codeforces ratings predict World Finals results (tau=0.596) better than any single ICPC tier (weighted average tau=0.407)—compares two non-comparable correlation domains. In Table 4, superregional-to-World-Finals Kendall's tau is computed only on pairs of teams from the same superregional contest in the same year, so cross-region pairs are excluded by construction; superregional rank is only meaningful within a region. In Table 5, the World Finals row is computed on all teams with Codeforces ratings across all regions and years, so cross-region pairs contribute to the tau. If Codeforces ratings differ systematically by region—which the paper's own Table 5 rows suggest, with CF-vs-superregional correlations ranging from 0.226 (Asia East) to 0.545 (Northern Eurasia)—and regions differ in World Finals performance, then the global CF-vs-WF tau is inflated by between-region signal. A within-region superregional rank cannot contain that signal. Therefore the inference that online contests \"capture decisive skill factors that many superregional sets miss\" does not follow from the reported numbers; the advantage of Codeforces may reflect cross-region rating differences rather than superior within-region predictive power. The paper flags missing-data concerns in Section 5.7, but does not address this comparability issue.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper studies the consistency of team rankings across tiers of the ICPC (regional, superregional, World Finals) and against Codeforces ratings. Using publicly available standings for 2015–2024 for five superregional championships, local regionals for North America and Northern Eurasia, and Codeforces rating data for 366 World Finals teams (2021–2024), the authors compute Kendall's tau between rank lists for pairs of teams that overlap between contests. They report a weighted superregional-to-World-Finals tau of 0.407, region-specific values from 0.188 (Asia West) to 0.521 (Northern Eurasia), strong regional-to-superregional consistency in North America (0.577) and Northern Eurasia (0.666), and a Codeforces-to-World-Finals tau of 0.596, which they interpret as evidence that high-frequency online contests capture skill factors that superregional contests miss. The paper concludes with recommendations for contest design and education, and releases code and data for replication.","tokens_in":12606,"tokens_out":7526,"duration_ms":74260,"significance":"If the headline comparisons are valid, the paper would offer a useful, data-driven benchmark for contest organizers and educators, and the public release of code and data is a strength that supports replication. The use of rank correlation is appropriate for the research question, and the authors are transparent about data gaps and about the correlational and non-causal nature of the design. However, the central RQ2 claim rests on comparing a global Codeforces correlation with within-region superregional correlations, and no uncertainty quantification is provided; these issues must be addressed before the comparative conclusions can be accepted.","major_comments":[{"comment":"The headline comparison that Codeforces ratings predict World Finals results better than any superregional (tau=0.596 vs 0.407) compares non-comparable quantities. The superregional-to-World-Finals taus in Table 4 are computed only on pairs of teams from the same superregional contest in the same year, so cross-region pairs are excluded by construction. The World Finals row in Table 5 is computed on pairs of all World Finals teams with Codeforces ratings in the same year, pooling teams across regions; this global tau includes between-region pairs. If Codeforces ratings differ systematically by region (as suggested by the range of region-specific CF correlations in Table 5, from 0.226 to 0.545) and regions differ in World Finals performance, the global CF-vs-WF tau is inflated by between-region signal that a within-region superregional rank cannot contain. The conclusion that online contests 'capture decisive skill factors that many superregional sets miss' therefore does not follow from the reported numbers. Please recompute the CF-vs-WF tau separately within each superregion (or otherwise condition on region) and compare those values with the corresponding superregional-vs-WF taus, or report a pooled analysis that is comparable across both settings.","section":"§3.2, Table 5; §3.1, Table 4"},{"comment":"No confidence intervals, standard errors, or hypothesis tests are reported for any of the Kendall's tau estimates, yet the Discussion (RQ1) states that Northern Eurasia correlates with World Finals 'at a significantly stronger scale' than other contests. With pair counts that overlap across years and the same teams appearing in multiple comparisons, the differences among the reported coefficients (e.g., 0.521 vs 0.407; 0.188 vs 0.521) may or may not be statistically meaningful. Please provide uncertainty estimates (e.g., bootstrap or analytic standard errors) and, where claims of difference are made, tests or intervals for the differences.","section":"§3.1–§3.4, Tables 4–9"},{"comment":"The paper never states how tied ranks are treated. The formula in Section 2 defines tau as (concordant - discordant)/(# pairs), which assumes no ties; but Codeforces team ratings are averages of three integer ratings and can easily tie, and ICPC standings can contain ties depending on tie-breaking rules. If tied pairs are omitted from the denominator, the effective pair count changes; if instead tau-b or tau-c is used, the formula should be stated. Please specify the tie convention and, if it changes any reported values, update Tables 4–9.","section":"§2, Eq. (1)"},{"comment":"The manuscript acknowledges in Section 5.7 that missing data at the regional level may bias tau estimates downward and that imputation and sensitivity analysis are needed, and in Section 5.6 that the correlational design cannot separate selection effects from causal ones. These are not merely future work: they also bear on the current estimates, because the analyzed set of teams is a non-random subset of all participants and the exclusion counts are not reported. Please document the exact number of teams and pairs excluded at each step for the main comparisons, and include at least a basic sensitivity check (e.g., dropping years with known data gaps, or comparing against a restricted sample) to show that the reported rankings of regions are robust.","section":"§5.6, §5.7; Tables 4, 5"}],"minor_comments":[{"comment":"The footnote and table list 'T ajikistan' and 'T auridia' with odd spacing; these should read 'Tajikistan' and 'Taurida'.","section":"Table 8"},{"comment":"The phrase 'more then .1 difference' should be 'more than .1 difference'.","section":"Discussion RQ1"},{"comment":"The abstract says 'five long-running superregional championships,' but Table 1 lists ten entries including the European contests; please clarify the counting convention in the abstract or in Table 1.","section":"Abstract and Table 1"},{"comment":"The caption uses 'T au coefficient' where it should be 'tau coefficient'.","section":"Figure 4"},{"comment":"For the World Finals row, please state explicitly whether the 16,596 pairs are within-year pairs pooled across all regions, since this is central to the comparability issue raised in Major Comment 1.","section":"§3.2, Table 5"}],"recommendation":"major_revision","confidential_remarks":null},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: this is the first decade-scale comparison of superregional, World Finals, and Codeforces rank consistency that I know of, and the dataset is a real contribution. But the abstract's marquee claim—that Codeforces ratings beat every ICPC tier at predicting World Finals—rests on a comparison that is not apples-to-apples.\n\nThe paper pairs teams within the same superregional for superregional→WF tau (Table 4), so that tau only measures within-region consistency. The Codeforces→WF tau (Table 5) is computed on all teams across all regions, which lets between-region skill differences contribute. The paper's own Table 5 shows that CF correlations with superregional contests vary by region, and if regions differ in both average CF rating and WF performance, the global tau is inflated. Computing a pooled within-region CF→WF tau, or conditioning on region, is the obvious fix, and the authors' conclusion that online contests 'capture decisive skill factors that many superregional sets miss' doesn't follow from the numbers as reported. This is the paper's most important soft spot.\n\nWhat the paper does well: the data assembly is careful, the exclusions are documented, and the authors are honest about limitations—Section 5.6 says the design cannot separate selection from cause, and Section 5.7 flags missing-data bias. The regional variation in consistency (NEF at 0.52, Asia West at 0.19) is a genuinely useful descriptive finding even if the causal story is speculative. My smaller gripe: no confidence intervals or hypothesis tests, and no mention of how tied ranks are handled, which matters for a rank-correlation method.\n\nThe paper is a solid exploratory contribution with a fixable methodological flaw. I would send it to review, but with the expectation that the CF-vs-superregional comparison be re-run within region before it's accepted.\n\nWho should read it: contest organizers, ICPC regional leaders, and anyone studying rank-based consistency in competitions. I probably wouldn't cite the abstract claim as-is, but I'd cite the dataset and the within-region correlations.","headline":"New dataset, useful descriptive results, but the flagship Codeforces-vs-World-Finals comparison pools across regions while the superregional comparison is within-region, so the headline claim is not supported as reported.","tokens_in":13076,"tokens_out":2818,"would_cite":false,"duration_ms":27103,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Codeforces ratings outpredict every ICPC tier for World Finals ranks, under Kendall's tau.","keywords":["competitive programming","ICPC","Kendall's tau","rank correlation","Codeforces rating","performance consistency","contest design","computer science education"],"falsifier":"Recompute the superregional-to-World Finals taus after imputing the missing editions (Asia West 2021–2022, several Northern Eurasia regionals in 2019 and 2021, and any unreported local results) and compare the resulting values with the reported ones; if the Codeforces advantage over the superregional average ($\\tau_{\\mathrm{CF}} = 0.596$ versus weighted $\\tau = 0.407$) shrinks to near zero or reverses, the paper's central predictive-validity claim would be refuted.","tokens_in":12160,"feed_emoji":"🏆","tokens_out":4869,"duration_ms":40417,"temperature":0.7,"pith_summary":"This paper tries to establish that competitive programming contests at different levels are not interchangeable measures of algorithmic skill, and that some formats are much better than others at forecasting who will do well at the ICPC World Finals. Using ten years (2015–2024) of standings from five superregional championships and the World Finals, it measures how often teams that beat one another at one tier also beat one another at the next tier with Kendall's rank correlation $\\tau$. The central finding is that superregional ranks predict World Finals ranks only moderately (weighted $\\tau = 0.407$), while Codeforces ratings predict World Finals better ($\\tau = 0.596$) than any single ICPC tier. The paper concludes that contest organizers could improve fairness and educational value by studying the Northern Eurasia model and well-curated online rounds.","feed_headline":"Codeforces outpredicts every ICPC tier for World Finals ranks","feed_subtitle":"Online ratings (tau=0.596) beat every single ICPC tier at forecasting World Finals.","key_machinery":"The central object is Kendall's tau rank correlation coefficient, computed from pairs of teams that competed in both contests in the same year. For each such pair, the team that placed higher in the first contest is compared with its placement in the second; concordant pairs agree, discordant pairs flip, and $\\tau$ is the difference between concordant and discordant pair counts divided by the total number of pairs. Applied to overlapping contest editions, this coefficient does the work of quantifying 'performance consistency' between tiers without assuming a linear relationship between ranks. The paper interprets $\\tau$ magnitudes using standard cutoffs (moderate at $0.26$, strong at $0.49$, very strong at $0.71$).","core_discovery":"Across five long-running ICPC superregionals, the rank ordering of teams at the superregional championship is only a moderate predictor of their rank ordering at that year's World Finals (weighted $\\tau = 0.407$), and this consistency varies sharply by region: Northern Eurasia leads at $\\tau = 0.521$, while Asia West trails at $\\tau = 0.188$. Within-region consistency is stronger, with Northern Eurasia's local regionals versus its finals at $\\tau = 0.666$ and North America's at $\\tau = 0.577$. A team's average Codeforces rating before the contest correlates with World Finals placement at $\\tau = 0.596$, higher than any single superregional predicts, and Northern Eurasia Finals also align closely with Codeforces ($\\tau = 0.545$). The paper interprets this as evidence that contest format and problem style matter: frequent online rounds capture skill factors, particularly observation and ad-hoc problem solving, that many superregional sets do not.","pith_inferences":["The same rank-alignment method could be applied to other online platforms such as AtCoder, LeetCode, or HackerRank to test whether Codeforces' predictive edge comes from contest frequency or from its particular problem style.","The paper's own data caveats suggest that filling missing regional results, such as Asia West 2021–2022 and several Northern Eurasia regionals in 2019 and 2021, could move the reported taus; the true regional ordering of consistency may shift once imputed data are included.","If a superregional held two rounds in different seasons, comparing each round's correlation with World Finals would directly test whether repeated high-stakes sampling, rather than format alone, explains Codeforces' higher predictive power.","The discrimination-factor idea proposed for future work could turn these pairwise correlations into a practical diagnostic: regions with low inversion rates among advancing teams could certify that their contest separates world-class teams on relevant skill dimensions."],"forward_implications":["In Northern Eurasia, a team's regional standing is a reliable low-stakes forecast of its World Finals prospects, so coaches can use regional results to diagnose weaknesses and retest improvements across cycles.","In low-consistency regions such as Asia West ($\\tau = 0.188$), regional feedback is noisy; teams there may gain more from high-frequency online contests than from interpreting local standings.","If the Northern Eurasian and Codeforces formats differentiate teams better, aligning regional problem style and selection rules with those formats should make advancement to the World Finals fairer and more predictable.","Codeforces ratings provide a practical, platform-based benchmark that correlates more strongly with World Finals performance than any single ICPC tier, suggesting online ratings can serve as an auxiliary training or selection signal.","Because internal regional-to-superregional consistency is high in North America and Northern Eurasia, changes in a team's regional rank are more likely to reflect genuine skill development rather than contest idiosyncrasy in those regions."],"supporting_citations":[{"why":"Supplies the official ICPC standings for World Finals and superregional championships that all rank-correlation comparisons are computed from.","marker":"[12]"},{"why":"Justifies Kendall's tau as the chosen rank-correlation measure because it is more reliable and interpretable than Spearman's coefficient for this setting.","marker":"[22]"},{"why":"Provides the pre-contest Codeforces ratings for teams going to World Finals 2021, one of the three rating datasets used in the predictive comparisons.","marker":"[24]"},{"why":"Provides the pre-contest Codeforces ratings for teams going to World Finals 2023, another of the rating datasets.","marker":"[25]"},{"why":"Provides the pre-contest Codeforces ratings for teams going to World Finals 2024, completing the 2021–2024 Codeforces coverage.","marker":"[26]"},{"why":"Supplies the tau magnitude cutoffs used to label correlations as weak, moderate, strong, or very strong.","marker":"[33]"}],"fun_headline_variants":["Codeforces rating beats every ICPC tier at forecasting Worlds","ICPC superregionals vary: tau 0.521 best, 0.188 worst","Northern Eurasia's ICPC circuit best predicts Worlds; Asia West worst","Online contests (tau=.596) beat all ICPC tiers for Finals","Superregional ICPC ranks weakly predict Worlds (tau=.407)"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing assumption is that the teams that appear in both contests in the available years are representative enough that missing results and the non-random selection of which teams advance do not bias the Kendall's tau estimates.","fun_headline_variants_meta":{"raw":{"variants":["Codeforces rating beats every ICPC tier at forecasting Worlds","ICPC superregionals vary: tau 0.521 best, 0.188 worst","Northern Eurasia's ICPC circuit best predicts Worlds; Asia West worst","Online contests (tau=.596) beat all ICPC tiers for Finals","Superregional ICPC ranks weakly predict Worlds (tau=.407)"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001168,"raw_usage":{"total_tokens":4905,"prompt_tokens":1091,"completion_tokens":3814,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":707,"completion_tokens_details":{"reasoning_tokens":3718}},"tokens_in":707,"tokens_out":3814,"duration_ms":28625,"temperature":1.0,"reasoning_tokens":3718,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T23:36:12.213255+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Recompute the superregional-to-World Finals taus after imputing the missing editions (Asia West 2021–2022, several Northern Eurasia regionals in 2019 and 2021, and any unreported local results) and compare the resulting values with the reported ones; if the Codeforces advantage over the superregional average ($\\tau_{\\mathrm{CF}} = 0.596$ versus weighted $\\tau = 0.407$) shrinks to near zero or reverses, the paper's central predictive-validity claim would be refuted.","supporting_citations":[{"cited_title":"The ICPC regional results","cited_arxiv_id":null,"evidence_quote":"Supplies the official ICPC standings for World Finals and superregional championships that all rank-correlation comparisons are computed from."},{"cited_title":"nonparametric","cited_arxiv_id":null,"evidence_quote":"Justifies Kendall's tau as the chosen rank-correlation measure because it is more reliable and interpretable than Spearman's coefficient for this setting."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the pre-contest Codeforces ratings for teams going to World Finals 2021, one of the three rating datasets used in the predictive comparisons."},{"cited_title":"N.Teams going to ICPC WF 2023 (Egypt 2023, 2nd final) — WIP list","cited_arxiv_id":null,"evidence_quote":"Provides the pre-contest Codeforces ratings for teams going to World Finals 2023, another of the rating datasets."},{"cited_title":"N.Teams going to ICPC WF 2024 (Kazakhstan 2024) — WIP list","cited_arxiv_id":null,"evidence_quote":"Provides the pre-contest Codeforces ratings for teams going to World Finals 2024, completing the 2021–2024 Codeforces coverage."},{"cited_title":"Weak or strong? how to interpret a spearman or kendall correlation, Apr 2023","cited_arxiv_id":null,"evidence_quote":"Supplies the tau magnitude cutoffs used to label correlations as weak, moderate, strong, or very strong."}],"review_version":1}