{"id":"e52eb7bd-00d4-4076-8e98-a96f9dd4eea6","arxiv_id":"1908.03512","paper_version":1,"verdict":"REJECT","confidence":"HIGH","novelty_score":5.0,"correctness_risk":"high","formal_verification":"none","parameter_count":4,"one_line_summary":"XGBoost with urban indicators predicts Brazilian intercity commuter links (90.4% balanced accuracy) and flow volumes (R²=77.6%) on a balanced sample, but the evaluation does not reflect true network sparsity.","lead":"Researchers trained machine learning models on 22 urban indicators for 5,565 Brazilian cities and report 90.4% accuracy in predicting which city pairs have commuter flows and R²=77.6% for flow amounts. The headline numbers come from a balanced sample of equal numbers of real and fake links, so they overstate how well the method reconstructs the true, very sparse network.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Headline accuracy and R² are measured on a 50/50 balanced sample rather than the true 0.18%-dense network, and no full-network precision, recall, F1, or R² is reported; the central 'reconstructs the commuters network' claim is not supported by the evaluation.","rationale":"I agree with the Reader's identification: the weakest assumption is that balanced-sample metrics measure network reconstruction. My reading of the paper confirms this. The phrase 'reconstruct the commuters network' requires predicting edges among all candidate pairs, not among a 50/50 sample; precision and recall (or ranking metrics such as AUPRC) are the relevant quantities. Applying the reported confusion matrix to the true prevalence shows the model would have precision around 1.8%, meaning it would overwhelm any downstream analysis with false positives. The absence of a distance/population-only ML baseline is an additional weakness, but it is secondary because even a perfect feature set cannot rescue an evaluation that does not address the true class distribution. I also note that the paper itself criticizes prior work for evaluating only on existing edges, yet the balanced sample introduces non-existing edges at 50%, which is equally unrepresentative of the reconstruction task. The data source and modeling choices are otherwise standard, and the paper is transparent about its sampling, so this is a fixable evaluation flaw rather than an irreparable one; but as written the central claim is unsupported. The verdict should remain REJECT.","tokens_in":16708,"tokens_out":5204,"duration_ms":54256,"concrete_test":"Apply the final XGBoost classifier to all 30,963,660 directed city pairs (same features and decision threshold as in Fig. 3C) and report precision, recall, F1, and accuracy on the full network; then recompute the regression R² on the same full set, treating non-links as zero flow, and compare with a zeros-only baseline. If precision is near 2% or the full-network R² is substantially below 77.6%, the headline claims do not hold.","verdict_should_be":"REJECT","load_bearing_attack":"The most load-bearing problem is the evaluation distribution. The Methods section 'Class balancing' builds a 110,494-pair dataset with 50% true links and 50% randomly chosen non-links matched by state pair, and all reported metrics (90.4% accuracy, Fig. 3C; 77.6% R², Fig. 4C) are computed on a test set with the same artificial 50% prevalence. The real network has 55,247 directed edges out of 5,565×5,564=30,963,660 possible ordered pairs, i.e. 0.178% density. On such a network, the Fig. 3C confusion matrix (90% true positives, 91% true negatives) would give about 49,700 true positives but roughly 2.78 million false positives, a precision of about 1.8%: almost every predicted link would be spurious. Accuracy alone on the full network would be about 91% simply from predicting negatives, so accuracy cannot support reconstruction. Similarly, the regressor's R² is computed on a balanced set where half the rows are zero-flow non-links; no R², MAE, or similar metric is reported on the full directed-pair set, and the gravity/radiation comparison is on existing edges only, so the 77.6% figure is not comparable to the stated baselines. Without full-network metrics, the abstract's claim that machine learning 'can reconstruct the commuters network' is not established.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a supervised machine-learning approach—XGBoost classification and regression—to reconstruct the directed Brazilian intercity commuter network from distance, population sizes, and 22 urban indicators. It reports 90.4% accuracy for predicting the existence of a commuting link and 77.6% R² for predicting the number of commuters, compares these results with gravity and radiation models, and uses SHAP values to identify important features. The reported metrics are computed on a class-balanced sample of 110,494 city pairs (50% real links, 50% fabricated non-links matched by state pair), not on the full space of directed city pairs, and no full-network precision, recall, F1, or regression metrics are reported.","tokens_in":16979,"tokens_out":10036,"duration_ms":100709,"significance":"If the claimed levels of accuracy and explained variance held on the full network, this would be a valuable applied contribution to mobility modeling, in particular the systematic benchmarking of 27 classifiers and 23 regressors and the SHAP-based feature analysis. However, the headline numbers are computed on an artificial distribution with 50% link prevalence, whereas the true network has 0.178% density; the central claim that the model 'reconstructs the commuters network' with 90.4% accuracy is therefore not established by the reported evaluation. The appropriate evidence—precision/recall on the full pair space and commensurable regression benchmarks—is missing.","major_comments":[{"comment":"The 90.4% accuracy is measured on a test set with 50% true links and 50% randomly chosen non-links matched by state pair, not on the true 0.178%-dense network (55,247 directed edges among 5,565 x 5,564 = 30,963,660 possible ordered pairs). Applying the confusion matrix in Fig. 3C (90% true-positive rate, 91% true-negative rate) to the full network would yield about 49,700 true positives and about 2.78 million false positives, i.e., precision of roughly 1.8%, so almost every predicted link would be spurious. Accuracy on the full network would be about 91%, while the trivial always-negative classifier would obtain 99.8% accuracy; accuracy therefore cannot support the abstract's reconstruction claim. The authors should report full-network precision, recall, F1, and precision-recall curves, or clearly reframe the task as link prediction on a balanced sample rather than network reconstruction.","section":"Methods – Class balancing; Results – Fig. 3C"},{"comment":"The comparison between the proposed regressor and the gravity/radiation baselines is not apples-to-apples: the baselines are fitted and evaluated only on the existing non-zero flows (as stated in the paragraph after Eq. 2), whereas the XGBoost regressor's R² = 77.6% is computed on the balanced test set in which half of the rows are zero-flow non-links. R² values computed on these two different outcome distributions are not comparable, so the claim that the machine-learning model describes 77.6% of the variance and outperforms the baselines is not supported. The authors should report R², MAE, and RMSE for all methods on the same set of pairs—ideally both on all directed pairs and on existing edges only—and should also report error metrics for the zero/non-zero distinction.","section":"Results – State-of-the-art models and Reconstructing the weighted commuters network; Fig. 2 and Fig. 4C"},{"comment":"SHAP values describe the fitted model's decision mechanism; they are not estimates of causal effects. The abstract's statement that GDP and unemployment rate are 'driven-forces for people to commute' and the Discussion's attract/repel language go beyond what the evidence can support. Since the SHAP analysis is performed on the balanced sample, the feature rankings may also reflect the artificial sampling scheme rather than the true predictive relationship in the full network. I recommend rephrasing these results as 'features the learned model relies on' and, if causal claims are intended, adding a separate identification strategy.","section":"Interpreting machine learning models using SHAP; Discussion"}],"minor_comments":[{"comment":"The text says 'Fig. 5C compares the predictions...' but this should refer to Fig. 4C, not Fig. 5C.","section":"Results – Reconstructing the weighted commuters network"},{"comment":"The phrase 'Fundaçõ de Amparo' should read 'Fundação de Amparo'.","section":"Acknowledgments"},{"comment":"The title 'Modeling the polycentric pransition of cities' contains a typo; it should be 'polycentric transition'.","section":"Reference 4"},{"comment":"The word 'Theelderly population indicator' is missing a space and should be 'The elderly population indicator'.","section":"Discussion"},{"comment":"The phrase 'decreasing the number of dependable variables' should read 'decreasing the number of dependent variables'.","section":"Methods – Feature selection"},{"comment":"The definition of 'balanced accuracy score' as 'the ratio between the correct classified instances and all instances' is inconsistent with the confusion-matrix accuracy reported in Fig. 3C; please clarify whether the reported 90.4% is ordinary accuracy or a class-balanced metric.","section":"Methods – Model evaluation"},{"comment":"The terms 'gravitation model' and 'gravity model' are used interchangeably; please choose one term for consistency.","section":"Throughout"}],"recommendation":"major_revision","confidential_remarks":"The paper has a useful data set and a broad algorithmic comparison, but the presented evaluation protocol does not support the abstract's reconstruction claim. I would be willing to reconsider a revised version that reports full-network metrics (precision, recall, F1, precision-recall curves, and regression errors on all pairs and on existing edges) and tempers the causal language around SHAP."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: the headline numbers are real but they measure the wrong thing. The classifier and regressor are evaluated on a 50/50 balanced sample of links and non-links, while the true network has 0.18% density. So the 90.4% accuracy and 77.6% R² don't support the claim that you can reconstruct the commuter network. That said, the paper is not a waste. It brings a large Brazilian dataset (5,565 cities, 55k directed edges), a fair comparison of gravity and radiation models, and a clear description of the class-balancing step. The SHAP analysis is standard but gives some plausible signals (distance, GDP, area).\n\nThe load-bearing flaw is the evaluation distribution. On the full network, even with 90% true positives and 91% true negatives, precision would be around 1.8%—almost every predicted link would be spurious. The paper never reports precision, recall, F1, or any metric on the full directed-pair set. The regression R² is also computed on the balanced set with half zero-flow pairs, and the comparison to gravity/radiation is on existing edges only, so the numbers are not comparable. There is also no ML baseline using only distance and population, so we don't know if the 22 urban indicators add anything beyond standard features.\n\nThese are not minor omissions; they undermine the central claim. But they are fixable. The authors could re-run the evaluation on the full network (or a subsample with realistic prevalence), report precision/recall and a proper baseline, and revise the abstract. I'd send it to peer review with that expectation. The dataset and the question are worth refereeing.","headline":"The headline accuracy and R² are computed on a balanced sample, so the network-reconstruction claim is not supported; the paper is worth a revision, not a desk reject.","tokens_in":17591,"tokens_out":2290,"would_cite":false,"duration_ms":23005,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Supervised machine learning on urban indicators can reconstruct Brazil's intercity commuting network, with 90.4% accuracy on link existence and 77.6% of flow variance explained.","keywords":["commuters network","human mobility","link prediction","network reconstruction","urban indicators","gradient boosting","gravity model","radiation model"],"falsifier":"Train and test the same procedure on a random sample of all ordered city pairs, not just the balanced subset, and compute precision, recall, and the coefficient of determination on the full set of 55,247 actual flows; if accuracy falls to the trivial no-link baseline or the variance explained drops far below 77.6%, the central reconstruction claim is not established.","tokens_in":16461,"feed_emoji":"🚆","tokens_out":8610,"duration_ms":80476,"temperature":0.7,"pith_summary":"The paper tries to establish that machine learning on everyday urban statistics can reconstruct the Brazilian intercity commuter network, both identifying which city pairs have commuter flows and estimating how many people travel, and that this beats the standard gravity and radiation models. It reports 90.4% accuracy on link existence and 77.6% of the variance in flow volumes. A sympathetic reader would care because commuting data is expensive to collect while urban indicators are already measured, so a reliable mapping from indicators to flows would give planners and epidemic modelers a cheaper route to mobility networks. Distance remains the leading predictor, but the paper argues that GDP, area, traffic accidents, elderly population, and unemployment carry real signal beyond population and distance.","feed_headline":"Urban indicators rebuild Brazil's commuter network","feed_subtitle":"Distance, GDP, and city data predict who commutes where with 90.4 percent accuracy.","key_machinery":"The central object is a per-pair feature vector $S_{st}=\\{r_{st}, m_s, n_t, U_s, U_t\\}$ built from the distance, both city populations, and the 22 urban indicators of source and target city, giving 45 features per ordered pair of cities. Gradient-boosted decision trees map that vector to a binary link label and to a commuter count; the training set is balanced to one real link per one invented non-link, and Shapley-value attribution, a game-theoretic measure of each feature's marginal contribution, ranks the drivers that carry the predictions.","core_discovery":"The paper claims that a supervised machine-learning pipeline, given the distance between two cities, their populations, and 22 urban indicators for each, can decide whether a commuting link exists and how many people use it; on the 2010 Brazilian network of 5,565 cities and 55,247 non-zero flows, it reports 90.4% classification accuracy and 77.6% of flow variance explained, against under 25% variance explained by gravity and radiation models. It further claims, through game-theoretic feature attribution, that distance is the strongest predictor but that target-city GDP, area, traffic accidents, elderly population, and unemployment also shape commuting.","pith_inferences":["Editorial inference: the reported 90.4% accuracy and 77.6% R-squared describe performance on a half-real, half-invented dataset; on the true sparse network, accuracy would be dominated by the trivial 'no link' prediction unless precision and recall are reported.","Editorial inference: the same training recipe could be applied to other countries or years, and the feature-attribution ranking could show whether distance and GDP remain the dominant drivers or whether the result is specific to Brazil in 2010.","Editorial inference: the paper's method requires only node attributes, so it suggests a possible shortcut for reconstructing networks where link data is missing but attribute data exists; the authors hint at this transfer but do not demonstrate it."],"forward_implications":["City-to-city commuter flows can be estimated from routinely collected statistics, reducing reliance on costly origin-destination surveys.","Gravity and radiation models are not sufficient for network reconstruction because they predict a fully connected network, whereas the supervised classifier provides a way to predict non-links as well as links.","A small set of indicators, including distance, GDP, area, traffic accidents, elderly population, and unemployment, carries most of the predictive signal, so a feasible data-collection effort could reproduce the approach.","Because the method learns from node attributes rather than network topology, it applies to other networks where attributes are available but links are unknown, such as trade, social, or metabolic networks.","Interpreting the models with game-theoretic feature attribution converts black-box predictions into named drivers of commuting, offering testable hypotheses about why people move."],"supporting_citations":[{"why":"Defines the radiation model, the main physics-inspired baseline whose inability to reconstruct sparse link structure motivates the supervised approach.","marker":"[15]"},{"why":"Provides an empirical gravity-model application that, with Eq. 1, supplies the gravity baseline the paper's results are compared against.","marker":"[14]"},{"why":"Reports that gravity and radiation models fail to fit commuting flows outside the authors' own data, supporting the paper's criticism of these baselines.","marker":"[16]"},{"why":"Proposes the cost-based radiation model and illustrates that such models require known links, so they cannot reconstruct network structure.","marker":"[17]"},{"why":"Supplies the feature-attribution technique used to identify distance, GDP, area, traffic accidents, and other indicators as drivers of commuting.","marker":"[44]"},{"why":"Supplies the tree-boosting algorithm with which the classifier and regressor are induced.","marker":"[54]"},{"why":"Provides the Brazilian census data: daily commuting flows among 5,565 municipalities and the 22 urban indicators.","marker":"[46]"},{"why":"Establishes that urban indicators are collinear and scale with city size, justifying the feature-selection step and the interpretation of indicators as urban drivers.","marker":"[3]"}],"fun_headline_variants":["ML + urban indicators rebuild commuter net with 90% accuracy","City data predicts commuting flows: 90.4% accurate","Machine learning maps commuters via urban indicators","Urban indicators reveal commuter links with 90% accuracy","Distance, GDP, and ML decode who commutes where"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The headline numbers come from an artificially balanced sample that is half real commuting pairs and half randomly invented pairs, so they do not by themselves tell how the model performs on the true network, where fewer than two in a thousand city pairs have commuters.","fun_headline_variants_meta":{"raw":{"variants":["ML + urban indicators rebuild commuter net with 90% accuracy","City data predicts commuting flows: 90.4% accurate","Machine learning maps commuters via urban indicators","Urban indicators reveal commuter links with 90% accuracy","Distance, GDP, and ML decode who commutes where"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000181,"raw_usage":{"total_tokens":1311,"prompt_tokens":954,"completion_tokens":357,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":570,"completion_tokens_details":{"reasoning_tokens":276}},"tokens_in":570,"tokens_out":357,"duration_ms":4524,"temperature":1.0,"reasoning_tokens":276,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-14T14:10:21.017848+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Train and test the same procedure on a random sample of all ordered city pairs, not just the balanced subset, and compute precision, recall, and the coefficient of determination on the full set of 55,247 actual flows; if accuracy falls to the trivial no-link baseline or the variance explained drops far below 77.6%, the central reconstruction claim is not established.","supporting_citations":[{"cited_title":"& Stanley, H","cited_arxiv_id":null,"evidence_quote":"Provides an empirical gravity-model application that, with Eq. 1, supplies the gravity baseline the paper's results are compared against."},{"cited_title":"P., Serras, J., Johansson, A","cited_arxiv_id":null,"evidence_quote":"Reports that gravity and radiation models fail to fit commuting flows outside the authors' own data, supporting the paper's criticism of these baselines."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Proposes the cost-based radiation model and illustrates that such models require known links, so they cannot reconstruct network structure."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the feature-attribution technique used to identify distance, GDP, area, traffic accidents, and other indicators as drivers of commuting."},{"cited_title":"Accessed: 2017-09-01","cited_arxiv_id":null,"evidence_quote":"Provides the Brazilian census data: daily commuting flows among 5,565 municipalities and the 22 urban indicators."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Establishes that urban indicators are collinear and scale with city size, justifying the feature-selection step and the interpretation of indicators as urban drivers."}],"review_version":1}