{"id":"5df4e01f-2618-43eb-9725-e919a67ae599","arxiv_id":"2507.14696","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"Co-authorship network features improve out-of-sample prediction of whether a computer science PhD's first faculty position is at a top-10 department, beyond PhD rank and bibliometrics.","lead":"This paper tests whether a researcher's co-authorship network, measured before their first faculty job, helps predict whether they land at a top-10 computer science department. The authors find that adding network features improves prediction by up to 10% over traditional measures like PhD program prestige and publication counts.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Static GNNs train on graphs that contain test nodes, so the reported out-of-sample co-authorship gains may be transductive artifacts rather than generalizable signal.","rationale":"The paper's central contribution is explicitly framed as out-of-sample generalization (Sections 1, 2.2, and 3.1.3), and the largest reported gains come from static GNNs whose training protocol includes test nodes in message passing. This is the most load-bearing concern because it directly threatens the interpretation of the headline numbers: if test node features and edges are visible during training, the co-authorship advantage could reflect transductive access to the test-year network rather than a reproducible signal for a truly unseen candidate. The mixed-effects model's zero group variance (Table 3) is a separate concern about p-value calibration, but it is secondary because even a correctly specified random effect would not fix the fact that test nodes participate in training for static GNNs. Appendix A's rewiring experiment is good evidence that network structure matters, but it inherits the same protocol and cannot distinguish structural signal from transductive artifact. The code is publicly available, making the proposed inductive re-run feasible. I am not claiming the co-authorship signal is certainly absent; I am claiming the current evaluation cannot establish the claimed generalization until this protocol is tested. Since the reader's conditional verdict already hinges on this same assumption, no change to the verdict is needed.","tokens_in":37277,"tokens_out":6479,"duration_ms":83973,"concrete_test":"Retrain GAT, GCN, and GraphSAGE under an inductive protocol: for each test year t, train on the subgraph induced by Vtrain union Vval plus Vfaculty, with all edges incident to Vtest removed from the training graphs; at inference, add the held-out nodes' pre-hire features and edges and evaluate. Recompute the mixed-effects contrasts from Table 3 (e.g., PhD+Bib+Coauthor vs. PhD+Bib) on these inductive runs; if the PR-AUC gain falls below 0.029 or loses statistical significance, the transductive access to test nodes is responsible for part of the claimed improvement.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Contribution 2 claims co-authorship features significantly improve out-of-sample prediction (8.48%, p=0.005 over PhD rank; 7.32%, p=0.003 over PhD+Bib). The evaluation protocol undermines the 'out-of-sample' reading for the static GNN models that dominate the top results. Section 3.1.2 fixes V to all 4,656 researchers; Section 3.2.1 states 'Message passing is performed over the full node set V, but predictions and loss computation are restricted to Vhire using binary node masks.' For each test year t in {2018,2019,2020}, training on snapshot G_{t-1} thus includes the test nodes themselves: their features are in the input tensor and their edges participate in message passing during every training epoch, because train-node embeddings aggregate messages from test nodes. Section 3.1.3's claim of 'strict temporal separation' separates labels, not nodes or network context. The headline gain is therefore estimated by comparing tabular models that never see test features with GNNs that see test features and test edges throughout training. A model could exploit the test-year graph (e.g., future hires' collaboration clustering or ties to faculty) without learning anything that transfers to a genuinely unseen candidate whose network context was not in the training snapshot. If so, the co-authorship advantage is at least partly an artifact of transductive evaluation, not a property of the network signal.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper frames faculty placement as an individual-level prediction task and asks whether temporal co-authorship network structure adds predictive signal beyond PhD department rank and bibliometric features. The dataset consists of 4,656 US computer science faculty, with 1,974 hired between 2010 and 2020; the test set comprises hires from 2018-2020. The authors benchmark tabular classifiers, static GNNs (GCN, GAT, GraphSAGE), and a temporal GNN (GConvGRU), and use linear mixed-effects models to test whether adding co-authorship features improves PR-AUC for placement in top-X departments. They report significant gains for top-10 placement: 8.48% over PhD rank alone, 10.08% over bibliometric features alone, and 7.32% over the combined tabular baseline, with the best model (GAT with all features) reaching PR-AUC 0.458. They also report that co-authorship gains diminish as the high-rank threshold broadens, while PhD rank becomes more predictive.","tokens_in":37530,"tokens_out":6969,"duration_ms":80997,"significance":"If the main claims survive scrutiny, the paper would be a useful contribution to the science-of-science and academic-stratification literature: it moves from population-level correlations to an individual-level predictive framing, releases code and uses public data, and compares multiple model families. The finding that co-authorship structure helps most at the most selective placement thresholds is an interesting and falsifiable pattern. However, the headline quantitative claim is currently threatened by the transductive evaluation used for the static GNNs and by the degenerate random effect in the mixed-effects model. Both issues are addressable, and the paper’s strengths—temporal split design, benchmark breadth, public data and code—make it worthwhile to revise.","major_comments":[{"comment":"For the static GNNs, the paper states that message passing is performed over the full node set V and that predictions and loss are restricted to Vhire via binary masks. Since V includes the 2018-2020 test hires, the test nodes’ features and edges are visible to the model during training. The claim in Section 3.1.3 of “strict temporal separation” therefore separates labels but not network context. This makes the comparison against tabular models asymmetric: tabular models never see test features during training, while static GNNs do. The reported co-authorship gains in Table 3 (e.g., 7.32%, p=0.003 over PhD+Bib) may reflect the transductive access of static GNNs rather than the predictive value of co-authorship features for genuinely unseen candidates. I request an inductive evaluation in which test nodes are excluded from the message-passing graph during training (e.g., train on snapshots built from Vtrain and Vval only, then apply to a graph that includes test nodes only at inference time), and a within-architecture comparison (e.g., GAT with and without co-author features) to isolate the feature-set contribution from the transductive effect.","section":"Section 3.2.1 (static GNNs) and Section 3.1.3"},{"comment":"The linear mixed-effects model used to support the significance claims reports Group Variance = 0.000 (SE 0.003) in every specification. A zero variance estimate means the random intercept for model architecture carries no information, and the model effectively reduces to ordinary linear regression without an architecture effect. The paper’s statement that the model “controll[s] for model type” via the random effect is therefore not supported by the reported fit. The p-values for adding co-authorship should be re-derived with a method that actually accounts for repeated observations per architecture (e.g., cluster-robust standard errors or a likelihood-ratio test against a fixed-effects-only model), and the boundary estimate should be discussed and interpreted.","section":"Table 3 and Eq. (4)"},{"comment":"The evaluation uses a single temporal split with hires from 2010-2017 as train/validation and 2018-2020 as test. All conclusions about out-of-sample generalization rest on this one window, which is an untested assumption about representativeness; market dynamics, the COVID-era academic job market, or DBLP data completeness could affect this period. The authors should either run a rolling-origin evaluation (e.g., multiple cut points: train on 2010-Y, test on Y+1 for a range of Y) or explicitly qualify the claims as specific to this split. This is especially relevant for the threshold analysis in Section 5.2, where the confidence intervals in Figure 4 are computed within a single test set.","section":"Section 3.1.3 and Section 5"}],"minor_comments":[{"comment":"The caption states that the GAT achieves an average PR-AUC of 0.424 and the transformer achieves 0.458, but Table 2 reports the reverse (GAT 0.458, Transformer 0.424); please correct the caption.","section":"Figure 3 caption"},{"comment":"“Spaciotemporal” should be “spatiotemporal” in the description of GConvGRU.","section":"Section 3.2.1"},{"comment":"There are typographical errors: “co-authorhship” appears in the Table A1 caption, and “function fo” appears in the Table A6-A9 captions; please fix these.","section":"Table A1 and Tables A6-A9 captions"},{"comment":"The percentages given in the text mix units: “top-10 (approximately 5% of the 186 departments)” refers to departments, while the immediately following percentages (22%, 36%, 49%, 67%, 77%) refer to researchers hired at those thresholds; please clarify the units to avoid confusion.","section":"Section 5.2"},{"comment":"The sentence “Features 2 and 6-11 use attributes of a researcher’s co-authors” should be “Features 6-11 use attributes of a researcher’s co-authors”, since feature 2 (average number of authors per paper) is a property of the paper rather than of co-author attributes.","section":"Section 4.2"},{"comment":"The claim that no prior work incorporates co-authorship networks into predictive models of faculty placement should be narrowed to “no prior work evaluates such models on temporally held-out individuals,” given that Zuo et al. and Barnes et al. are cited as examining co-authorship and placement; the current wording overstates the novelty.","section":"Introduction, Contribution 1"}],"recommendation":"major_revision","confidential_remarks":"The paper’s central empirical claim is interesting but depends on an evaluation protocol that conflates co-authorship features with transductive access to test nodes. If the authors can provide an inductive comparison and fix the mixed-effects reporting, the paper could become a solid contribution. The scope fits a network-science or scientometrics venue. I also recommend checking the claim of being the first predictive treatment against the cited related work more carefully."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Bottom line: this is a solid empirical study with a real flaw in its headline generalization claim, and the flaw is fixable. The stress-test note is right.\n\nWhat it does well: it is genuinely the first to frame faculty placement as an individual prediction task with temporal co-authorship networks and a held-out test window. Code and data are public. The experiments are extensive: multiple architectures, feature sets, rank thresholds, and a rewiring control. The result that co-authorship structure adds predictive signal beyond PhD rank and bibliometrics for top-10 placement is plausible and consistent with prior descriptive work.\n\nThe soft spots are real. The static GNNs (GCN, GAT, GraphSAGE) are transductive: the graph snapshots used in training include the test nodes, and message passing runs over them. The paper claims 'strict temporal separation,' but that separates labels, not nodes or network context. A model could be exploiting the test candidates' collaboration patterns or ties to already-placed faculty without learning anything that transfers to a genuinely unseen candidate. This makes the comparison against tabular models unfair, because tabular models never see test features during training. The temporal GNN also uses the full node set, so the same concern applies, though less severely.\n\nSecond, the linear mixed-effects model reports zero estimated variance for the architecture random effect in every table. That means 'controlling for architecture' is empirically empty, and the p-values are computed as if all 480 observations were independent, which they are not (10 runs per model). The significance claims in Contribution 2 are weaker than they look.\n\nA smaller concern is the single 2018-2020 test window; one historical period does not establish robustness across eras.\n\nThat said, the central idea is not circular: labels are temporally held out, and the features are public pre-hire information. The paper is transparent about limitations, and the rewiring experiment is a good extra, even though it does not fully separate transductive artifact from real signal.\n\nWho it is for: people working on science-of-science prediction or academic hiring, and anyone wanting a clean cautionary example of transductive evaluation in GNNs. It deserves a serious referee. With a properly inductive evaluation (masking test nodes out of the training graph) and a corrected statistical test, it could be a genuinely useful contribution.","headline":"A credible first pass at predicting faculty placement from co-authorship networks, but the headline out-of-sample gains are overstated because the static GNNs see test nodes during training.","tokens_in":38084,"tokens_out":2714,"would_cite":false,"duration_ms":33501,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A researcher's pre-hire position in the co-authorship network predicts elite faculty placement beyond what PhD rank and publication metrics capture.","keywords":["faculty placement","co-authorship network","graph neural networks","temporal networks","academic hiring","prestige hierarchy","prediction","computer science faculty"],"falsifier":"Train the leading model on hires from 2010-2016 and evaluate on hires from 2017-2019, rebuilding each year's co-authorship graph so that no test-year candidate or any edge incident to them appears in a training snapshot; if the co-authorship gain over PhD rank alone falls below the reported 7-10% and loses statistical significance, the central claim would be falsified.","tokens_in":37044,"feed_emoji":"🎓","tokens_out":11183,"duration_ms":116313,"temperature":0.7,"pith_summary":"The paper aims to show that faculty placement—where a newly minted PhD lands their first tenure-track job—can be forecast at the individual level from pre-hire information, and that a candidate's position in the temporal co-authorship network carries predictive signal that PhD department prestige and bibliometric features do not. Using records for 4,656 US computer science faculty, the best model, a graph attention network trained on all three feature types, reaches a precision-recall area under the curve (PR-AUC) of 0.458 for placement at top-10 departments, and adding co-authorship features improves PR-AUC by 8.48% over PhD rank alone, 10.08% over bibliometrics alone, and 7.32% over both combined. The gains are largest at the top-10 threshold, where candidates look similar on paper, and fade as the definition of 'high-rank' broadens. If the claim holds, public collaboration records reveal part of what hiring committees respond to, which would give researchers a concrete way to study and potentially counter structural bias in academic hiring.","feed_headline":"Co-author networks beat PhD rank at predicting elite faculty hires","feed_subtitle":"Adding co-authorship structure lifts placement prediction up to 10%, with largest gains at top-10 departments.","key_machinery":"The object that carries the argument is the temporal co-authorship network, a sequence of cumulative undirected graphs with edge weights equal to the number of joint publications, plus node feature tensors for PhD rank and 22 bibliometric features. Over this network the paper trains graph neural networks, most notably a graph attention network (GAT) that weights neighbor contributions during message passing, and a convolutional gated recurrent unit that unrolls across snapshots. The analytic device that isolates network structure is feature-set comparison combined with degree-preserving rewiring: as edges are progressively randomized, PR-AUC falls sharply, showing that the specific higher-order arrangement of co-authorship, not just node degree, is what the models exploit.","core_discovery":"The central claim is that the structure of a candidate's pre-hire co-authorship network encodes information about the prestige of their first faculty placement that is not captured by PhD department rank or publication-based features. In the paper's terms, co-authorship acts as a proxy for the informal endorsement and social capital that hiring committees observe but that standard curriculum-vitae data omit. The evidence is a systematic model comparison: graph models that run message passing over the co-authorship network outperform all tabular models at every selectivity threshold, and the best configuration, a graph attention network using all three feature sets, raises PR-AUC for top-10 placement to 0.458 from 0.424 for the best non-network model. Mixed-effects regression attributes the gain to the co-authorship term, with the three relevant comparisons all statistically significant. The paper further shows that the effect is concentrated at the most elite departments and that degree-preserving rewiring of the network erodes performance, indicating that higher-order structure, not just degree, carries the signal.","pith_inferences":["The paper leaves implicit that public co-authorship data could be used as an audit tool: if network position predicts placement beyond merit proxies, hiring outcomes could be checked against collaboration patterns without access to confidential letters or committee deliberations.","A natural test of the 'social endorsement' interpretation is to add citation-based quality measures or explicit advisor-advisee and mentor ties; if those variables absorb the network effect, the recommendation-letter proxy story would need revision.","A stricter inductive evaluation, in which test-year candidates and their incident edges are absent from every training snapshot, would directly test how much of the reported co-authorship gain survives true out-of-sample prediction.","Porting the same framework to disciplines or countries with different collaboration norms would reveal whether the top-10 concentration of network effects is a general feature of prestige hierarchies or a computer-science-specific pattern."],"forward_implications":["A candidate's pre-hire co-authorship context is a measurable predictor of elite placement, on top of productivity and pedigree.","Graph-based models outperform all tabular models at every high-rank threshold tested, so network architecture is not incidental to the result.","The co-authorship signal is strongest at the top-10 level and weakens as the threshold broadens, while PhD rank becomes more useful for excluding candidates from less selective definitions of high-rank.","Progressive rewiring degrades performance, so the predictive content lies in higher-order co-authorship structure rather than degree alone."],"supporting_citations":[{"why":"Establishes the prestige-hierarchy finding in faculty hiring that motivates PhD rank as a baseline predictor.","marker":"[24]"},{"why":"Earlier predictive attempt using candidate attributes; supplies the descriptive-to-predictive contrast this paper extends.","marker":"[23]"},{"why":"Correlates co-authorship centrality with placement prestige in computer science, motivating the network-feature hypothesis.","marker":"[19]"},{"why":"Correlates co-authorship patterns with faculty placement in information schools, a prior network-placement result.","marker":"[10]"},{"why":"Supplies the department prestige ranks used for PhD rank features and for the placement labels.","marker":"[63]"},{"why":"Supplies the publication records from which co-authorship edges and bibliometric features are built.","marker":"[65]"},{"why":"Supplies the faculty roster with hire years and placement institutions that define the prediction cohort and labels.","marker":"[66]"},{"why":"The attention-based graph architecture that attains the best reported PR-AUC of 0.458.","marker":"[73]"},{"why":"The temporal graph model used to test whether network evolution across snapshots adds signal.","marker":"[75]"},{"why":"The graph convolutional architecture that tops performance at broader high-rank thresholds.","marker":"[72]"}],"fun_headline_variants":["Co-authorship networks reveal elite faculty hiring signals","Network structure predicts top-10 faculty placement better than rank","Co-author ties improve faculty hire predictions by 10%","PhD prestige less telling than collaboration patterns for elite hires","Graph neural nets forecast faculty placement from co-authorship"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The claim that co-authorship features generalize to unseen candidates rests on treating the 2018-2020 hires as an out-of-sample test, yet the graph models pass messages over the full network that includes those test candidates, so a truly unseen candidate might not show the same gains.","fun_headline_variants_meta":{"raw":{"variants":["Co-authorship networks reveal elite faculty hiring signals","Network structure predicts top-10 faculty placement better than rank","Co-author ties improve faculty hire predictions by 10%","PhD prestige less telling than collaboration patterns for elite hires","Graph neural nets forecast faculty placement from co-authorship"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000163,"raw_usage":{"total_tokens":1249,"prompt_tokens":958,"completion_tokens":291,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":574,"completion_tokens_details":{"reasoning_tokens":212}},"tokens_in":574,"tokens_out":291,"duration_ms":3667,"temperature":1.0,"reasoning_tokens":212,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T15:50:25.951868+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Train the leading model on hires from 2010-2016 and evaluate on hires from 2017-2019, rebuilding each year's co-authorship graph so that no test-year candidate or any edge incident to them appears in a training snapshot; if the co-authorship gain over PhD rank alone falls below the reported 7-10% and loses statistical significance, the central claim would be falsified.","supporting_citations":[{"cited_title":"& Larremore, D","cited_arxiv_id":null,"evidence_quote":"Establishes the prestige-hierarchy finding in faculty hiring that motivates PhD rank as a baseline predictor."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Correlates co-authorship patterns with faculty placement in information schools, a prior network-placement result."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the department prestige ranks used for PhD rank features and for the placement labels."},{"cited_title":"DBLP computer science bibliogra- phy","cited_arxiv_id":null,"evidence_quote":"Supplies the publication records from which co-authorship edges and bibliometric features are built."},{"cited_title":"Computer science open data","cited_arxiv_id":null,"evidence_quote":"Supplies the faculty roster with hire years and placement institutions that define the prediction cohort and labels."}],"review_version":1}