{"id":"e49ec386-4cb2-41cd-8cd5-e9ac4c41eefe","arxiv_id":"2608.10187","paper_version":1,"verdict":"CONDITIONAL","confidence":"LOW","novelty_score":5.0,"correctness_risk":"high","formal_verification":"none","parameter_count":4,"one_line_summary":"ConnectionMind turns social recommendation into graph path discovery with an LLM policy trained by supervised fine-tuning and reinforcement learning, and reports a 0.43% watch-time gain in a Meta A/B test.","lead":"ConnectionMind is a Meta recommendation system that uses a large language model to walk paths through a social graph of friends, groups, creators, and videos, and to recommend videos found at the end of those paths. The authors report gains on public benchmarks and a 0.43% increase in video watch time in a production A/B test at Meta.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Production A/B lift is measured on a hybrid teacher–student system where a distilled Student GNN serves most traffic; without stratification or teacher–student agreement, the gain cannot be attributed to the LLM reasoning policy.","rationale":"I read the paper in good faith. The architecture is coherent, public experiments show consistent improvements, and the production deployment is described with plausible latency engineering. The single most load-bearing point is the attribution of the online A/B gain to the LLM reasoning policy. Section 6.1 explicitly states that most traffic is served by a distilled Student GNN. The paper provides no disaggregated A/B results and no teacher-student fidelity measurement. This is not an external-consensus disagreement; it is an internal evidential gap: the reported metric cannot be assigned to the component the paper is about. The reader's weakest assumption identified exactly this, and I agree. Other potential concerns (e.g., unavailability of code/data, qualitative production observations) are secondary. A conditional acceptance with a request for the stratified analysis and distillation agreement is appropriate; if those are unavailable, the production claim should be treated as unverified.","tokens_in":12789,"tokens_out":4495,"duration_ms":43671,"concrete_test":"Re-analyze the production A/B logs to report the watch-time lift separately for the heavy-user cohort served by the full LLM and for the standard cohort served by the Student GNN, and compute the Student GNN's agreement with teacher-generated paths on a held-out sample of standard users (e.g., top-10 item overlap or path-following recall). If the standard-cohort lift is not significantly positive or the student–teacher agreement is low, the reported gain should not be attributed to LLM reasoning.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The production evidence underpinning the abstract's deployment claim comes from an A/B test of the hybrid teacher–student system described in §6.1. There, the full LLM policy is routed only to the top 5–10% of active users, while the remaining majority is served by a Student GNN trained offline via distillation. Table 2 reports a single set of lifts for the combined system, with no breakdown by serving path and no measure of how faithfully the student emulates the teacher. Because the student handles most traffic, the measured +0.43% watch time could be driven by the student's learned graph features, by the heavy-user cohort, or by unrelated production changes, rather than by the LLM's multi-hop reasoning. The paper's central scientific claim—that LLM-guided graph reasoning improves recommendation at scale—is therefore not actually tested on the majority of users. Without teacher–student agreement statistics (e.g., path overlap or ranking agreement) and without cohort-stratified A/B results, the production result cannot be attributed to the reasoning policy. This is load-bearing because the abstract and conclusion generalize the result to 'ConnectionMind' and 'reasoning-based graph exploration.'","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"ConnectionMind casts social recommendation as typed path discovery over a heterogeneous graph of users, pages, groups, and items, and uses an LLM policy trained by SFT followed by RL with rule-based rewards to select evidence paths. The paper reports strong public-dataset results on Delicious and Foursquare, and claims production deployment at Meta with a hybrid inference scheme in which the full LLM serves only the top 5–10% of active users and a distilled Student GNN serves the remaining traffic, reporting online A/B lifts in exposure, watch time, and video sessions. The manuscript is clearly written and the training pipeline is described in enough detail to be reproduced on the public datasets, but the central production claim is not yet fully supported because the online A/B test evaluates the hybrid system rather than the LLM reasoning policy in isolation, and the public experiments lack statistical reporting.","tokens_in":13081,"tokens_out":6019,"duration_ms":59042,"significance":"If the attribution gap is closed, this would be a notable systems contribution: it demonstrates a scalable way to inject relational, interpretable path evidence into industrial recommendation, with a clearly described two-stage SFT+RL training recipe and a plausible teacher–student deployment pattern. The graph schema and the path-discovery formulation are useful for follow-up work. However, as presented, the paper does not yet establish that LLM-guided graph reasoning, rather than the distilled student or the heavy-user serving path, is responsible for the measured production gains. The public experimental reporting also lacks variance estimates, so the claimed 'clear margin' over strong social-recommendation baselines is not yet rigorously established. The contribution at this stage is a well-specified framework with preliminary evidence, not a fully validated production result.","major_comments":[{"comment":"The online A/B test evaluates the full hybrid teacher–student system, in which only the top 5–10% of active users receive direct LLM reasoning and the remaining majority is served by a Student GNN trained via offline distillation. The paper reports a single set of lifts for the combined deployment and does not provide teacher–student agreement statistics (e.g., path overlap or ranking agreement) or cohort-stratified A/B results. Because the student handles most traffic, the +0.43% watch-time gain could be driven by the student's graph features, by the heavy-user cohort, or by unrelated production changes, and the abstract and conclusion attribute the gain to ConnectionMind's reasoning policy. This is load-bearing for the paper's central claim; please report teacher–student agreement and A/B lifts separately for the two serving paths, or substantially soften the claim to a system-level result.","section":"§6.1, Table 2"},{"comment":"The public benchmark comparison reports a single number per method and metric, without standard deviations, confidence intervals, or significance tests, and no information is given about the number of random seeds or the tuning protocol for the baselines. The claim of a 'clear margin' over MHCN and SEPT cannot be assessed from point estimates alone, especially since the absolute Recall@5 values are very low (e.g., 0.0631 on Delicious). Please add repeated-run statistics, state the number of seeds, and describe the hyperparameter selection procedure for all baselines.","section":"§5.2, Table 1"},{"comment":"The offline production evaluation is underspecified: the 'Production GNN Baseline' is never named or described, and there is no definition of the offline evaluation set, the ground-truth labels, the candidate pool, or the way Recall@10 is computed in the production offline setting. The +88% relative lift is therefore not interpretable. Please name the baseline, describe the evaluation methodology, and report the underlying absolute metric values rather than only relative lifts.","section":"§6.2, Table 2"},{"comment":"The three 'consistent patterns' from production-scale interaction logs are reported only qualitatively, with the authors explicitly stating that the findings are not shown due to privacy and platform policy constraints. While this constraint is understandable, these observations are used as the primary motivation for the graph and path-based design. Please provide sanitized summary statistics or an explicit internal reference, or clearly mark these observations as unverifiable motivating hypotheses rather than as established empirical findings.","section":"§3"}],"minor_comments":[{"comment":"The y-axis is labeled 'Recall' even though the figure plots Recall@5 for Delicious and Recall@10 for Foursquare in the same panel; the axis and caption should state this explicitly, and error bars should be added.","section":"Figure 3"},{"comment":"Dataset statistics (number of users, items, relations, and density) and the exact train/validation/test split procedure are not reported; please add them, and clarify whether the split respects temporal order given that the graph edges are time-stamped.","section":"§5.1"},{"comment":"The confidence intervals are reported without p-values, sample sizes, or a description of the statistical test, so the phrase 'statistically significant' is not substantiated; please specify the test and the randomization unit.","section":"§6.2, Table 2"},{"comment":"The step-wise reward uses shortest paths to reachable positives in the sampled subgraph; this privileged information should be discussed as a potential bias, since the policy's behavior may then depend heavily on the subgraph sampler and on shortest-path coverage rather than on learned reasoning.","section":"§4.3"},{"comment":"The phrase 'deployed in Meta's large-scale recommendation pipeline' should be qualified in both the abstract and the conclusion, since the full LLM policy serves only a small slice of traffic; consider saying 'deployed in a hybrid teacher–student configuration.'","section":"Abstract and Conclusion"},{"comment":"The related-work section is adequate but does not discuss recent path-based or reasoning-based recommendation methods beyond metapath and GNN aggregation; adding a few recent references would sharpen the positioning of the contribution.","section":"§2.2"}],"recommendation":"major_revision","confidential_remarks":"I see no evidence of misconduct, and the paper has genuine potential as a systems contribution. My main concern is that the production attribution claim is not yet supported by the reported evidence and may be difficult to verify from outside Meta; the public benchmark results also need variance reporting. I would ask the editor to require the authors to either supply teacher–student agreement and cohort-stratified A/B numbers, or explicitly narrow the production claim to the hybrid system. This is a significant but fixable issue, so I do not think rejection is warranted at this stage."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The paper is a serious system description, and the architecture is coherent. What's new is the full recipe: a heterogeneous social–item graph, an LLM policy trained with SFT plus RL, and a teacher–student distillation step that makes the reasoning approach cheap enough for production. The public benchmark results, while lacking error bars and significance tests, show a large relative margin over strong GNN baselines, which suggests the method is doing something real. The hybrid inference design is itself a practical contribution.\n\nThe soft spot is load-bearing: the production A/B test is run on the hybrid system, where the full LLM serves only the top 5–10% of users and a distilled Student GNN serves the rest. The reported +0.43% watch-time lift is aggregated over both paths, with no teacher–student agreement metric and no cohort-stratified breakdown. So the gain could plausibly come from the student's graph features or from the heavy-user cohort, not from the LLM's reasoning. The abstract and conclusion overstate by attributing the result to \"reasoning-based graph exploration.\" That's a real overreach, but fixable: add path-overlap or ranking-agreement statistics between teacher and student, and report A/B lifts by serving path.\n\nOther issues are more minor: no code or data, an unnamed production baseline, and the three qualitative production observations are asserted without numbers. These are common in industry papers and limit independent verification but don't disqualify the work.\n\nOverall, this paper deserves serious refereeing. The architecture is plausible, the public experiments are internally consistent, and the deployment is real. The central weakness is the evidence chain from the A/B test to the LLM reasoning claim. A major revision should address that directly. I would send it to peer review with referees who will push hard on the attribution question.","headline":"A credible system paper with a coherent architecture and a real production deployment, but the A/B evidence doesn't actually show that LLM reasoning drives the gains, because the distilled student GNN serves most traffic.","tokens_in":651,"tokens_out":1004,"would_cite":true,"duration_ms":46394,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"ConnectionMind claims that on social platforms, recommendation should be treated as LLM-guided path discovery over a typed user–item–social graph, and that this approach is production-viable and measurably improves engagement.","keywords":["social recommendation","heterogeneous graph","path reasoning","large language models","reinforcement learning","teacher-student distillation","production deployment","graph neural networks"],"falsifier":"Re-run the online A/B test with results stratified by serving path, or measure teacher–student agreement on held-out subgraphs. If the watch-time lift is concentrated in the heavy-user segment served directly by the LLM and the standard-user segment served by the distilled GNN shows no significant lift, the claim that LLM-guided reasoning improves recommendation for the general population is not supported.","tokens_in":12592,"feed_emoji":"📈","tokens_out":8943,"duration_ms":77404,"temperature":0.7,"pith_summary":"ConnectionMind sets out to prove that social evidence in recommendation does not need to be squeezed into a single learned embedding: it can be kept as explicit paths from a user to an item, such as user → friend → liked video or user → group → shared video. The paper constructs a heterogeneous graph of users, pages, groups, and items, and trains an LLM policy to walk that graph and surface candidates with their evidence chains. The policy is first taught to imitate reference paths from logged interactions, then sharpened by reinforcement learning with format, recommendation-quality, and step-wise rewards. On the Delicious and Foursquare benchmarks it beats GNN-based social recommenders and an LLM baseline, and in production it reports +88% relative Recall@10 offline and +0.43% watch time, +0.33% exposure, and +0.22% video sessions online. The paper is establishing that selective, interpretable social reasoning can be deployed at billion-user scale.","feed_headline":"LLM-guided social paths lift watch time 0.43% in production A/B test","feed_subtitle":"Explicit friend, group, and creator paths beat GNN and vanilla-LLM baselines; distillation makes it production-fast.","key_machinery":"The load-bearing mechanism is the typed temporal heterogeneous graph $G_T=(V,E,R)$, with $V=U\\cup P\\cup I$, together with an LLM exploration policy that consumes sampled subgraphs and outputs structured expansions and surfaced item–path pairs. The graph makes every social or semantic relationship an explicit, weighted, time-stamped edge, turning 'which friends or groups matter for this user' into a search problem rather than an aggregation problem. The two-stage training—SFT on reference shortest paths, then GRPO over rule-based rewards combining format validity, final-item F1, and step-wise shaping—supplies the policy with both structural validity and recommendation utility, while the teacher–student hybrid inference is what reconciles autoregressive reasoning with millisecond-level serving latency.","core_discovery":"The central claim is that recommendation can be reformulated as query-conditioned path exploration over a typed heterogeneous graph, with the path itself serving as both the scoring evidence and the explanation. ConnectionMind defines a graph $G_T=(V,E,R)$ whose nodes are users, pages/creators, and items, and whose typed edges encode friendships, follows, group membership, shares, posts, co-watches, and semantic similarity. An LLM policy expands partial paths one typed hop at a time and surfaces item–path pairs; supervised fine-tuning on logged engagement trajectories gives it graph-navigation competence, and end-to-end reinforcement learning with rule-based rewards optimizes the full rollout. The paper reports that this system outperforms GNN-based, diffusion-based, and LLM baselines on public datasets and, in the platform's production pipeline, an 88% relative offline Recall@10 gain over the production GNN baseline, with online A/B lifts of +0.33% exposure, +0.43% watch time, and +0.22% video sessions. The production version splits traffic: the full LLM serves only the top 5–10% of active users, while a distilled GNN emulates the teacher's path selection for everyone else.","pith_inferences":["If path-based selectivity is the true driver of the gains, then teacher–student agreement—not LLM size—becomes the binding constraint on production quality; improving distillation fidelity is the most direct next step.","The same 'typed graph + LLM policy + distillation' recipe could transfer to other relational domains, such as job, commerce, or health recommendations, wherever a decision can be justified by an explicit chain of relationships.","The paper does not test whether surfacing the evidence paths to users changes trust, retention, or perceived relevance; an A/B test that shows or hides the path would isolate that effect.","Routing by user activity level may be a proxy; a router that serves direct LLM reasoning based on subgraph complexity or expected path value could allocate reasoning budget more precisely."],"forward_implications":["Off-the-shelf LLMs cannot perform this kind of recommendation out of the box: the untrained backbone achieves near-zero recall, so task-specific post-training is a prerequisite.","Bigger reasoning capacity helps: the 8B policy consistently outperforms the 3B policy on both public benchmarks.","Semantic understanding alone is not enough: a vanilla 70B LLM ranker gains +39% offline Recall@10, while graph-reasoning ConnectionMind gains +88%, showing that structure adds signal beyond language.","The reasoning benefits can be scaled to latency-critical traffic by distilling the LLM teacher's chosen paths into a lightweight GNN, with the full LLM reserved for the most active users.","Recommendations come with explicit evidence paths, which makes social influence traceable for debugging and user-facing explanation."],"supporting_citations":[{"why":"LLM-based recommender baseline without social information; its weaker results support the claim that semantic modeling alone is insufficient.","marker":"[1]"},{"why":"Delicious benchmark dataset used in the public experiments.","marker":"[2]"},{"why":"The backbone models used for the policy and for the vanilla-LLM ranker comparison.","marker":"[5]"},{"why":"GraphRec baseline representing GNN-based social recommendation that the method must beat.","marker":"[6]"},{"why":"Source of GRPO, the reinforcement-learning optimizer used in Stage II training.","marker":"[8]"},{"why":"Video-understanding LLM survey behind the VidLLM component that turns videos into compact textual representations.","marker":"[27]"},{"why":"Foursquare dataset source used for the second public benchmark.","marker":"[34]"},{"why":"MHCN baseline, a strong self-supervised hypergraph social recommender used in comparisons.","marker":"[38]"}],"fun_headline_variants":["0.43% watch-time lift from LLM reasoning over social graphs","Meta's LLM-social graph recommender boosts watch time by 0.43%","0.43% watch-time gain: LLM navigates social graph for recommendations","Reasoning over social ties with LLMs lifts Meta watch time 0.43%","Meta's LLM path-reasoning beats GNN, lifts watch time 0.43%"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"In production, the reported gains mostly come from a lightweight student GNN that imitates the LLM teacher, because the full LLM is served only to the top 5–10% of active users; the whole result stands on the assumption that the student faithfully reproduces the teacher's path-finding behavior, which the paper never measures.","fun_headline_variants_meta":{"raw":{"variants":["0.43% watch-time lift from LLM reasoning over social graphs","Meta's LLM-social graph recommender boosts watch time by 0.43%","0.43% watch-time gain: LLM navigates social graph for recommendations","Reasoning over social ties with LLMs lifts Meta watch time 0.43%","Meta's LLM path-reasoning beats GNN, lifts watch time 0.43%"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001015,"raw_usage":{"total_tokens":4352,"prompt_tokens":1077,"completion_tokens":3275,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":693,"completion_tokens_details":{"reasoning_tokens":3165}},"tokens_in":693,"tokens_out":3275,"duration_ms":22448,"temperature":1.0,"reasoning_tokens":3165,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-14T04:11:17.801200+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run the online A/B test with results stratified by serving path, or measure teacher–student agreement on held-out subgraphs. If the watch-time lift is concentrated in the heavy-user segment served directly by the LLM and the standard-user segment served by the distilled GNN shows no significant lift, the claim that LLM-guided reasoning improves recommendation for the general population is not supported.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Delicious benchmark dataset used in the public experiments."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Foursquare dataset source used for the second public benchmark."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"MHCN baseline, a strong self-supervised hypergraph social recommender used in comparisons."}],"review_version":1}