{"id":"19a6b884-9d84-4693-9e4b-f222c9a5813c","arxiv_id":"2509.02942","paper_version":1,"verdict":"CONDITIONAL","confidence":"LOW","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"RankGraph combines RGCN-style message passing, contrastive learning, and graph-token injection into foundation-model recommenders, reporting small online CTR and CVR gains.","lead":"RankGraph is a Meta system that builds a large heterogeneous graph of users, posts, and ads, runs a GPU-accelerated GNN with contrastive learning, and feeds the learned embeddings into sequence-based recommendation models as extra tokens. The paper reports modest online gains in click and conversion rates, but gives almost no details about the tests that produced them.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Offline recall evidence may be inflated by semantic edges that encode next-day interactions via multi-hop paths with no specified time window.","rationale":"The reader's verdict is CONDITIONAL, and my reading does not move it. The strongest claim — online A/B gains — is asserted without experimental protocol; that is a reporting gap, not by itself a demonstrated error. The most concrete, testable soft spot is the temporal integrity of the graph construction. Section 3.1.1 says embeddings from day t are evaluated on edges from day t+1, but Section 2.1 defines semantic edges as multi-hop indirect interactions without specifying any time boundary. If semantic edges are built from paths that include future edges, or if they include multi-hop paths whose transitive closure already contains a day-t+1 direct edge, the offline recall numbers in Tables 1 and 2 are inflated and cease to support the central claim. The paper's own admission in Section 3.1.2 that offline metrics did not match online A/B results, followed by the post hoc introduction of an engagement recall metric, makes this concern more acute. A clean experiment that restricts or removes semantic edges would settle the issue. Because the paper has plausible internal consistency and no demonstrated leakage, CONDITIONAL remains appropriate; no rejection or acceptance is warranted without the requested check.","tokens_in":4977,"tokens_out":6040,"duration_ms":75526,"concrete_test":"Re-run the §3.1.1 and §3.1.2 evaluations with two modified graph constructions: (a) semantic edges restricted to multi-hop paths whose constituent edges all have timestamps strictly earlier than the evaluation split (day t for Table 1, hour t for Table 2); (b) semantic edges removed entirely. If RankGraph's Recall@100 advantage over Filament2 collapses (e.g., from roughly 1.5x to near parity) in either condition, temporal leakage is confirmed and the central claim lacks offline support. If the advantage persists under both conditions, the concern is refuted.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central claim is the online A/B lift (+0.92% click, +2.82% conversion), but the only quantitative support provided is the offline recall tables in §3.1. The load-bearing assumption for those tables is temporal integrity: embeddings from day t must not have seen day t+1 edges. Section 2.1 defines semantic edges as 'indirect interactions through multi-hop neighbors' with no time window or construction rule. If the day-t graph's semantic edges are the transitive closure of engagement edges, then a direct edge (i,j) that appears at day t+1 may already be represented as a day-t multi-hop path, so top-k retrieval would recall it via graph closure rather than predictive generalization. Under that condition, Table 1 measures graph reconstruction, not future-edge prediction, and Table 2's engagement recall is similarly vulnerable if trigger-item neighbor lists include future-adjacent semantic edges. The reader's concern is sharpened by §3.1.2, which introduces a new engagement recall metric after noting a discrepancy with online A/B results. The paper never rules out leakage by stating that every edge used to build the day-t semantic graph has timestamp < t. This is not an accusation of intentional leakage; it is an unstated, checkable assumption on which the offline support depends. The A/B numbers alone are unverifiable as reported (no sample size, duration, CIs), so the offline tables are the only testable evidence.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper describes RankGraph, a heterogeneous graph learning framework for cross-domain recommendation at Meta. It constructs a multi-relation graph over users, posts, ads, and other entities, combining direct engagement edges with \"semantic\" multi-hop edges, then trains a GPU-accelerated RGCN with contrastive learning. The learned embeddings are used for retrieval and are also injected as tokens into a sequence-based foundation model. The central empirical claims are offline recall gains over the Filament2 baseline (Tables 1 and 2) and online A/B lifts of +0.92% in clicks and +2.82% in conversions, stated in the abstract. The paper is a short system description with no external datasets, no code, and no detailed experimental protocol.","tokens_in":5305,"tokens_out":4164,"duration_ms":51371,"significance":"If the claims hold, the work is practically significant: it demonstrates a scalable way to inject structured graph knowledge into a recommendation foundation model and to derive item-item/user-user subgraphs for retrieval. The described system components—GPU-accelerated RGCN, out-of-batch negative sampling, graph tokens for FMs—are plausible and useful. The offline next-day link-prediction design is not formally circular, and I credit the authors for evaluating on future edges rather than only on training edges. However, the paper's headline online A/B result is reported without any experimental detail, and the offline evaluation has a load-bearing, unstated temporal-integrity assumption. Moreover, the new engagement-recall metric was introduced after the authors observed that the original offline metric disagreed with online A/B results, which weakens its confirmatory value. As reported, the evidence is insufficient to support the abstract's quantitative claims.","major_comments":[{"comment":"The definition of \"semantic edges\" as \"indirect interactions through multi-hop neighbors\" is not accompanied by any time-window or construction rule. The offline evaluation uses embeddings generated on day t to recall edges that first appear on day t+1. If the day-t graph contains multi-hop paths connecting two nodes whose direct edge appears at day t+1, then the model can retrieve that future edge via graph closure rather than by predictive generalization. This could inflate Table 1 substantially. Please specify how each semantic edge is constructed, whether every edge (direct and semantic) is timestamped and strictly before t, and how pathways are truncated or decayed. The same concern applies to the engagement-recall metric in Section 3.1.2 if the neighbor lists or trigger-item graphs include future-adjacent nodes. Without this specification, the offline recall numbers do not establis","section":"Section 2.1; Section 3.1.1"},{"comment":"The paper introduces an \"engagement recall\" metric after acknowledging that the original offline recall metrics \"not necessarily present the power of the embeddings in predicting user’s future engagement\" and that there was a discrepancy with online A/B results. This is a post-hoc design choice: the metric is selected after seeing the online outcome, which introduces selection bias. Furthermore, the metric definition is incomplete: trigger weights, the number of triggers per user, how trigger-item recommendations are merged, how ground-truth interactions are matched, and how the hourly results are aggregated are not specified. As reported, Table 2 cannot be independently reconstructed or audited. Please provide a complete, pre-specified definition of the metric, justify it independently of the observed A/B outcome, and report day-to-day variance.","section":"Section 3.1.2"},{"comment":"No error bars, confidence intervals, or significance tests are reported. Table 1 is based on only 1000 sampled edges, and Table 2 is an average over a single day on a surface with billions of users. The reported differences between RankGraph and Filament2 may be real, but the current presentation does not rule out noise or sampling artifacts. Because the offline tables are the only quantitative evidence that can be inspected, the lack of variability estimates is a load-bearing weakness. Please report multiple independent samples (e.g., different edge samples, different days) with bootstrapped confidence intervals and, where possible, paired significance tests.","section":"Tables 1 and 2"},{"comment":"The abstract states that RankGraph \"has demonstrated improvements in click (+0.92%) and conversion rates (+2.82%) in online A/B tests,\" but the paper contains no section describing these A/B tests. There is no information about test duration, number of users, unit of analysis, confidence intervals, or whether the lifts are statistically significant. This is the paper's headline quantitative claim and it is currently unverifiable. Please either add an appendix with the complete A/B experiment design and results, or qualify the abstract to state the claim as directional and remove the specific percentages. Without this, the abstract overstates what the paper actually demonstrates.","section":"Abstract; Section 3"}],"minor_comments":[{"comment":"Typo: \"The graph consist of\" should be \"The graph consists of.\" Also, \"semantic edges\" are described only qualitatively; a small pseudocode or construction algorithm would help readability.","section":"Section 2.1"},{"comment":"The notation in the message-passing equation is unclear: the reuse of M_t as both a node-type feature mixer in 2.2.1 and in the aggregation layer is confusing, and the role of the self-loop described in the text is not visible in the equation. Please define all symbols and explicitly include self-loops.","section":"Section 2.2.2"},{"comment":"The baseline, Filament2, is not described or cited. At minimum, state whether it is a homogeneous graph method, whether it uses the same features, and give a reference or system description.","section":"Section 3"},{"comment":"A \"Speaker Bio\" section is unusual in a research paper. Consider removing it or moving it to a non-archival part of the submission.","section":"Section 5"}],"recommendation":"major_revision","confidential_remarks":"This is an industrial system paper with a potentially interesting architecture, but the evaluation is currently too weak to support the headline A/B numbers. The temporal-leakage concern in Section 2.1 is the most serious technical issue and is fixable by specifying the graph construction rules. The lack of A/B experimental detail is also fixable in principle, but the authors may not be able to retroactively provide the full experiment. I would not reject on novelty grounds—the system is plausible and the subgraph-extraction/FM integration is worth publishing—but the paper needs either substantially stronger offline evidence or a realistically framed abstract. The referee report should emphasize the need for time-stamped semantic edges and statistical rigor."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Bottom line: this is an industry short paper about a real deployed graph-learning component at Meta. The architecture is an integration of RGCN-style message passing, triplet + InfoNCE contrastive learning, and graph tokens fed into sequence FMs. No single component is new; the integration at this scale and the reported deployment are what would make it interesting. There is no serious academic novelty, but there can be industrial value in a clear system description.\n\nCredit where due: the paper is honest about the offline/online discrepancy and says it introduced a new engagement-recall metric in response (Section 3.1.2). That is the right instinct, even if the post-hoc metric needs scrutiny. The offline tables show RankGraph roughly 1.5-2.8x the baseline in recall, which is a real signal if the evaluation is clean.\n\nThe soft spots are in the evidence, not the writing. The abstract claims +0.92% CTR and +2.82% conversion from online A/B tests, but there is no section on duration, sample size, confidence intervals, or whether the test was on the full surface. As far as I can tell, the only testable evidence is the offline recall, which has no error bars and is built on 1000 sampled edges in Table 1. The bigger issue is the stress-test concern: semantic edges are defined as \"indirect interactions through multi-hop neighbors\" with no timestamp constraint. If the day-t semantic graph is the transitive closure of engagement edges, then a day-t+1 edge could already appear as a multi-hop path at day t, and Table 1 would measure graph reconstruction rather than predictive recall. The paper never states that every edge used to build the day-t graph has timestamp < t. That is a checkable assumption, and it is load-bearing. I don't see proof of leakage, but the paper needs to rule it out explicitly.\n\nAlso note: the engagement recall metric in Section 3.1.2 uses hour-t embeddings and hour t+1 to t+4 ground truth, which is cleaner, but the trigger-item neighbor lists could still include future-adjacent semantic edges if the graph update isn't timestamped. Same fix.\n\nWho is this for? Read it if you want to see how Meta combines graph learning with sequence models in production. It is not a methods paper, and it does not deserve to be judged as one. But it deserves a serious referee because the claimed online gains are important if real, and the reviewer can ask for the missing methodology and a leakage guarantee.\n\nRecommendation: send to peer review if the venue accepts industry short papers; otherwise the right verdict is 'not enough evidence,' not 'wrong.'","headline":"Deployed system, thin evidence: the A/B lift is unverifiable as reported, and the offline recall depends on an unstated no-leakage assumption.","tokens_in":5793,"tokens_out":1884,"would_cite":false,"duration_ms":20956,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A heterogeneous graph spanning users, posts, and ads, trained with relation-specific message passing and contrastive learning, can serve as the relational backbone of a cross-domain recommendation foundation model and lift online engagement","keywords":["cross-domain recommendation","heterogeneous graph learning","graph neural network","contrastive learning","foundation models","retrieval","graph embeddings","online A/B testing"],"falsifier":"Run the offline evaluation with a strict temporal split: construct every edge, including the semantic multi-hop edges, only from interactions timestamped before t, then measure recall@k on edges that first appear between t and t+1. If RankGraph's recall advantage over its baseline shrinks to near zero, label leakage from next-day edges explains the reported gains. The paper's current specification does not rule out that outcome.","tokens_in":4882,"feed_emoji":"📈","tokens_out":9271,"duration_ms":101916,"temperature":0.7,"pith_summary":"The paper sets out to show that one heterogeneous graph spanning users, posts, ads, and other entities can serve as the relational core of a cross-domain recommendation foundation model. RankGraph builds that graph from weighted engagement signals and 'semantic edges' that encode multi-hop relationships, then learns node embeddings with a GPU-accelerated, relation-aware graph neural network trained by contrastive learning. The embeddings feed real-time similar-item retrieval and are injected as extra tokens into sequence-based ranking models. The payoff claim is concrete: click-through rose 0.92 percent and conversions 2.82 percent in online A/B tests, and offline recall roughly doubled to tripled compared with the production baseline. If those results hold, cross-domain recommenders can gain measurable engagement by adding a shared graph layer rather than redesigning their sequence model.","feed_headline":"Graph layer lifts conversions 2.82% in live recommendation tests","feed_subtitle":"A heterogeneous graph embedding layer raises clicks 0.92% and conversions 2.82% in production A/B tests.","key_machinery":"The carrying mechanism is a typed, relation-aware message-passing update on a heterogeneous graph. Each node type has its own feature encoder and mixer that project disparate feature spaces into one embedding space; for each relation, neighbor messages are aggregated with relation-specific weight matrices and a normalization factor; self-loops preserve the node's own features. Contrastive learning with a combined triplet and InfoNCE loss shapes the embedding space, while GPU-side negative sampling across three strategies and on-demand subgraph extraction make the graph usable for real-time retrieval and clustering.","core_discovery":"The central claim is that a production-scale heterogeneous graph can act as a cross-domain representation layer for recommendation foundation models. The model's update rule projects each node type's heterogeneous features into a common space, aggregates messages along typed relations with relation-specific weights, and is trained by a contrastive loss that combines local triplet separation with global InfoNCE separation. The learned embeddings are used both for real-time nearest-neighbor retrieval and, as contextual tokens, inside a sequence-based foundation model. The paper reports that on 1000 sampled next-day edges the method reaches 0.143 recall@5 and 0.614 recall@100 versus 0.051 and 0","pith_inferences":["Separating the 'semantic edges' contribution from direct engagement edges is the natural next experiment; an ablation without multi-hop semantic edges would show how much of the gain comes from that specific construction.","Because the graph embeddings are consumed as tokens, the same encoder could be attached to any sequence-based recommender; the paper does not test whether the reported gains transfer to a non-graph backbone.","A public-data replication, even on a smaller surface, would clarify how much of the offline recall advantage depends on graph scale; the current evidence is tied to the production deployment.","The online lifts look small in percentage but are measured on a billion-user surface, so the absolute user impact is material; the paper does not report absolute interaction counts."],"forward_implications":["A graph pretraining layer can plug into a sequence-based foundation model as ordinary input tokens, giving the sequence model relational structure without changing its training loop.","The same heterogeneous graph can be sliced into item-item and user-user subgraphs on demand, enabling similarity retrieval and clustering in real time.","The engagement recall protocol ties offline evaluation to future user engagement, narrowing the gap between offline metrics and online A/B outcomes.","If the reported lifts generalize, cross-domain recommenders can share one graph encoder across surfaces instead of training a separate graph per domain."],"supporting_citations":[{"why":"Supplies the relation-specific message-passing formulation that RankGraph adapts.","marker":"[6]"},{"why":"Defines the InfoNCE contrastive objective used for global positive/negative separation.","marker":"[4]"},{"why":"Provides the theoretical result that InfoNCE learns cluster-preserving representations.","marker":"[5]"},{"why":"Supplies the triplet loss used for local positive/negative separation.","marker":"[7]"},{"why":"Represents the sequence-based foundation model architecture RankGraph's graph tokens augment.","marker":"[11]"},{"why":"Motivates graph foundation models for personalization, the direction RankGraph operationalizes.","marker":"[1]"}],"fun_headline_variants":["Cross-domain graph lifts conversions 2.82% in live A/B tests","Heterogeneous graph learning boosts clicks and conversions in production","RankGraph: Graph layer improves cross-domain recommendations, clicks up 0.92%","Graph embeddings raise conversion 2.82%, clicks 0.92% in production tests","Graph framework for cross-domain recommendation raises conversions 2.82%"],"cache_read_input_tokens":2688,"weakest_assumption_plain":"The offline recall results assume that the graph used to produce day-t embeddings contains no edges from day t+1; the paper never specifies a time window for its 'semantic edges', so next-day edges could leak into the training graph and inflate the recall numbers.","fun_headline_variants_meta":{"raw":{"variants":["Cross-domain graph lifts conversions 2.82% in live A/B tests","Heterogeneous graph learning boosts clicks and conversions in production","RankGraph: Graph layer improves cross-domain recommendations, clicks up 0.92%","Graph embeddings raise conversion 2.82%, clicks 0.92% in production tests","Graph framework for cross-domain recommendation raises conversions 2.82%"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000509,"raw_usage":{"total_tokens":2284,"prompt_tokens":680,"completion_tokens":1604,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":424,"completion_tokens_details":{"reasoning_tokens":1504}},"tokens_in":424,"tokens_out":1604,"duration_ms":13641,"temperature":1.0,"reasoning_tokens":1504,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-05T11:14:36.276934+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the offline evaluation with a strict temporal split: construct every edge, including the semantic multi-hop edges, only from interactions timestamped before t, then measure recall@k on edges that first appear between t and t+1. If RankGraph's recall advantage over its baseline shrinks to near zero, label leakage from next-day edges explains the reported gains. The paper's current specification does not rule out that outcome.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the theoretical result that InfoNCE learns cluster-preserving representations."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Motivates graph foundation models for personalization, the direction RankGraph operationalizes."}],"review_version":1}