{"id":"0d74d90f-91e7-4bbb-88c2-936bbd911af8","arxiv_id":"2505.13526","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"GA-LLM augments a fine-tuned LLM with quadkey-Fourier geographic encoding and a projection of precomputed POI embeddings, beating baselines on next-POI Acc@1 on NYC, Tokyo, and California datasets.","lead":"The paper augments a large language model with map-tile and trajectory-pattern modules to predict users' next check-in location, reporting higher top-1 accuracy than prior methods on three city datasets. It is relevant to location-based services because it shows how to make LLMs handle GPS coordinates and learned POI-to-POI transitions more efficiently.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Potential label leakage from precomputed MTNet POI embeddings: the paper never states these embeddings are trained only on the 80% training split.","rationale":"The reader identified the same weakest assumption, and I agree it is the single most load-bearing concern. The paper's evaluation has other omissions, such as no error bars and no ranking metrics, but those would weaken confidence rather than overturn the central claim. The leakage concern directly threatens the headline accuracy numbers because PAM's input comes from external POI embeddings whose training data is unspecified. If those embeddings are trained on the full dataset, the test ground truths can influence the LLM's predictions through transition-pattern leakage, and the reported 16.19%, 14.73%, and 16.69% improvements over the best baselines would not be trustworthy. The concern is testable: the anonymous code link is provided, and a careful inspection of the preprocessing script plus a retraining experiment would settle it. I agree with the reader's CONDITIONAL verdict; my review does not move it because the concern is the one already identified, and the proposed check could convert the conditional acceptance into either a clear acceptance or a rejection. I credit the paper for providing a code link, a reasonably detailed architecture description, and consistency in the ablation direction; none of these, however, removes the need for explicit split hygiene around the external embeddings.","tokens_in":12887,"tokens_out":2812,"duration_ms":31571,"concrete_test":"Inspect the released code to determine the split point used for MTNet/PAM pre-training, and rerun the pipeline with MTNet embeddings trained strictly on the 80% temporal training split (excluding validation and test check-ins). Then re-evaluate Acc@1 on NYC, TKY, and CA. If Acc@1 drops by more than 2-3 absolute points from 0.3919/0.3482/0.2566, the reported gains are at least partly leakage-driven.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 4.3 and Figure 2 specify that PAM consumes POI embeddings produced by a separate sequential model (e.g., MTNet), and Section 5.5 states the experiments use MTNet. Section 5.1 describes dataset filtering and a chronological 80/10/10 split, but it never states which split is used to train MTNet or to pre-train GCIM/PAM (the latter is referred to in Section 5.6). If MTNet is fit on all check-ins before splitting, its POI embeddings encode transition statistics involving test check-ins; PAM then feeds those embeddings as token-level inputs to the fine-tuned LLM. That would leak information about ground-truth next-POI transitions into the model, inflating the Acc@1 numbers in Table 2 and the ablations in Table 3. This assumption is load-bearing because the central claim is state-of-the-art accuracy. The paper offers no safeguard statement, no leakage check, and no description of how the external embeddings were split. I am not alleging misconduct; this is a missing procedural guarantee that determines whether the headline numbers mean what they appear to mean.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes GA-LLM, a framework that augments a fine-tuned LLM (Llama-2-7b-longlora-32k) with two components for next-POI recommendation: the Geographic Coordinate Injection Module (GCIM), which encodes GPS coordinates via quadkey-based hierarchical discretization, n-gram/self-attention, and learnable Fourier features, and the POI Alignment Module (PAM), which projects POI embeddings from a sequential model (e.g., MTNet) into the LLM semantic space. Experiments on Foursquare-NYC, Foursquare-TKY, and Gowalla-CA report state-of-the-art Acc@1 (0.3919, 0.3482, 0.2566), with ablations showing that both GCIM and PAM contribute to performance and that the approach improves cold-start and cross-city settings relative to LLM4POI. The paper also reports efficiency gains from compact spatial encoding.","tokens_in":13134,"tokens_out":3556,"duration_ms":34303,"significance":"If the reported results hold, GA-LLM would be a meaningful advance in applying LLMs to spatially grounded recommendation: it addresses a real limitation of text-only LLMs (poor spatial reasoning and hallucinated distant POIs) with two well-motivated, plug-in components, and it demonstrates consistent gains over strong sequential and graph baselines across three datasets. The ablation study is internally consistent, and the authors make the code available through an anonymous link. The main value lies in showing that explicit geographic encoding and POI-transition alignment can be injected into an LLM without representing every coordinate as verbose text, which is a practical contribution to the LLM-for-recommendation literature. The significance is conditioned, however, on resolving a critical data-split question: if the precomputed POI embeddings used by PAM were trained on the test portion of the check-in data, the reported accuracy gains would be inflated and the central claim would not stand.","major_comments":[{"comment":"The paper never states whether the MTNet POI embeddings consumed by PAM were trained only on the chronological 80% training split or on the full dataset. Section 4.3 and Section 5.5 describe PAM as using embeddings from sequential models such as MTNet, and Section 5.5 says the experiments use MTNet, but Section 5.1 only describes the 80/10/10 split for the LLM fine-tuning data. If MTNet was fit on all check-ins before the split, transition statistics involving test POIs would leak into the LLM through the PAM input, inflating Table 2 and Table 3. Please state the split used to train MTNet (and any GCIM/PAM pretraining mentioned in Section 5.6), retrain the embeddings on the training split only, or provide a leakage check such as comparing against randomly permuted POI embeddings.","section":"§4.3, §5.1, §5.5"},{"comment":"The cross-city cold-start protocol is not explained. Table 4's header is ambiguous (it reads 'Model Trained on NYC Acc@1 TKY Acc@1 CA Acc@1'), and the text does not say whether rows or columns index the training city, how POI IDs and categories are matched between cities with disjoint POI vocabularies (4,981 vs. 7,833 vs. 9,690), or what 'trained on TKY and tested on NYC' means for the GA-LLM w/o PAM variant. Without a precise protocol, the cross-city generalization claim in Section 5.4 is not verifiable from the table.","section":"§5.4, Table 4"},{"comment":"Table 2 reports only a single Acc@1 value per dataset with no runs, variance, or error bars, and the significance statement ('p-value ≤ 0.05') omits the test used and the number of repetitions. Additionally, SeCor, LLMmove, and LLaRA are discussed in Section 2.2 as relevant LLM-based recommenders, but none appears in the comparison. Please either include these baselines or justify their exclusion from the main table; the current evaluation does not fully support the claim of state-of-the-art performance among LLM-based methods.","section":"§5.1, §5.2"},{"comment":"The efficiency comparison in Section 5.6 is not controlled: LLM4POI is stated to use 200 check-ins per query while GA-LLM uses fewer, so the reported shorter fine-tuning and inference times could simply reflect shorter input sequences rather than the efficiency of the proposed modules. Also, the statement in Section 5.6 that 'excluding pre-training for GCIM and PAM' is excluded from timing appears to conflict with Section 5.5, where PAM is described as a projector trained during LLM fine-tuning. Please clarify what pretraining is performed for GCIM/PAM, on which data split, and report token counts or input lengths per method alongside timing.","section":"§5.5, §5.6, Figure 7"}],"minor_comments":[{"comment":"The text contains typos: 'Dose GA-LLM outperforms' should read 'Does GA-LLM outperform', and 'which also exploiting an LLM like GA-LLM dose' should read 'which also exploits an LLM like GA-LLM does'.","section":"§5.1"},{"comment":"The baseline list in the text includes MTNet [Huang et al., 2024], but Table 2 has a row labeled 'MTNext'; please unify the naming.","section":"Table 2"},{"comment":"The legend in Figure 3 uses 'GAM-LLM text-geo' and 'GAM-LLM w/o PAM'; the acronym should be 'GA-LLM' consistently.","section":"Figure 3"},{"comment":"The caption and text are inconsistent: the caption mentions 'LLM4POI and GA-LLM w/o GCIM' while the surrounding text discusses PAM's effect on predictions where the target POI is absent. Please align the description with the actual comparison.","section":"Figure 6"},{"comment":"The row/column orientation of Table 4 should be made explicit (e.g., rows as training city, columns as test city) and the table should note whether the numbers are Acc@1 values for the same model or for different model variants.","section":"§5.4, Table 4"}],"recommendation":"major_revision","confidential_remarks":"The central issue is the missing statement on the training/test split for the precomputed MTNet POI embeddings. This is a load-bearing reproducibility and validity concern, not a question of author intent: without an explicit guarantee that MTNet (and any GCIM/PAM pretraining) saw only the training split, the headline Acc@1 gains cannot be interpreted. I recommend asking for a leakage check or a retraining protocol. The paper otherwise presents a coherent architecture and internally consistent ablations, and the topic fits the journal's scope. The omission of cited LLM baselines (SeCor, LLMmove, LLaRA) and the lack of error bars should also be addressed in revision."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Over coffee: GA-LLM is a solid, well-written empirical paper that shows a clean way to inject geographic and POI-transition signal into an LLM for next-POI recommendation. The gains over LLM4POI are large (24% relative on CA) and the ablations behave sensibly. But there's a load-bearing loose end: the PAM module consumes POI embeddings from MTNet, and the paper never states that MTNet was trained only on the 80% training split. If those embeddings saw the test check-ins, the Acc@1 numbers in Tables 2 and 3 are inflated. That's not an accusation of misconduct—just a missing procedural guarantee that determines whether the headline means what it looks like. The reader's stress-test note is on point.\n\nWhat's actually new: the specific combination of quadkey encoding (from GeoSAN), learnable Fourier features, and a projection layer that aligns pre-trained POI embeddings into the LLM's semantic space. Each part is borrowed, but the integration into a single fine-tuning pipeline is a useful contribution, and the paper does a good job showing that spatial encoding reduces hallucinated distances (Figures 4-5) and that cross-city generalization works better than text-only LLMs. The efficiency analysis is also a nice addition.\n\nWhere the soft spots are: the leakage risk is the big one, and it's easy to fix—just state that the embedding model was fit on training only, and ideally run a check recomputing the POI embeddings on the training split. Second, the evaluation reports only Acc@1, with no error bars or repeated runs; given how close some ablations are (e.g., w/o Fourier on TKY: 0.3435 vs 0.3482), that's a real weakness. Third, several LLM baselines cited in related work (SeCor, LLaRA, LLMmove) are absent from Table 2; the authors justify only LLM4POI's inclusion, but that leaves the SOTA claim incomplete. Fourth, the cross-city setup (Table 4) is under-explained—are the models trained on one city's POI vocabulary and tested on another's? That matters for interpretation.\n\nOverall: if the leakage question resolves well (the authors likely did split properly, given they mention a chronological split), the paper is a solid contribution for the POI-recommendation community. As written, it deserves peer review, but with the expectation that the authors provide the missing split statement, add error bars or significance tests, and either include or justify excluding the other LLM baselines. I'd be moderately optimistic after revision.","headline":"GA-LLM is a serious, well-engineered empirical paper that combines known components into a strong next-POI recommender, but the evaluation has an unaddressed leakage risk from precomputed POI embeddings and thin statistics that prevent full confidence in the headline numbers.","tokens_in":13662,"tokens_out":2759,"would_cite":false,"duration_ms":26831,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper claims that adding quadkey-Fourier geographic encodings and POI alignment projections to a fine-tuned LLM improves next-POI top-1 accuracy by about 15-17% over strong prior baselines.","keywords":["next POI recommendation","large language models","geographic coordinate injection","quadkey encoding","learnable Fourier positional encoding","POI transition alignment","cold-start recommendation","LLM fine-tuning"],"falsifier":"Retrain GA-LLM with POI embeddings computed only from the training split (or audit the released code for the split boundary) and compare Acc@1 on the three datasets; a large drop would indicate leakage, while a small drop would confirm the reported gains.","tokens_in":12707,"feed_emoji":"📍","tokens_out":6322,"duration_ms":56347,"temperature":0.7,"pith_summary":"The paper argues that large language models struggle at next-POI recommendation because they cannot reason about precise GPS coordinates and have no built-in knowledge of which places people tend to visit next. To fix this, GA-LLM feeds each location through a compact spatial encoder and injects transition-aware POI embeddings into the model's token space. On three real-world check-in datasets the model reports top-1 accuracy of 0.3919 on NYC, 0.3482 on TKY, and 0.2566 on CA, beating the strongest prior baselines by 16.19%, 14.73%, and 16.69%. The central point is that structured geographic and transition signals, rather than text descriptions of coordinates, let an LLM predict destinations it never saw in its prompt.","feed_headline":"Geography-aware LLM beats prior next-POI models by 15-17%","feed_subtitle":"Compact quadkey-Fourier coordinates and aligned POI embeddings let an LLM predict destinations it has not seen.","key_machinery":"The central mechanism is the pairing of two injection modules. GCIM converts each GPS point into a quadkey string (a base-4 quadtree tile identifier whose prefix length encodes the zoom level), builds overlapping n-grams from that string, applies self-attention, and adds a learnable Fourier positional encoding so the LLM receives a compact multi-scale spatial representation instead of a long string of coordinate tokens. PAM maps low-dimensional POI embeddings, taken from a pretrained sequential recommender such as MTNet, through an MLP into the LLM's token embedding space, so transition knowledge is available as input tokens during fine-tuning. The two modules together are what allow the LLM to predict a target POI that never appears in its prompt.","core_discovery":"GA-LLM establishes that next-POI recommendation can be improved by treating geography and POI transitions as structured inputs to an LLM rather than as raw text. GCIM discretizes GPS coordinates into hierarchical quadkey strings, enriches them with overlapping n-grams and self-attention, and applies a learnable Fourier positional encoding to produce a compact spatial token; PAM projects precomputed POI embeddings from a sequential model into the LLM's semantic space via an MLP. With Llama-2-7b fine-tuned on formatted check-in queries, the model reports Acc@1 of 0.3919 on NYC, 0.3482 on TKY, and 0.2566 on CA, outperforming the strongest prior baseline by 16.19%, 14.73%, and 16.69%, and reducing the average distance of wrong predictions on CA from 61.38 km to 37.63 km. The paper also reports that removing either module lowers accuracy, and that the geographic encoder alone improves cross-city cold-start accuracy.","pith_inferences":["If the accuracy gains survive a clean train/test split of the sequential model's POI embeddings, the same injection recipe could transfer to other spatial LLM tasks such as destination prediction or trajectory generation, where discrete grid encodings may outperform raw coordinate text.","Because GCIM relies on the Mercator projection and quadkey tiling, its transferability to non-Mercator or globe-spanning settings is untested; a direct experiment would evaluate GA-LLM on data from extreme latitudes or a different map projection.","The cross-city results hint that spatial structure is learnable independently of locale, but the ceiling of GA-LLM may be set by the external sequential model that produces POI embeddings; replacing MTNet with a simple bigram transition matrix would reveal how much of the gain comes from transition knowledge versus the LLM's own reasoning."],"forward_implications":["Next-POI top-1 accuracy can be raised by roughly 15-17% over prior best models on standard benchmarks, suggesting that LLM-based recommenders need structured geographic encoding rather than coordinate text.","Compact quadkey-Fourier representations reduce the number of tokens per check-in, making fine-tuning faster and per-query inference faster than a text-only LLM baseline that uses 200 historical check-ins per query.","GCIM alone enables cross-city cold-start transfer: a model trained on one city predicts POIs in another city more accurately than a text-only LLM, even when the target POI is absent from the input.","PAM improves prediction accuracy specifically for cases where the target POI does not appear in the prompt, and it is designed to work with different sequential or graph-based POI embedding sources.","Wrong predictions land closer to the true location on sparse datasets, indicating that the spatial encoding reduces geographically implausible LLM outputs."],"supporting_citations":[{"why":"The text-only LLM baseline that GA-LLM must beat, and the reference for the extreme position-bias evaluation and hallucination comparison.","marker":"[Li et al., 2024b]"},{"why":"MTNet supplies the pretrained POI embeddings that PAM projects into the LLM semantic space, and MTNet also serves as a sequential baseline.","marker":"[Huang et al., 2024]"},{"why":"GeoSAN's quadkey encoding is the basis for GCIM's hierarchical geographic discretization.","marker":"[Lian et al., 2020]"},{"why":"Introduces the learnable Fourier positional encoding that GCIM adapts to quadkey strings.","marker":"[Li et al., 2021]"},{"why":"ROTAN is the strongest sequential baseline on CA and an example of a graph/sequential model whose POI embeddings PAM can align.","marker":"[Feng et al., 2024b]"},{"why":"STHGCN provides the chronological data split and preprocessing used in all experiments and is a hypergraph baseline.","marker":"[Yan et al., 2023]"},{"why":"LongLoRA is the fine-tuning method that enables the 32k-token context used in GA-LLM.","marker":"[Chen et al., 2024]"},{"why":"Llama-2-7b is the base LLM that GA-LLM fine-tunes.","marker":"[Touvron et al., 2023]"}],"fun_headline_variants":["GA-LLM maps GPS to tokens, boosting next-POI accuracy 16%","LLM learns POI transitions: 16% better next-destination picks","Geography-aware LLM cuts wrong-pick distance by 23 km","Quadkey-Fourier encoding makes LLMs spatial for POI prediction","GA-LLM: structured geo inputs give LLM 16% edge on next-POI"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The reported accuracy gains rest on POI embeddings precomputed by a separate sequential model, and the paper never states that those embeddings were trained only on the training split; if the embedding model saw test check-ins, transition patterns from ground-truth POIs would leak into the LLM and inflate Acc@1.","fun_headline_variants_meta":{"raw":{"variants":["GA-LLM maps GPS to tokens, boosting next-POI accuracy 16%","LLM learns POI transitions: 16% better next-destination picks","Geography-aware LLM cuts wrong-pick distance by 23 km","Quadkey-Fourier encoding makes LLMs spatial for POI prediction","GA-LLM: structured geo inputs give LLM 16% edge on next-POI"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000677,"raw_usage":{"total_tokens":3102,"prompt_tokens":994,"completion_tokens":2108,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":610,"completion_tokens_details":{"reasoning_tokens":2004}},"tokens_in":610,"tokens_out":2108,"duration_ms":13585,"temperature":1.0,"reasoning_tokens":2004,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T20:37:45.240980+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Retrain GA-LLM with POI embeddings computed only from the training split (or audit the released code for the split boundary) and compare Acc@1 on the three datasets; a large drop would indicate leakage, while a small drop would confirm the reported gains.","supporting_citations":[{"cited_title":"Learning time slot preferences via mobility tree for next POI recommenda- tion","cited_arxiv_id":null,"evidence_quote":"MTNet supplies the pretrained POI embeddings that PAM projects into the LLM semantic space, and MTNet also serves as a sequential baseline."},{"cited_title":"Geography-aware sequential lo- cation recommendation","cited_arxiv_id":null,"evidence_quote":"GeoSAN's quadkey encoding is the basis for GCIM's hierarchical geographic discretization."},{"cited_title":"Spatio-temporal hypergraph learning for next POI recommendation","cited_arxiv_id":null,"evidence_quote":"STHGCN provides the chronological data split and preprocessing used in all experiments and is a hypergraph baseline."},{"cited_title":"Lon- glora: Efficient fine-tuning of long-context large language models","cited_arxiv_id":null,"evidence_quote":"LongLoRA is the fine-tuning method that enables the 32k-token context used in GA-LLM."}],"review_version":1}