{"id":"547c5006-7185-444c-a14e-21e9fa66c62a","arxiv_id":"1909.00154","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":7,"one_line_summary":"Embedding encodings of categorical variables improve a Swissmetro mode choice model's out-of-sample log-likelihood compared with dummy and PCA encodings, but the gain is modest and mostly visible in the best of many random runs.","lead":"This paper applies word2vec-style embeddings to categorical variables in travel mode choice models, mapping categories such as trip purpose and origin-destination pairs into a low-dimensional learned space. It reports better out-of-sample fit than dummy or PCA encodings on the Swissmetro dataset, and releases an open-source Python package, PyTre.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Repeated observations from the same 1,188 respondents appear to be split at the row level, so the test set is not out-of-sample for individuals; the reported log-likelihood gains may largely reflect within-person leakage.","rationale":"The reader identifies the low-rank OD assumption as the weakest point. I think there is a more fundamental check: the validity of the test set itself. The paper supplies the ingredients for the concern (1,188 individuals, splits described only in vectors), and no limitation passage acknowledges the panel structure. If the split is at the row level, every out-of-sample number in Table 12 and the abstract is suspect, and the 164-significant-coefficients issue is secondary. This is not an accusation of bad faith; it is a concrete design question that the open code and data can settle. I keep the reader's CONDITIONAL verdict because a grouped re-split is straightforward and the PyTre package makes it checkable, but the burden is on the authors to show the split was by respondent or to reframe the claims as within-sample choice prediction rather than out-of-sample prediction for new travelers.","tokens_in":18547,"tokens_out":15067,"duration_ms":219663,"concrete_test":"Re-run the full PyTre pipeline with a grouped split: assign all rows of each respondent ID to exactly one of train/dev/test (60/20/20 of individuals), then compare test log-likelihood and pseudo-R2 for embeddings, PCA, and reduced dummies, including clustered standard errors by respondent in the MNL step. If the embeddings margin over PCA shrinks below roughly 10 LL points or reverses, the reported out-of-sample advantage is substantially attributable to within-individual leakage rather than to the embedding representation.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 5.1 reports that the Swissmetro data comprise 10,469 responses from 1,188 individuals, then describes the experimental split only as '60% of the dataset (6373 vectors)', '20% (2003 vectors)', and '20% (2003 vectors)', with no statement that all choices from a given respondent were kept in the same fold. Swissmetro is a stated-preference panel, so a row-level random split places repeated choices from the same person in both training and test sets. The load-bearing result in Table 12 is the test-set comparison (embeddings best LL -1,389.9; mean -1,415.7; PCA -1,427.0; reduced dummies -1,429.3), and the abstract's 'up to 20% out-of-sample' claim is drawn from this comparison. If the same individuals appear in train and test, the model can exploit respondent-specific patterns (e.g., through OD and TICKET combinations) that would not be available when predicting a new traveler. This also invalidates the pseudo-R2 as an out-of-sample measure and can differentially favor the embeddings specification. The paper's limitation discussion acknowledges stochasticity but does not address the panel structure or clustered standard errors. Without a respondent-level split or an explicit justification for treating choices as independent, the headline empirical claim is not established.","agreement_with_reader":"disagree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes 'travel behavior embeddings,' a method that re-represents categorical variables from travel surveys (e.g., origin-destination, ticket type, who pays, age, income) as low-dimensional continuous vectors learned by a neural network, in the spirit of word2vec. The embeddings are trained simultaneously with a regularizer layer, then fed as regressors into a multinomial logit mode-choice model for the Swissmetro dataset. The authors compare out-of-sample performance against dummy-variable encodings (with and without origin-destination effects) and PCA encodings, reporting that their best embeddings model achieves the highest test pseudo-R2 and can estimate statistically significant coefficients for 164 of 178 OD pairs. They also present scenario analyses for efficient survey design and release a Python package, PyTre, with replication code and data.","tokens_in":18883,"tokens_out":2433,"duration_ms":23913,"significance":"If the empirical claims hold, the paper makes a useful methodological contribution by adapting embedding techniques to discrete-choice travel behavior modeling, with the practical benefits of dimensionality reduction and shared latent structure. The open-source package and reproducible notebook are clear strengths, as is the repeated-run design with a development set (300 runs) and the explicit reporting of mean and standard deviation. The projection back into dummy-variable space for interpretability is also a valuable feature. However, the headline result depends on several load-bearing assumptions about the experimental design and the stability of the reported gains; the stated 'up to 20% out-of-sample improvement' is not robustly established by the evidence in the current version.","major_comments":[{"comment":"Section 5.1 states that the Swissmetro data consist of 10,469 responses from 1,188 individuals, and then describes the split only as '60% of the dataset (6373 vectors)', '20% (2003 vectors)', and '20% (2003 vectors)'. Because this is a stated-preference panel with repeated choices per person, a row-level random split likely places the same respondents in both the training and test sets. The test-set log-likelihoods in Table 12, which are the basis of the paper's central out-of-sample claims, would then reflect within-person leakage rather than prediction for new travelers, and the pseudo-R2 differences would not measure true out-of-sample generalization. The paper needs either an individual-level split, a clustered evaluation that accounts for repeated observations, or an explicit and defensible justification for treating each choice as independent.","section":"Section 5.1 (experimental split)"},{"comment":"The test-set comparison in Table 12 uses the single best embeddings model out of 300 runs, selected by development-set log-likelihood. The reported mean test log-likelihood is -1415.7 with standard deviation 11.3, versus -1427.0 for PCA and -1429.3 for reduced dummies; the 'best' value is -1389.9. The mean improvement over PCA is only about 0.8% and is well within one standard deviation of the run-to-run variation. The headline 'up to 20%' improvement is therefore an extreme-statistic artifact unless the full distribution of test performances and the selection procedure justify it. The authors should report the distribution, compare mean embedding performance against PCA, and, if claiming superiority, provide a statistical test that accounts for the multiple comparisons across the 300 runs.","section":"Section 5.3 and Table 12 (model selection and reported gains)"},{"comment":"The embedding dimensions K in Table 3 are chosen through an 'incremental grid search' whose exact range, criterion, and number of configurations are not specified; this makes it unclear how much model selection is being performed and whether the reported results capitalize on chance. Additionally, the 164 'statistically significant' projected OD coefficients in Table 6 treat the learned embeddings as fixed when applying normal-distribution linear-combination rules, ignoring the sampling variability of the embedding estimates themselves. The p-values are thus conditional on a stochastic, data-driven representation and are likely anticonservative. The authors should either provide a bootstrap procedure that re-estimates embeddings, or clearly state and justify the conditional interpretation.","section":"Section 5.2 (Table 3) and Section 5.3 (projected OD coefficients)"}],"minor_comments":[{"comment":"The abstract contains the typo 'model choice model' instead of 'mode choice model,' and Section 1 has 'efects' for 'effects.' These should be corrected.","section":"Abstract and Section 1"},{"comment":"The row labeled 'Embeddings (mean)' reports the same training log-likelihood (-4,397.0) as the 'Embeddings (best)' row despite the standard deviation of 10.7; this is either a typo or requires clarification of what the mean was computed over.","section":"Table 12"},{"comment":"The choice of K using the '90% variance explained' PCA rule is mentioned only parenthetically, and the reader is told the results were 'generally worse' but not shown. Since this is a natural data-driven baseline for setting K, a brief summary of those results would improve transparency.","section":"Section 5.2"},{"comment":"The 'Big Data + detailed survey' simulation (Figure 10, right) is described at a high level, but the exact procedure for splitting the Swissmetro data into 'big data' and 'detailed' portions, and the number of repetitions used, are not specified; error bars or confidence intervals would help interpret the comparisons.","section":"Section 5.6"},{"comment":"Reference [1] is given as 'Tech. rep.' without a number or year; the standard Swissmetro report (Bierlaire, Axhausen, and Abay, 2001) should be cited in a complete form.","section":"References"}],"recommendation":"major_revision","confidential_remarks":null},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The genuinely new thing here is the application of word2vec-style embeddings to categorical variables inside a multinomial logit mode choice model, with a regularizer that reconstructs the one-hot input and a projection back to dummy space for interpretability. On the practical side, the author has done the right things: open data, released PyTre, repeated runs, an explicit train/dev/test split, and a candid limitations section that admits the stochasticity problem. The MDS visualizations and the idea of shared latent spaces for small categoricals are a real contribution to travel behavior modeling.\n\nBut the central empirical claim is not established. The stress-test note about the panel structure is correct and important. Swissmetro is 10,469 responses from 1,188 individuals, and the split is described as 60/20/20 of vectors, not of respondents. Nothing in the text says all choices from a given person were kept in the same fold. A row-level split puts repeated choices from the same respondent in train and test. For a stated-preference panel, that is leakage: the model can memorize person-level response patterns through OD, ticket, age, and income combinations. The test log-likelihood then no longer measures generalization to new travelers, and the comparison with PCA and dummies is unfair. This is not a minor quibble; it undermines the abstract's 'up to 20% out-of-sample' claim.\n\nEven taking the numbers at face value, the typical run is only slightly better than PCA: mean test LL -1415.7 vs PCA -1427.0, with SD 11.3. The headline uses the best of 300 runs, which is a selected maximum. The paper honestly reports the range but does not correct for selection. Also, K values and regularization hyperparameters come from an unspecified grid search on the development set, and the projection-back p-values treat embeddings as known rather than estimated, so the 164 'significant' OD coefficients should be read as illustrative, not inferential.\n\nWho gets value? Travel demand modelers curious about representation learning will find the idea and the package worth their time. The paper is clearly written and the method is reproducible. But as it stands, the empirical section needs major work: redo the split at the respondent level, report mean/median and a proper selection rule, and either fix the p-value procedure or drop the significance claims. I would send this to peer review, not desk reject, because the core method is novel and the author has provided the tools to verify it. A serious referee should run the leakage check first.","headline":"Genuinely novel application of embeddings to categorical variables in discrete choice, but the headline out-of-sample gain probably reflects respondent-level train/test leakage.","tokens_in":19352,"tokens_out":2757,"would_cite":false,"duration_ms":27917,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Re-representing categorical travel variables as learned low-dimensional embeddings improves out-of-sample mode choice models.","keywords":["travel behavior embeddings","categorical variables","mode choice","multinomial logit","latent representation","word2vec","Swissmetro","representation learning"],"falsifier":"Using synthetic mode-choice data with known OD effects that are deliberately high-rank (e.g., 178 independent draws with no shared structure), fit the $K=3$ embedding model and a full dummy model; if the embedding still reports most OD coefficients as statistically significant, those significance claims are an artifact of forcing data through a rank-3 linear bottleneck rather than evidence of true structure.","tokens_in":18293,"feed_emoji":"🚆","tokens_out":8704,"duration_ms":65436,"temperature":0.7,"pith_summary":"Travel behavior embeddings re-represent the discrete variables used in mode choice models—trip purpose, age, income, ticket type, who pays, and origin–destination pair—as points in a low-dimensional latent space learned from data, rather than as dummy variables or PCA components. The paper applies this to the Swissmetro dataset and reports out-of-sample gains of up to 20% in log-likelihood over the corresponding dummy-variable model (and up to 60% over an initial baseline model), with test pseudo-$R^2$ of 0.326 versus 0.307–0.308 for the alternatives. Because the embedding space is shared, estimates for rare categories borrow strength from the whole dataset: the model recovers statistically significant coefficients for 164 of 178 origin–destination pairs, where a full dummy specification yields none. The practical stakes are that cheaper, coarser travel surveys could support rich choice models by reusing embeddings learned from larger, lighter datasets.","feed_headline":"Embeddings outperform dummies in Swissmetro mode-choice tests","feed_subtitle":"Mapping trip purpose, age, and origin-destination pairs into a shared latent space lifts out-of-sample fit and recovers 164 OD effects.","key_machinery":"The Travel Embeddings layer is a softmax-output neural network that takes a one-hot input for each categorical variable, multiplies by an embedding matrix $W$ of size $K \\times D$ ($D$ categories in, $K$ embedding coordinates out), and feeds the resulting vector, together with other variables, into the final choice softmax. A second softmax regularizer branch tries to reconstruct the original one-hot vector, penalizing embeddings that throw away too much category identity. All embedding matrices are trained simultaneously on a train split, with the model chosen by development-set likelihood; after estimation the embeddings are projected back to dummy space to recover interpretable category coefficients and $p$-values. MDS plots of the resulting vectors let modelers read closeness between categories as similarity of effects on mode choice.","core_discovery":"The central claim is that a categorical variable's effect in a random-utility mode choice model can be carried by a small vector in an embedding space, estimated jointly with the choice model by a neural network with a softmax output, and then projected back to the original dummy-variable coefficient space without losing interpretability. In the Swissmetro experiment, the best embedding model estimates 39 parameters instead of the 232 of the full dummy specification, reaches test pseudo-$R^2$ 0.326, and yields 164 statistically significant OD-pair coefficients; the full dummy model collapses on the test set (pseudo-$R^2$ -0.789), the reduced dummy model reaches 0.307, and PCA reaches 0.308. The paper reads these results as evidence that sharing a latent space lets every observation contribute to every category's estimate, and that supervised, task-specific encodings carry information an unsupervised PCA projection does not.","pith_inferences":["A testable extension of the paper's logic is that one categorical variable should receive different embeddings for different targets (mode choice versus departure time versus destination choice), and the distances between categories should reshuffle accordingly; this follows from the supervised design but is not demonstrated in the paper.","The strong OD result suggests a geometric prediction: if OD effects are truly low-rank in $K=3$, the dominant axes should align with major transport corridors or city-pair volumes, a check that the paper's MDS plots only hint at.","The transferability claim predicts that embeddings trained on one city's or country's travel survey should improve out-of-sample fit in another survey precisely when travel behavior is consistent between them; running that cross-dataset test would settle the community-database vision.","Because the embedding is a linear projection, it could be inserted into nested or mixed logit specifications as continuous covariates in taste parameters, preserving the projection-to-dummy interpretation; this is a natural next step the paper mentions only as future work."],"forward_implications":["High-cardinality categorical variables like OD pairs can enter choice models without the dimensionality explosion that makes dummy-variable estimation fail; the Swissmetro result of 164 significant OD coefficients is the direct evidence.","Survey design can be split into a cheap light part (mode plus OD plus ticket) and a small detailed part; embeddings learned from the light part keep the choice model competitive at detailed-sample percentages as low as 10%.","Because embeddings are task-specific, the same variable will receive different encodings for different choice problems, unlike PCA or dummy schemes, which are fixed representations.","Coefficients remain interpretable: by projecting back to the dummy-variable space, the model yields category-level coefficients and significance tests, so the method is not a black box.","The advantage shrinks as datasets grow; with large, well-balanced samples the paper expects dummy variables to be sufficient."],"supporting_citations":[{"why":"Supplies the Swissmetro survey data on which all models are trained and tested.","marker":"[1]"},{"why":"Introduces the skip-gram formulation that the paper adapts to learn embeddings.","marker":"[5]"},{"why":"Source of the word-embedding algorithm that the paper transfers to travel behavior.","marker":"[6]"},{"why":"Earlier demonstration that word embeddings improve demand prediction, motivating the transfer.","marker":"[11]"},{"why":"Provides the multinomial logit estimation package used to fit the baseline and comparison models.","marker":"[12]"},{"why":"Provides the mode-inference method used in the simulated big-data survey scenario.","marker":"[14]"}],"fun_headline_variants":["Travel behavior embeddings beat dummy variables by 20% out-of-sample","PyTre embeddings lift mode-choice fit 20% over dummy coding","Data-driven embeddings outperform dummies in Swissmetro choice tests","Embeddings: 20% better out-of-sample, 164 OD effects recovered","Neural embeddings map travel categories, beat traditional dummies"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the effect of every category—especially each of the 178 origin–destination pairs—can be captured by a point in a three-dimensional Euclidean space, so that Euclidean distance between points faithfully mirrors similarity of the categories' effects on mode choice.","fun_headline_variants_meta":{"raw":{"variants":["Travel behavior embeddings beat dummy variables by 20% out-of-sample","PyTre embeddings lift mode-choice fit 20% over dummy coding","Data-driven embeddings outperform dummies in Swissmetro choice tests","Embeddings: 20% better out-of-sample, 164 OD effects recovered","Neural embeddings map travel categories, beat traditional dummies"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000197,"raw_usage":{"total_tokens":1411,"prompt_tokens":1041,"completion_tokens":370,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":657,"completion_tokens_details":{"reasoning_tokens":280}},"tokens_in":657,"tokens_out":370,"duration_ms":21740,"temperature":1.0,"reasoning_tokens":280,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-14T06:00:29.310752+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Using synthetic mode-choice data with known OD effects that are deliberately high-rank (e.g., 178 independent draws with no shared structure), fit the $K=3$ embedding model and a full dummy model; if the embedding still reports most OD coefficients as statistically significant, those significance claims are an artifact of forcing data through a rank-3 linear bottleneck rather than evidence of true structure.","supporting_citations":[{"cited_title":"Bierlaire, K","cited_arxiv_id":null,"evidence_quote":"Supplies the Swissmetro survey data on which all models are trained and tested."},{"cited_title":"Guthrie, B","cited_arxiv_id":null,"evidence_quote":"Introduces the skip-gram formulation that the paper adapts to learn embeddings."},{"cited_title":"Rodrigues, I","cited_arxiv_id":null,"evidence_quote":"Earlier demonstration that word embeddings improve demand prediction, motivating the transfer."},{"cited_title":"Brathwaite, J","cited_arxiv_id":null,"evidence_quote":"Provides the multinomial logit estimation package used to fit the baseline and comparison models."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the mode-inference method used in the simulated big-data survey scenario."}],"review_version":1}