REVIEW 3 major objections 4 minor 1 cited by
Exploring the Potential of Large Language Models in Public Transportation: San Antonio Case Study
T0 review · 3 major / 4 minor · reviewed 2026-08-10 · deepseek-v4-flash
Pith's one-line read The paper reports that ChatGPT models score 47.97% to 98.44% on GTFS transit understanding questions and 60.53% to 90.48% on GTFS retrieval, concluding that fine-tuning is necessary before real-world deployment.
desk verdict A competent, modest GTFS benchmark extension whose headline accuracy numbers rest on a hand-written answer key that already shows one ambiguous question in the paper's own sample. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The argument is carried by a two-task benchmarking design on a public data standard. The understanding task uses 195 multiple-choice questions (later expanded to 444, and augmented into 1,776 variants by swapping in a 'none of these' option) authored from the official GTFS Schedule documentation, scored by exact match. The retrieval task feeds the model slices of the San Antonio VIA GTFS feed — three routes, 34 trips, 60 stops — and asks 80 short-answer questions that require search, filtering, sorting, grouping, and joining across files, scored by semantic equivalence. This setup lets the authors separate what the model already knows from what it can extract from supplied data, and the accuracy gap between the two tasks provides the evidence for the conclusion that both data coverage and architecture matter.
What would settle it
Recruit two independent GTFS experts to write the answer key for the same 195 core multiple-choice questions and 80 retrieval questions from the same VIA feed snapshot and GTFS Schedule documentation, without seeing the authors' key, and compare agreement; if the experts frequently disagree with the paper's key, every reported accuracy figure is called into question. A second, cheaper check is to rerun each zero-shot prompt dozens of times at the same temperature and look at the spread in accuracy, which would test whether the reported single-run numbers are stable.
Extended reading notes
Core claim
The central finding is a measured capability profile, not a single number: pre-trained ChatGPT answers straightforward GTFS questions well (best 98.44% on attribute mapping), but performance drops to roughly chance on categorical mapping, and adding a 'none of these' option costs about ten points of accuracy on average, with GPT-4 more robust than GPT-3.5. On retrieval, the models handle simple lookups in one or two GTFS files well (best 90.48%) but average only about 64% when a question requires joining data across several files such as stops.txt, stop_times.txt, trips.txt, and routes.txt. The paper takes this pattern as evidence that LLM failures in this domain are partly missing pre-training coverage and partly reasoning limits, and that real transit deployments should not assume a general-purpose model is reliable without fine-tuning and prompt engineering.
Load-bearing premise
The load-bearing premise is that the hand-written answer key is correct and unambiguous for all 195 core multiple-choice and 80 retrieval questions, so that every accuracy number in the paper is a true measure of the model's knowledge rather than of quirks in the key.
Editorial extensions
If this is right
- Direct users of a general-purpose ChatGPT should expect high accuracy on simple GTFS definitions but near-chance performance on categorical distinctions like Rail versus Light Rail.
- A transit agency that hands an LLM a GTFS feed can rely on it for single-file lookups but should verify multi-file join answers, where average accuracy falls to about 64%.
- Because adding a 'none of these' option cuts accuracy by around 10 points, benchmark designers and agencies should treat forced-choice multiple choice as systematically easier than open-ended questions.
- GPT-4's smaller performance drop on augmented questions and its consistent edge in retrieval suggest that newer, larger models are the safer default for transit experiments, though still not deployment-ready.
Reading between the lines
- An implication the authors leave implicit is that the same 80-question retrieval set could be run against other GTFS feeds, not just San Antonio's, to test whether these accuracy ranges generalize across cities without retraining.
- Because the paper reports single-run zero-shot results, its accuracy ranges are likely upper bounds for deployment; a natural extension is to sample each prompt repeatedly and report variance, which would quantify the inconsistency the authors discuss in Section 7.2.
- If Categorical Mapping's near-chance performance is really caused by semantic similarity, then fine-tuning on GTFS-specific categories or using a retrieval-augmented taxonomy should move that category well above 50%; that prediction is testable and follows directly from the paper's diagnosis.
- The authors' estimate that prompt engineering can add about 8% accuracy suggests a cheaper intervention than full fine-tuning, and measuring whether optimized prompts close most of the gap on complex retrieval would directly inform transit agencies' deployment choices.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. This paper presents a case study evaluating two OpenAI chat models (GPT-3.5-turbo and GPT-4/GPT-4o) on two tasks: understanding GTFS semantics and retrieving information from a reduced San Antonio VIA GTFS feed. The authors construct 195/780/444/1776 multiple-choice questions for Experiments I-IV plus 80 short-answer retrieval questions for Experiment V, score by exact match and semantic equivalence, and report accuracy ranges of 47.97-98.44% for understanding and 60.53-90.48% for retrieval. They conclude that LLMs are promising for transit applications but require prompt engineering and fine-tuning. The paper also contains a qualitative discussion of LLM inconsistency and prompt engineering.
Significance. The task design, separating pre-trained 'understanding' from context-provided 'information retrieval,' is a useful framing, and using a real GTFS feed as the retrieval corpus is a strength. If the reported numbers were reliable, the paper would provide a practical benchmark for transit agencies considering LLM-based passenger information systems. However, the paper provides no code, data, or answer key; the ground-truth key is shown to contain at least one ambiguous item; and the single-run evaluation with no confidence intervals or significance tests does not support the comparative claims. The contribution is therefore preliminary rather than conclusive, with modest novelty beyond the prior GTFS benchmark it follows ([31]).
major comments (3)
- [Section 5.1, Table 2] The sample Attribute Mapping question in Table 2, 'In which file does the shape_dist_travelled attribute appear in GTFS?', is not uniquely answerable: the official GTFS Schedule reference defines shape_dist_traveled as an optional field in both shapes.txt and stop_times.txt (using a single-'l' spelling), so both answer (b) and answer (d) are defensible. Since all Experiment I-IV accuracies in Table 3 are exact matches against a hand-written answer key that is not released in the paper, a single ambiguous item in a small category can move the reported percentages by several points (e.g., one question is 7.1% of the 14-question Term Definition category). The authors should release the full question bank, answer key, and GTFS snapshot, or provide a second-annotator agreement study showing that every question has a unique correct answer.
- [Section 5.3, Section 6.2] Section 5.3 says that Experiment V accuracy is based on 'semantic equivalence' between the model output and the ground-truth answer, but no scoring rubric, no second annotator, and no inter-annotator agreement are reported. The retrieval questions are free-form and the paper itself documents substantial LLM output variability in Section 7.2, so the reported retrieval accuracies (60.53-90.48%) and the claimed GPT-4o advantage ('average performance increase of approximately 15%' in Section 6.2) are not verifiable as stated. A defined scoring protocol or released scored transcripts is needed before these numbers can be interpreted.
- [Section 6, Table 3, Figure 1] All accuracy numbers in Table 3 and Figure 1 appear to come from a single run per model, with no confidence intervals, repeated runs, or significance tests. Some categories contain very few items, and the paper itself acknowledges in Section 6.1.1 that the 14-question Term Definition category is too small for reliable assessment. Nevertheless, Sections 6.1.2 and 6.2 draw comparative conclusions such as GPT-4o being more robust and 'consistently outperforming' GPT-3.5-turbo. Given the sampling variability at these sample sizes and the model nondeterminism the authors describe in Section 7.2, the observed differences are not established; repeated runs or statistical significance testing are required to support the central comparison.
minor comments (4)
- [Section 5] The paper states that the five experiments comprise 3275 multiple-choice questions, but the counts given in Section 5.1 sum to 3195 (195 + 780 + 444 + 1776 = 3195). The number 3275 is repeated in Section 7.1 and should be corrected.
- [Sections 5.1, 6.2, Table 3, Figure 1] The manuscript uses 'GPT-4' in the text and '4o' in Table 3 and Figure 1 without stating whether the same model version and API endpoint were used across experiments. Please clarify the exact model identifiers and configuration (temperature, max tokens) for each run.
- [Section 7.3] The claim that 'performance of our experiments may improve by up to ≈ 8% through optimized prompting' is unsupported: no prompt-engineering experiment is reported, and the cited references [37, 38] are general introductions rather than evidence for this specific figure. Remove the quantitative claim or provide the supporting experiment.
- [Section 2] The assertion that findings from a three-route, 34-trip, 60-stop subset of one agency's feed 'can be applied to other cities' overgeneralizes; the authors should temper this claim given the severely reduced retrieval dataset.
Circularity Check
No circularity: the accuracy claims are computed against an externally grounded GTFS answer key and VIA feed, not against the models' own outputs.
full rationale
The paper's central claims—accuracy ranges of 47.97–98.44% on understanding and 60.53–90.48% on retrieval (Sections 7.1 and 8)—are benchmark measurements, not derivations from the models. Ground truth is constructed from the official GTFS Schedule documentation and a static snapshot of San Antonio's VIA GTFS feed (Sections 5.1 and 5.2), and the evaluation metric (Eq. 1) compares model outputs to that externally fixed key. There are no fitted parameters, no parameter identifiability argument, and no 'prediction' that is a renamed fit. The only methodological dependency is on the hand-written answer key and on the 'semantic equivalence' judgment in Experiment V (Section 5.3); those are validity/reproducibility concerns about ground-truth correctness and annotator reliability, not circularity, because the key is not derived from the LLM outputs. Even if one sample MCQ in Table 2 is ambiguous (shape_dist_travelled appears in both shapes.txt and stop_times.txt), that would undermine label quality, not indicate circularity. The paper's self-citations (e.g., refs. [1]–[9]) are contextual and not load-bearing for the evaluation; ref. [31], which inspired the question design, is an external prior benchmark rather than a self-citation. The paper itself flags limitations such as small category sizes and the need for prompt engineering/fine-tuning, which are honest scope statements rather than circular moves. Accordingly, the derivation chain is self-contained against external benchmarks and no circular step is present.
Assumptions & free parameters
assumptions (5)
- domain assumption The manually authored ground-truth answers in the benchmark are correct and unambiguous.
- domain assumption The 'none of these' augmentation replaces only incorrect answer choices, preserving a single correct answer.
- domain assumption Accuracy via exact match (Experiments I-IV) and semantic equivalence (Experiment V) is a valid measure of LLM understanding.
- ad hoc to paper The reduced San Antonio VIA GTFS subset (3 routes, 34 trips, 60 stops) is representative of full-scale transit data.
- domain assumption OpenAI model versions and API configurations are stable across the experiments.
Cite this review
Pith. "Pith review of Exploring the Potential of Large Language Models in Public Transportation: San Antonio Case Study." pith.science (2026). https://pith.science/paper/SLM6CALJ
@misc{pith2026250103904,
author = {Pith},
title = {Pith review of: Exploring the Potential of Large Language Models in Public Transportation: San Antonio Case Study},
year = {2026},
howpublished = {\url{https://pith.science/paper/SLM6CALJ}},
note = {Machine review of arXiv:2501.03904}
}
read the original abstract
The integration of large language models (LLMs) into public transit systems presents a transformative opportunity to enhance urban mobility. This study explores the potential of LLMs to revolutionize public transportation management within the context of San Antonio's transit system. Leveraging the capabilities of LLMs in natural language processing and data analysis, we investigate their capabilities to optimize route planning, reduce wait times, and provide personalized travel assistance. By utilizing the General Transit Feed Specification (GTFS) and other relevant data, this research aims to demonstrate how LLMs can potentially improve resource allocation, elevate passenger satisfaction, and inform data-driven decision-making in transit operations. A comparative analysis of different ChatGPT models was conducted to assess their ability to understand transportation information, retrieve relevant data, and provide comprehensive responses. Findings from this study suggest that while LLMs hold immense promise for public transit, careful engineering and fine-tuning are essential to realizing their full potential. San Antonio serves as a case study to inform the development of LLM-powered transit systems in other urban environments.
Figures
Forward citations
Cited by 1 Pith paper
-
CrashSage: A Large Language Model-Centered Framework for Contextual and Interpretable Traffic Crash Analysis
A fine-tuned Llama-3-8B on LLM-augmented crash narratives achieves a Macro-F1 of 0.736 for crash severity prediction, outperforming zero-shot and few-shot GPT-4o and Llama-3-70B baselines but with no error bars or cod...
Reference graph
Works this paper leans on
-
[31]
Chatgpt for gtfs: benchmarking llms on gtfs semantics and retrieval,
S. Devunuri, S. Qiam, and L. J. Lehe, “Chatgpt for gtfs: benchmarking llms on gtfs semantics and retrieval,” Public Transport, pp. 1–25, 2024
work page 2024
-
[1]
Adversarial nlp for social network applications: Attacks, defenses, and research directions,
I. Alsmadi, K. Ahmad, M. Nazzal, F. Alam, A. Al-Fuqaha, A. Khreishah, and A. Algosaibi, “Adversarial nlp for social network applications: Attacks, defenses, and research directions,” IEEE Transactions on Computational Social Systems, vol. 10, no. 6, pp. 3089–3108, 2022
work page 2022
-
[2]
A deep learning ensemble approach to detecting unknown network attacks,
R. Ahmad, I. Alsmadi, W. Alhamdani, and L. Tawalbeh, “A deep learning ensemble approach to detecting unknown network attacks,” Journal of Information Security and Applications, vol. 67, p. 103196, 2022
work page 2022
-
[3]
Enhancing neural text detector robustness with µ attacking and rr-training,
G. Liang, J. Guerrero, F. Zheng, and I. Alsmadi, “Enhancing neural text detector robustness with µ attacking and rr-training,” Electronics, vol. 12, no. 8, p. 1948, 2023
1948
-
[4]
Joint 2d-3d breast cancer classification,
G. Liang, X. Wang, Y . Zhang, X. Xing, H. Blanton, T. Salem, and N. Jacobs, “Joint 2d-3d breast cancer classification,” in 2019 IEEE International Conference on Bioinformatics and biomedicine (BIBM). IEEE, 2019, pp. 692–696
work page 2019
-
[5]
X. Xing, G. Liang, C. Wang, N. Jacobs, and A.-L. Lin, “Self-supervised learning application on covid-19 chest x-ray image classification using masked autoencoder,” Bioengineering, vol. 10, no. 8, p. 901, 2023
work page 2023
-
[6]
L. Liu, J. Chang, G. Liang, and S. Xiong, “Simulated quantum mechanics-based joint learning network for stroke lesion segmentation and tici grading,” IEEE Journal of Biomedical and Health Informatics, 2023. 8 This work is accepted to AAAI 2025 Workshop on AI for Urban Planning
work page 2023
-
[7]
Applications of deep machine learning to highway safety and usage assessment,
M. Chen, A. Hadzic, W. Song, and N. Jacobs, “Applications of deep machine learning to highway safety and usage assessment,” in Transportation Research Board Workshop (Sponsored by AED50), Jan. 2021, (oral)
work page 2021
Show all 38 references
-
[8]
U2-net: A very-deep convolutional neural network for detecting distracted drivers,
N. O. Alsrehin, M. Gupta, I. Alsmadi, and S. A. Alrababah, “U2-net: A very-deep convolutional neural network for detecting distracted drivers,” Applied Sciences, vol. 13, no. 21, p. 11898, 2023
2023
-
[9]
Unveiling roadway hazards: Enhancing fatal crash risk estimation through multiscale satellite imagery and self-supervised cross-matching,
G. Liang, J. Zulu, X. Xing, and N. Jacobs, “Unveiling roadway hazards: Enhancing fatal crash risk estimation through multiscale satellite imagery and self-supervised cross-matching,” IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, vol. 17, pp....
2023
-
[10]
Introducing chatgpt,
“Introducing chatgpt,” OpenAI.com, 2022. [Online]. Available: "https://openai.com/index/chatgpt/"
2022
-
[11]
Gpt-4 technical report,
J. Achiam, S. Adler, S. Agarwal, L. Ahmad, I. Akkaya, F. L. Aleman, D. Almeida, J. Altenschmidt, S. Altman, S. Anadkat et al., “Gpt-4 technical report,” arXiv preprint arXiv:2303.08774, 2023
2023 arXiv
-
[12]
Study and analysis of chat gpt and its impact on different fields of study,
D. Kalla, N. Smith, F. Samaah, and S. Kuraku, “Study and analysis of chat gpt and its impact on different fields of study,”International journal of innovative science and research technology, vol. 8, no. 3, 2023
2023
-
[13]
Exploring gpt-4’s characteristics through the 5vs of big data: A brief perspective,
O. M. Alyasiri, A. H. Ali et al., “Exploring gpt-4’s characteristics through the 5vs of big data: A brief perspective,” Babylonian Journal of Artificial Intelligence, vol. 2023, pp. 5–9, 2023
2023
-
[14]
Human-like problem-solving abilities in large language models using chatgpt,
G. Orrù, A. Piarulli, C. Conversano, and A. Gemignani, “Human-like problem-solving abilities in large language models using chatgpt,” Frontiers in artificial intelligence, vol. 6, p. 1199350, 2023
2023
-
[15]
Integrating gpt-technologies with decision models for explainability,
A. Goossens and J. Vanthienen, “Integrating gpt-technologies with decision models for explainability,” inWorld Conference on Explainable Artificial Intelligence. Springer, 2023, pp. 428–448
2023
-
[16]
Revolutionizing neurosurgery with gpt-4: a leap forward or ethical conundrum?
W. Li, M. Fu, S. Liu, and H. Yu, “Revolutionizing neurosurgery with gpt-4: a leap forward or ethical conundrum?” Annals of Biomedical Engineering, vol. 51, no. 10, pp. 2105–2112, 2023
2023
-
[17]
Sustainable mass transit: Challenges and opportunities in urban public transportation,
T. Abdallah, “Sustainable mass transit: Challenges and opportunities in urban public transportation,” 2023
2023
-
[18]
Urban smart public transport studies: a review and prospect,
M. Xu, T. Liu, S.-P. Zhong, and Y . Jiang, “Urban smart public transport studies: a review and prospect,”J. Transp. Syst. Eng. Inf. Technol., vol. 22, no. 2, pp. 91–108, 2022
2022
-
[19]
A systematic overview of transportation equity in terms of accessibility, traffic emissions, and safety outcomes: From conventional to emerging technologies,
Y . Guo, Z. Chen, A. Stuart, X. Li, and Y . Zhang, “A systematic overview of transportation equity in terms of accessibility, traffic emissions, and safety outcomes: From conventional to emerging technologies,”Transportation research interdisciplinary perspectives, vol. 4, p. ...
2020
-
[20]
Agent-based optimizing match between passenger demand and service supply for urban rail transit network with netlogo,
J. Zhang, “Agent-based optimizing match between passenger demand and service supply for urban rail transit network with netlogo,” IEEE Access, vol. 9, pp. 32 064–32 080, 2021
2021
-
[21]
Flexible route optimization for demand-responsive public transit service,
A. Huang, Z. Dou, L. Qi, and L. Wang, “Flexible route optimization for demand-responsive public transit service,” Journal of Transportation Engineering, Part A: Systems, vol. 146, no. 12, p. 04020132, 2020
2020
-
[22]
To thrive, san antonio must enhance, redefine transportation,
B. Altamirano, J. Paredes, and J. Pawlik, “To thrive, san antonio must enhance, redefine transportation,” https: //www.expressnews.com/opinion/commentary/article/san-antonio-transportation-mobility-planning-18710066. php, 2024, accessed: 2024-06-22
2024
-
[23]
San antonio needs to fund public transit,
C. Hounston, “San antonio needs to fund public transit,” https://trinitonian.com/2023/02/23/ san-antonio-needs-to-fund-public-transit/, 2023, accessed: 2024-06-22
2023
-
[24]
Plato dialogue system: A flexible conversational ai research platform,
A. Papangelis, M. Namazifar, C. Khatri, Y .-C. Wang, P. Molino, and G. Tur, “Plato dialogue system: A flexible conversational ai research platform,”arXiv preprint arXiv:2001.06463, 2020
2001 arXiv
-
[25]
Gpt (generative pre-trained transformer)–a comprehensive review on enabling technologies, potential applications, emerging challenges, and future directions,
G. Yenduri, M. Ramalingam, G. C. Selvi, Y . Supriya, G. Srivastava, P. K. R. Maddikunta, G. D. Raj, R. H. Jhaveri, B. Prabadevi, W. Wang et al., “Gpt (generative pre-trained transformer)–a comprehensive review on enabling technologies, potential applications, emerging challeng...
2024
-
[26]
Bus bunching and bus bridging: What can we learn from generative ai tools like chatgpt?
S. V oß, “Bus bunching and bus bridging: What can we learn from generative ai tools like chatgpt?”Sustainability, vol. 15, no. 12, p. 9625, 2023
2023
-
[27]
Advanced learning technolo- gies for intelligent transportation systems: Prospects and challenges,
R. A. Khalil, Z. Safelnasr, N. Yemane, M. Kedir, A. Shafiqurrahman, and N. Saeed, “Advanced learning technolo- gies for intelligent transportation systems: Prospects and challenges,”IEEE Open Journal of Vehicular Technology, 2024
2024
-
[28]
The role of llms in sustainable smart cities: Applications, challenges, and future directions,
A. Ullah, G. Qi, S. Hussain, I. Ullah, and Z. Ali, “The role of llms in sustainable smart cities: Applications, challenges, and future directions,” arXiv preprint arXiv:2402.14596, 2024
2024 arXiv
-
[29]
From text to transformation: A comprehensive review of large language models’ versatility,
P. Kaur, G. S. Kashyap, A. Kumar, M. T. Nafis, S. Kumar, and V . Shokeen, “From text to transformation: A comprehensive review of large language models’ versatility,”arXiv preprint arXiv:2402.16142, 2024
2024 arXiv
-
[30]
Trafficsafetygpt: Tuning a pre-trained large language model to a domain-specific expert in transportation safety,
O. Zheng, M. Abdel-Aty, D. Wang, C. Wang, and S. Ding, “Trafficsafetygpt: Tuning a pre-trained large language model to a domain-specific expert in transportation safety,”arXiv preprint arXiv:2307.15311, 2023. 9 This work is accepted to AAAI 2025 Workshop on AI for Urban Planning
2023 arXiv
-
[32]
Chatgpt is on the horizon: Could a large language model be all we need for intelligent transportation?
O. Zheng, M. Abdel-Aty, D. Wang, Z. Wang, and S. Ding, “Chatgpt is on the horizon: Could a large language model be all we need for intelligent transportation?” arXiv preprint arXiv:2303.05382, 2023
2023 arXiv
-
[33]
Attention is all you need,
A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, and I. Polosukhin, “Attention is all you need,” Advances in neural information processing systems, vol. 30, 2017
2017
-
[34]
Improving language understanding by generative pre-training,
A. Radford, K. Narasimhan, T. Salimans, I. Sutskeveret al., “Improving language understanding by generative pre-training,” 2018
2018
-
[35]
Better language models and their implications,
A. Radford, J. Wu, D. Amodei, D. Amodei, J. Clark, M. Brundage, and I. Sutskever, “Better language models and their implications,” OpenAI blog, vol. 1, no. 2, 2019
2019
-
[36]
Language models are few-shot learners,
T. Brown, B. Mann, N. Ryder, M. Subbiah, J. D. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell et al., “Language models are few-shot learners,” Advances in neural information processing systems, vol. 33, pp. 1877–1901, 2020
1901
-
[37]
An introduction to large language models: Prompt engineering and p-tuning,
T. Varshney and A. Surla, “An introduction to large language models: Prompt engineering and p-tuning,” https: //developer.nvidia.com/blog/an-introduction-to-large-language-models-prompt-engineering-and-p-tuning/, 2023, accessed: 2024-07-31
2023
-
[38]
What is prompt engineering?
“What is prompt engineering?” https://aws.amazon.com/what-is/prompt-engineering/#:~:text=Prompt% 20engineering%20is%20the%20process,solutions%20to%20generate%20desired%20outputs., 2024, accessed: 2024-07-31. 10
2024
Reviewed August 10, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.