{"id":"d3a16507-2d84-4188-a8d8-307a4be1b480","arxiv_id":"2606.07486","paper_version":1,"verdict":"UNVERDICTED","confidence":"LOW","novelty_score":5.0,"correctness_risk":"high","formal_verification":"none","parameter_count":0,"one_line_summary":"OpenPath is a supervisor-specialist multi-agent system combining LLM intent parsing with classical route optimization for personalized, accessible, multi-stop urban trip planning and city-scale accessibility measurement.","lead":"The paper describes OpenPath, a multi-agent system that uses LLM agents to understand user requests and classical algorithms to plan personalized multi-stop trips while enforcing wheelchair accessibility. A smart generalist might read it to see how AI can help make urban transportation more equitable and reveal infrastructure gaps in cities like New York.","discovery_kind":"new_application","skeptic_critique":{"model":"grok-4.3","headline":"NYC gap quantification assumes curated accessibility data is complete and error-free","rationale":"Reader's weakest assumption (data + LLM accuracy) directly targets the measurement-instrument claim; the data-completeness issue is the most load-bearing technical precondition for that claim to be reliable. Full-text details on data sources would not remove the need for external validation.","tokens_in":1661,"tokens_out":291,"duration_ms":10289,"concrete_test":"Sample 200 random NYC locations from the dataset used in the paper; cross-check each against NYC DOT ADA compliance records and recent satellite/street-view imagery for presence/absence of barriers; if disagreement rate >15% on accessibility labels, recompute the job-accessibility delta and report change in headline numbers.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The strongest claim—that OPENPATH applied to NYC reveals substantial ADA gaps and quantifies their effect on wheelchair job accessibility—requires that the curated mobility/accessibility dataset accurately captures all relevant infrastructure (sidewalks, ramps, transit features) without systematic omissions or labeling errors. The architecture delegates route optimization to classical algorithms over this fixed data; any incompleteness directly produces spurious gaps or accessibility deltas that the supervisor-specialist agents cannot detect or correct. The abstract and design description provide no independent validation step (e.g., ground-truth sampling or comparison to official NYC ADA inventories) for the data layer.","agreement_with_reader":"agree"},"referee_report":{"model":"grok-4.3","summary":"The manuscript presents OPENPATH, a supervisor-specialist multi-agent architecture in which LLM agents handle natural-language parsing, intent classification, and orchestration while classical algorithms perform route optimization over curated mobility and accessibility datasets. The system is claimed to support personalized preferences, multi-stop itineraries, and strict end-to-end wheelchair accessibility. Applied to New York City, the same framework is said to reveal substantial ADA infrastructure gaps and to quantify their effect on job accessibility for wheelchair users.","tokens_in":1767,"tokens_out":538,"duration_ms":14812,"significance":"If the empirical claims are substantiated, the hybrid LLM-classical design offers a concrete way to enforce hard accessibility constraints that pure LLM planners often violate, while the city-scale measurement use-case could supply a reproducible method for accessibility auditing. The absence of free parameters in the optimization layer and the explicit separation of concerns are strengths that would distinguish the work from purely generative approaches.","major_comments":[{"comment":"Abstract: the claim that OPENPATH 'reveals substantial ADA infrastructure gaps and quantifies their effect on job accessibility for wheelchair users' is presented without any reported quantitative results, error rates, baseline comparisons, or ground-truth validation of the curated accessibility layer. Because the NYC analysis is the central empirical contribution, this omission renders the magnitude and validity of the reported gaps impossible to assess.","section":"Abstract"},{"comment":"System description (route-optimization module): the architecture delegates all accessibility enforcement to classical algorithms operating on a fixed curated dataset; any systematic omissions or labeling errors in sidewalks, ramps, or transit features propagate directly into spurious gap measurements. No independent validation step (sampling, comparison to official NYC ADA inventories, or cross-check against alternative data sources) is described, which is load-bearing for the city-scale accessibility claim.","section":"System Architecture"}],"minor_comments":[{"comment":"Notation for the supervisor versus specialist agent roles and the hand-off protocol between LLM and classical components should be made explicit, preferably with a diagram or pseudocode listing the exact message types.","section":null},{"comment":"The manuscript would benefit from a dedicated limitations subsection that explicitly discusses the dependence on curated data completeness.","section":null}],"recommendation":"major_revision","confidential_remarks":"The abstract supplies no numerical results whatsoever for the NYC case study; if the full manuscript likewise lacks tables or figures reporting accessibility deltas, job-accessibility changes, or data-validation statistics, the paper would be better suited to a workshop or demo track than to a full archival systems journal."},"author_rebuttal":{"model":"grok-4.3","summary":"We thank the referee for the constructive feedback highlighting the need for stronger substantiation of the NYC accessibility claims. We address each major comment below and will revise the manuscript accordingly.","responses":[{"response":"We agree that the abstract summarizes the NYC findings at too high a level. The full manuscript contains quantitative results in the evaluation section (specific percentages of inaccessible infrastructure segments, estimated job-accessibility deltas for wheelchair users, and comparisons against non-accessible baselines). We will revise the abstract to report these key figures explicitly and add a sentence on the scale of the analysis. This change will allow readers to assess the claims without altering the core contribution.","revision_made":"yes","referee_comment":"[Abstract] Abstract: the claim that OPENPATH 'reveals substantial ADA infrastructure gaps and quantifies their effect on job accessibility for wheelchair users' is presented without any reported quantitative results, error rates, baseline comparisons, or ground-truth validation of the curated accessibility layer. Because the NYC analysis is the central empirical contribution, this omission renders the magnitude and validity of the reported gaps impossible to assess."},{"response":"The referee is correct that the manuscript does not describe an independent validation procedure beyond the initial curation from public sources. We will add a dedicated subsection on data provenance and quality control, including any sampling or cross-checks that were performed, and a limitations paragraph that explicitly discusses the risk of labeling errors and their potential impact on the reported gaps. This revision will make the empirical claims more transparent.","revision_made":"yes","referee_comment":"[System Architecture] System description (route-optimization module): the architecture delegates all accessibility enforcement to classical algorithms operating on a fixed curated dataset; any systematic omissions or labeling errors in sidewalks, ramps, or transit features propagate directly into spurious gap measurements. No independent validation step (sampling, comparison to official NYC ADA inventories, or cross-check against alternative data sources) is described, which is load-bearing for the city-scale accessibility claim."}],"tokens_in":1347,"tokens_out":430,"duration_ms":15863,"standing_objections":[]},"desk_editor":{"model":"grok-4.3","letter":"The main takeaway is that this paper describes a supervisor-specialist agent setup where LLMs handle natural-language parsing, intent classification, and orchestration while classical algorithms do the route optimization over fixed mobility and accessibility data. That division lets the system manage personalized preferences and enforce strict accessibility rules without routing everything through the language model.\n\nIt does a reasonable job spelling out why this split is useful for heterogeneous user needs and multi-stop itineraries. Turning the same system into a measurement tool for city-scale accessibility analysis, specifically the effect on wheelchair users' job access in New York, is a straightforward extension that follows from the design.\n\nThe soft spot is the data layer behind the NYC findings. The claim that the system reveals substantial ADA infrastructure gaps and quantifies their impact assumes the curated dataset is complete and free of systematic omissions or labeling errors. The abstract gives no indication of ground-truth sampling, comparison to official inventories, or any other check on data quality. Any incompleteness would produce spurious gaps that the agents cannot detect or correct, since optimization runs over the fixed data. If the full paper contains no such validation step, that part of the result is weaker than presented.\n\nThis is aimed at researchers working on hybrid agent systems for transportation planning or urban accessibility analysis. A reader already interested in LLM-classical combinations for constrained routing could extract the design rationale. It deserves peer review so referees can examine the implementation details and any quantitative backing for the measurement claims.","headline":"OPENPATH's LLM-plus-classical split for accessible multi-stop planning is a clean architecture, but the NYC ADA gap quantification rests on an unvalidated data assumption.","tokens_in":2236,"tokens_out":367,"would_cite":false,"duration_ms":14335,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"grok-4.3","headline":"A supervisor-specialist multi-agent system plans personalized multi-stop urban trips while enforcing wheelchair accessibility and measuring city-wide ADA gaps.","keywords":["multi-agent systems","urban trip planning","wheelchair accessibility","ADA compliance","LLM agents","route optimization","NYC transportation","job accessibility"],"falsifier":"A wheelchair-accessible trip request for which the system returns a route that violates accessibility rules, or an NYC job-accessibility calculation that diverges from independent census or transit data.","tokens_in":2568,"feed_emoji":"♿","tokens_out":594,"duration_ms":16766,"temperature":0.7,"pith_summary":"OpenPath combines LLM agents that parse natural language requests and classify intent with classical algorithms that optimize routes over curated mobility and accessibility data. The architecture produces itineraries that respect user preferences, handle multiple stops, and meet strict accessibility rules when requested. Applied to New York City, the same system identifies substantial gaps in ADA-compliant infrastructure and calculates how those gaps reduce job accessibility for wheelchair users. A reader would care because the work shows one framework can serve both individual trip planning and quantitative analysis of transportation equity.","feed_headline":"Multi-agent system plans accessible trips and measures NYC wheelchair gaps","feed_subtitle":"Supervisor LLM agents direct classical route algorithms to honor preferences and accessibility while exposing infrastructure effects on empl","key_machinery":"Supervisor-specialist multi-agent architecture that uses LLM agents for intent classification and orchestration while classical algorithms perform route optimization over accessibility data.","core_discovery":"OpenPath divides labor so that LLM agents parse input, classify intent, and orchestrate execution while classical algorithms compute routes over curated mobility and accessibility data. This separation produces trip plans that honor heterogeneous preferences and enforce accessibility constraints. When run on NYC data the system functions as a measurement instrument that reveals ADA infrastructure gaps and quantifies their effect on job accessibility for wheelchair users.","pith_inferences":["The measurement approach could be repeated in other cities that maintain comparable mobility and accessibility datasets.","Adding real-time accessibility updates would allow the system to reflect construction or temporary closures.","The framework could be extended to other mobility constraints such as visual impairment or transit reliability preferences."],"forward_implications":["Natural-language requests for multi-stop trips can be translated into routes that respect stated preferences and accessibility constraints.","City-scale runs can expose infrastructure gaps and their quantitative impact on employment access for wheelchair users.","The same data pipeline supports both per-user planning and aggregate equity measurements.","Strict accessibility enforcement is maintained by separating LLM orchestration from classical optimization steps."],"fun_headline_variants":["Agents plan accessible trips and expose NYC wheelchair gaps","Agent system plans accessible trips and measures NYC wheelchair gaps","Supervisor agents plan personalized accessible trips and NYC gaps","Framework uses agents for accessible trips and quantifies NYC gaps"],"cache_read_input_tokens":2112,"weakest_assumption_plain":"Curated mobility and accessibility data plus LLM-based intent classification are accurate and complete enough to enforce strict accessibility requirements without planning errors.","fun_headline_variants_meta":{"raw":{"variants":["Agents plan accessible trips and expose NYC wheelchair gaps","Agent system plans accessible trips and measures NYC wheelchair gaps","Supervisor agents plan personalized accessible trips and NYC gaps","Framework uses agents for accessible trips and quantifies NYC gaps"]},"model":"grok-4.3","cost_usd":0.014235,"raw_usage":{"total_tokens":6112,"prompt_tokens":621,"num_sources_used":0,"completion_tokens":54,"cost_in_usd_ticks":142349500,"prompt_tokens_details":{"text_tokens":621,"audio_tokens":0,"image_tokens":0,"cached_tokens":256},"completion_tokens_details":{"audio_tokens":0,"reasoning_tokens":5437,"accepted_prediction_tokens":0,"rejected_prediction_tokens":0}},"tokens_in":621,"tokens_out":54,"duration_ms":30539,"temperature":1.0,"reasoning_tokens":5437,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-06-27T20:51:46.961305+00:00","model_set":{"reader":"grok-4.3"},"falsifier":"A wheelchair-accessible trip request for which the system returns a route that violates accessibility rules, or an NYC job-accessibility calculation that diverges from independent census or transit data.","supporting_citations":[],"review_version":1}