{"id":"fc41ced4-9db9-40d0-aa4a-cd65fdd8db70","arxiv_id":"2501.03287","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"high","formal_verification":"none","parameter_count":4,"one_line_summary":"OpenLKA is an open dataset showing that commercial lane keeping assist systems deviate significantly on sharp curves and in low-contrast, adverse conditions.","lead":"This paper introduces OpenLKA, an open dataset of over 130 hours of real-world driving from 15 vehicle models, including CAN-bus messages, video, and human driving logs. It documents when commercial lane keeping assist systems fail, which is useful for automakers, road planners, and self-driving researchers.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Section 4.2's curvature-deviation result depends on OpenPilot's lane-line estimates as ground truth, with no independent validation; the same perception system is used to define detection failures and then quantify deviations, so the reported LKA errors may be perception artifacts rather than…","rationale":"The paper's dataset and code release are creditworthy, and the VLM annotation validation on 2,000+ images with reported >95% agreement is a positive check on the annotation pipeline. However, the load-bearing empirical claim -- that LKA deviation grows with curvature and reaches about 1.5 m for curvature above 0.05 -- is computed from OpenPilot's lane-line estimates. Since OpenPilot is an aftermarket vision system with its own perception failures in the very scenarios under study, and since no independent validation or missing-data handling is documented, this is the least secure link in the causal chain from sensor reading to safety conclusion. The paper's own statement that lane lines were not detected while simultaneously reporting a lane deviation from those lines makes the measurement circular. If the proposed ground-truth comparison shows OpenPilot error to be small and unbiased, the concern is resolved; if not, every downstream statistic and safety recommendation is at risk. I therefore keep the reader's CONDITIONAL verdict: the dataset can still be valuable, but the quantitative claims must be re-validated with an independent lane-position reference before they are accepted.","tokens_in":20626,"tokens_out":14924,"duration_ms":145789,"concrete_test":"On a stratified sample of about 200 frames from the published OpenLKA data spanning curves, rain, glare, faded markings, and lane transitions, compare OpenPilot's left_laneline/right_laneline values and derived LKA error against an independent reference: human-annotated lane boundaries and/or a surveyed lane center from differential GPS. Report signed bias, RMS error, and whether the error magnitude correlates with curvature or scene condition. If median absolute error exceeds roughly 0.1-0.15 m, or if errors grow in exactly the challenging scenes where large deviations are claimed, then the Section 4.2 regression cannot be attributed to native LKA behavior.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"Section 4.1.2 defines LKA error as the mean of OpenPilot's left_laneline and right_laneline distances and treats it as the vehicle's true lateral position relative to lane center. No calibration or independent reference (differential GPS, surveyed lane geometry, human-annotated lane boundaries) is reported, and the paper does not state how missing values or low detection-probability frames are handled. This matters because OpenPilot is itself a vision-based system that fails in the same conditions under test. The paper's own Section 4.3.1 exposes a concrete contradiction: in the worn-marking case (Fig. 16(d)) the text says the left and right edge lines are not detected, yet it reports a lane deviation reaching 0.8 m. If the lane lines are not detected, the left_laneline/right_laneline values required by the Section 4.1.2 formula are unavailable, so that deviation cannot be computed as specified. Consequently, the Section 4.2 quantitative relationship (slope -8.33; 'curvature larger than 0.05' leading to 'almost 1.5 m' deviation) may be measuring OpenPilot's perception error rather than native LKA's actual lane deviation. This is a circularity in the core measurement, not a disagreement with external consensus.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper presents OpenLKA, an open multi-modal dataset of lane keeping assist (LKA) driving collected from rental vehicles in Tampa, Florida, comprising over 130 hours of CAN-bus data, synchronized video, OpenPilot perception outputs, trajectory data, and GPT-4o-based scene annotations. The authors use the dataset to characterize LKA performance under normal and challenging conditions, reporting that LKA is vulnerable to faint markings and low contrast, struggles in lane transitions, deviates on sharp curves due to torque limits, and rigidly centers the lane without the flexibility of human drivers. The paper also proposes infrastructure recommendations (e.g., a curvature warning threshold of 0.006 and a recommended LKA speed limit of 60.7 mph) and demonstrates a VLM-based planner, iLKA, fine-tuned on human driving data.","tokens_in":20907,"tokens_out":3873,"duration_ms":38518,"significance":"If the measurement methodology is sound, OpenLKA would be a valuable community resource: it combines CAN, video, perception, and VLM-annotated metadata across a diverse fleet, which is rare in the LKA/ADAS literature. The qualitative findings about perception, planning, and control limitations are plausible and actionable for infrastructure planning and LKA development. However, the load-bearing quantitative claims—especially the curvature–deviation regression (slope −8.33, deviation approaching 1.5 m at curvature >0.05) and the Section 5.2 thresholds—rest on lane-deviation measurements derived from OpenPilot's vision system without independent validation, and the regression statistics are not reported. The paper's strength is its dataset and annotation pipeline rather than validated safety conclusions; with additional validation and reframing, the contribution could be significant.","major_comments":[{"comment":"The lane-deviation metric is defined as LKA Error = (Lleft_lane + Lright_lane)/2 using OpenPilot's vision-based left_laneline and right_laneline values. This is the same type of perception system whose failures are the subject of the study, and no independent ground truth (differential GPS, surveyed lane geometry, human-annotated boundaries) is provided anywhere in the paper. Since OpenPilot's detection is known to degrade under the same challenging conditions tested (faint markings, glare, rain), the reported deviations and the Section 4.2 regression slope of −8.33 conflate perception errors with native LKA control errors. The authors should validate the deviation metric against an independent reference on a subset of data, report the number of trips/vehicles, R², confidence intervals, and the handling of frames with missing or low-confidence lane detections, and otherwise reframe the quantitative claims as 'OpenPilot-observed deviations' rather than native LKA performance.","section":"§4.1.2 and §4.2"},{"comment":"The text states that in the worn-marking case 'the left and right edges are not detected due to wear and tear' and simultaneously reports that 'the lane deviation reaches 0.8 at one point.' This is internally inconsistent with the Section 4.1.2 formula, which requires both left_laneline and right_laneline to compute the deviation. If neither line is detected, the LKA Error cannot be computed as specified. The paper must explain how missing or undetected lane lines are handled in the deviation computation and clarify what quantity the 0.8 m value represents.","section":"§4.3.1, Fig. 16(d)"},{"comment":"The recommended thresholds—a curvature warning at 0.006 and a speed limit of 60.7 mph—are derived by fitting Random Forest and XGBoost models to the OpenLKA data and then presented as prescriptive judgments ('we get the judgment that...'). These are descriptive summaries of the dataset, not independent predictions; no cross-validation, uncertainty measures, or external validation are reported. The paper should present these as preliminary data-driven suggestions with appropriate caveats, or provide out-of-sample validation before recommending them to transportation authorities.","section":"§5.2"}],"minor_comments":[{"comment":"The regression in Fig. 14 is described by a single slope (−8.33) without the number of observations, number of vehicles/trips, R², or confidence intervals; given the temporal autocorrelation of driving data, the effective sample size should also be discussed.","section":"§4.2"},{"comment":"The approximation R'(x) = (Rc − ∞)/L in Eq. (10) is not a valid finite-difference expression; the derivation of the torque-rate design principle should be rewritten using a finite transition length and a concrete radius profile.","section":"§5.1, Eq. (10)"},{"comment":"The iLKA demonstration fine-tunes GPT-4o on human steering data and then shows that the fine-tuned model's outputs are close to human outputs; this is a fit-to-training-data demonstration and should be explicitly labeled as such, with no claim of generalization to unseen drivers or scenarios.","section":"§5.3"},{"comment":"The VLM annotation validation reports an accuracy rate of over 95% on 2,000+ images, but the sampling procedure and per-category agreement are not given; a category-wise breakdown or Cohen's kappa would make the reliability claim more interpretable.","section":"§3.4"},{"comment":"There are numerous typographical errors and placeholder artifacts, e.g., 'safty problems issues' (§4.2), 'V olkswagen' (§4.3.3), 'sensoring cmera' (§4.3.2), 'disinterment' (§5.2), 'comparision' (start of Section 4), and 'Figure X' references (§3.4 and §4.3.4).","section":"Throughout"},{"comment":"The dataset size is inconsistently reported: Section 2 states 'over 100 hours', Section 3.3 states 'over 130 hours', and Table 1 states '150+'; the fleet description says 'more than a dozen' while the contribution list says 15 models. These numbers should be reconciled.","section":"§2 and §3.3"}],"recommendation":"major_revision","confidential_remarks":"The dataset and annotation pipeline are the strongest parts of the submission, and the qualitative failure-mode analysis is likely to be useful to the community. However, the central quantitative claim—the curvature–deviation relationship—is built on an unvalidated vision-based measurement with an internal inconsistency in one of the supporting examples, and the regression statistics are not reported. These issues are fixable within the manuscript's scope by adding independent validation, reporting proper statistics, and reframing the thresholds and iLKA demonstration as preliminary. I would not recommend rejection, but the authors need to address the measurement-validity issue before the safety-related conclusions can be accepted."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Two things to know. First, the dataset is real and fills a gap: 130+ hours, 7000+ km, 15 vehicle models, with CAN messages, synchronized video, VLM annotations, and human driving logs from Open Comma. That's a useful resource for ADAS evaluation and infrastructure planning, and it's genuinely new compared to comma2k19, BDD100K, nuScenes. Second, the quantitative claims about lane deviation—the slope of -8.33 and the 'almost 1.5m' deviation on sharp curves—rest on an unvalidated measurement assumption: they compute LKA error from OpenPilot's left_laneline/right_laneline values and treat that as ground truth for the vehicle's true lane position. OpenPilot is itself a vision system that fails under the same conditions the paper is testing, so the reported deviations may be perception artifacts rather than actual LKA tracking errors. That's not a minor quibble; it's the foundation of Section 4.2.\n\nThe paper does several things well. The qualitative findings—LKA struggles with faint markings, lane transitions, torque limits on sharp curves, and rigid lane-centering—are plausible and consistent with prior assessments. The VLM annotation pipeline, with a reported 95% agreement on 2000+ images, is a nice addition, and the iLKA demonstration is a reasonable proof of concept, though it needs broader validation.\n\nThe soft spots are real but fixable. The biggest is the missing independent validation: no DGPS, surveyed lane geometry, or human-annotated lane boundaries to check OpenPilot's estimates. The paper also doesn't say how it handles frames where OpenPilot fails to detect a lane line. There's a concrete internal contradiction in Section 4.3.1 (Fig. 16(d)): the text says the left and right edge lines are not detected due to wear, yet reports a lane deviation of 0.8m—if the lines aren't detected, the formula in Section 4.1.2 can't produce that number. The statistics are under-reported: the -8.33 slope has no R², confidence intervals, or sample size, and the thresholds (0.006 curvature, 60.7 mph) are presented without uncertainty. These are descriptions of the dataset, not independent predictions, which is fine for an empirical study, but they should be labeled as such.\n\nI'd send this to peer review. The dataset deserves referee time, and the methodological gaps are addressable in revision. The right outcome is likely a major revision that either validates the measurement or softens the quantitative claims to speculation. The paper is for researchers in ADAS testing, infrastructure planning, and driving datasets; they'll get value from the resource even if they don't trust the headline numbers.","headline":"Valuable new LKA dataset, but the headline deviation numbers rest on an unvalidated vision-based measurement that needs independent checking before they're trusted.","tokens_in":21471,"tokens_out":3911,"would_cite":true,"duration_ms":32654,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper introduces OpenLKA, an open dataset, and uses it to argue that commercial lane-keeping assist fails predictably on sharp curves, low-contrast markings, and lane transitions.","keywords":["lane keeping assist","autonomous vehicles","CAN bus","open dataset","lane departure","road curvature","vision-language model","empirical evaluation"],"falsifier":"Collect independent lane-position ground truth on a subset of the same Tampa routes, for example centimeter-level GNSS/INS with mapped lane centerlines or manually annotated video, and compare it with the OpenPilot-derived LKA error. If the deviations at curvature above 0.05 are much smaller or uncorrelated with the fitted slope of -8.33, the paper's central quantitative claim is refuted.","tokens_in":20419,"feed_emoji":"🚗","tokens_out":6979,"duration_ms":62653,"temperature":0.7,"pith_summary":"This paper introduces OpenLKA, an open multimodal dataset of over 130 hours of real-world driving from 15 vehicle models, collected in Tampa with synchronized CAN bus messages, front-view video, and vision-based lane detection. Using this dataset, the authors try to establish that commercial lane-keeping assist systems have separable and recurring failure modes: perception loses faint or low-contrast markings, planning goes wrong at lane merges and splits, control cannot deliver enough steering torque on sharp curves, and the systems rigidly chase the lane center instead of adapting like human drivers. The paper reports a quantitative relationship between curvature and lane deviation, with a fitted slope of -8.33 and deviations approaching 1.5 meters once curvature exceeds 0.05 per meter. A sympathetic reader would care because the dataset and its findings give road agencies, automakers, and self-driving researchers a concrete, public evidence base for improving markings, curve design, and LKA algorithms.","feed_headline":"Lane-keeping systems drift 1.5 m on sharp curves, tests show","feed_subtitle":"Open dataset from 15 car models pinpoints perception, planning, and steering-torque limits in commercial LKA.","key_machinery":"The central mechanism is the OpenLKA collection pipeline built around a Comma Three-X device mounted at the windshield center and run in dash-cam mode, so it passively reads CAN messages without touching the vehicle's controls. Openpilot's vision detection supplies the distances to the left and right lane lines, and the paper defines LKA error as $E = (L_{\\text{left}} + L_{\\text{right}})/2$, a continuous lane-deviation measure. The quantitative argument is carried by the curvature–deviation regression, with fitted slope $-8.33$, and by the lateral-acceleration identity $a_{\\text{lateral}} = \\kappa v^2 - \\text{roll}\\cdot g$, which links path curvature $\\kappa$, speed $v$, and roll angle to the steering torque the LKA must produce.","core_discovery":"On the paper's own terms, the central discovery is that commercial LKA performance degrades in a predictable, module-by-module way. In perception, low-contrast pavement and marking combinations, heavy rain, glare, and worn markings make lane detection flicker or disappear. In planning, lane transitions such as merges, diverges, and intersections make the system lock onto the wrong line or disengage. In control, torque-limited steering cannot supply the lateral acceleration required on sharp turns, so the vehicle drifts from lane center. The paper quantifies this control failure by regressing lane deviation against curvature, obtaining a slope of -8.33, and states that beyond curvature 0.05 the deviation reaches almost 1.5 meters. It further argues that LKA's rigid lane-centering behavior contrasts with human corner-apexing and obstacle-avoidance strategies, and it demonstrates that a vision-language model fine-tuned on human driving logs produces steering outputs closer to human choices.","pith_inferences":["Editorial inference: because the lane-deviation measurement comes from Openpilot's own vision, the reported errors may combine native LKA control error with measurement-system perception error; an independent ground-truth comparison would be needed to separate the two.","Editorial inference: the torque-derivative equations imply a testable road-design rule: transition-curve length, not just final radius, controls LKA failure, so shortening spiral transitions at constant radius should increase deviation even when curvature at the apex is unchanged.","Editorial inference: the synchronized CAN-video-human dataset could support a departure early-warning model that predicts the 1.5-meter regime before it occurs, something the paper only gestures at through its disengagement prediction models."],"forward_implications":["Roadway agencies can use the dataset and fitted models to set marking-maintenance priorities, with model results recommending LKA deviation warnings where curvature exceeds 0.006 and speed guidance near 60.7 mph.","LKA developers gain a public, synchronized CAN-plus-video corpus of over 130 hours from 15 models, covering perception, planning, and control failure modes, enabling standardized benchmarking.","A fine-tuned vision-language model with chain-of-thought prompting outputs steering decisions closer to human drivers than the base model, suggesting a concrete path to more adaptive lane-keeping.","Because deviation grows with curvature and reaches about 1.5 meters above curvature 0.05, vehicles with torque-limited steering may predictably depart the lane on sharp curves unless speed is reduced or torque authority increased."],"supporting_citations":[{"why":"Supplies the Comma Three-X device whose dash-cam mode passively records the CAN bus and synchronized video used throughout the study.","marker":"[21]"},{"why":"Provides the prior pavement-marking and automated-vehicle-readiness findings that the paper's perception-failure results extend.","marker":"[10]"},{"why":"Offers prior limited LDW/LKA response testing that OpenLKA scales up.","marker":"[16]"},{"why":"Documents safety issues of level 2 automation on curved sections, the result this paper quantifies with curvature data.","marker":"[17]"},{"why":"Establishes the real-world robustness testing approach for commercial lane departure warning systems that motivates the campaign.","marker":"[19]"},{"why":"Supplies the critical-scenario identification method that shapes the test route and scenario design.","marker":"[20]"}],"fun_headline_variants":["LKA drifts 1.5 m on sharp turns, new open dataset shows","Open dataset exposes lane-keeping failures on faint lines and merges","Lane-keeping systems lose the lane at curvature 0.05, data finds","Commercial LKA struggles: sharp bends, faint marks, and merges","New LKA dataset reveals torque limits cause 1.5 m drift"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The whole measurement depends on the aftermarket vision system's lane detection being an unbiased ruler for lane position; if that detection is biased or fails on these roads, the reported deviations describe the measuring device, not the car's lane-keeping system.","fun_headline_variants_meta":{"raw":{"variants":["LKA drifts 1.5 m on sharp turns, new open dataset shows","Open dataset exposes lane-keeping failures on faint lines and merges","Lane-keeping systems lose the lane at curvature 0.05, data finds","Commercial LKA struggles: sharp bends, faint marks, and merges","New LKA dataset reveals torque limits cause 1.5 m drift"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000202,"raw_usage":{"total_tokens":1445,"prompt_tokens":1074,"completion_tokens":371,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":690,"completion_tokens_details":{"reasoning_tokens":270}},"tokens_in":690,"tokens_out":371,"duration_ms":4232,"temperature":1.0,"reasoning_tokens":270,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T22:04:43.869761+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Collect independent lane-position ground truth on a subset of the same Tampa routes, for example centimeter-level GNSS/INS with mapped lane centerlines or manually annotated video, and compare it with the OpenPilot-derived LKA error. If the deviations at curvature above 0.05 are much smaller or uncorrelated with the fitted slope of -8.33, the paper's central quantitative claim is refuted.","supporting_citations":[{"cited_title":"Comma three-x","cited_arxiv_id":null,"evidence_quote":"Supplies the Comma Three-X device whose dash-cam mode passively records the CAN bus and synchronized video used throughout the study."},{"cited_title":"Assessing pavement markings for automated vehicle readiness","cited_arxiv_id":null,"evidence_quote":"Provides the prior pavement-marking and automated-vehicle-readiness findings that the paper's perception-failure results extend."},{"cited_title":"Testing and analysis of lane departure warning and lane keeping assist system response","cited_arxiv_id":null,"evidence_quote":"Offers prior limited LDW/LKA response testing that OpenLKA scales up."},{"cited_title":"Evaluation of level 2 automated driving safety on curved sections","cited_arxiv_id":null,"evidence_quote":"Documents safety issues of level 2 automation on curved sections, the result this paper quantifies with curvature data."},{"cited_title":"Testing the robustness of commercial lane departure warning systems","cited_arxiv_id":null,"evidence_quote":"Establishes the real-world robustness testing approach for commercial lane departure warning systems that motivates the campaign."},{"cited_title":"Identifying critical test scenarios for lane keeping assistance system using analytic hierarchy process and hierarchical clustering","cited_arxiv_id":null,"evidence_quote":"Supplies the critical-scenario identification method that shapes the test route and scenario design."}],"review_version":1}