{"id":"748e404a-5b78-4590-8c14-a0ec3fcc8e53","arxiv_id":"2508.21271","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":3.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"On a visually complex simulated track, a slim 3D CNN beat LSTM and GRU recurrent models on average lap time, but the model choice was made on the same test track.","lead":"This paper tests whether 3D convolutional neural networks drive a simulated miniature car faster than recurrent networks. On the harder track, the 3D CNN averaged about 35 seconds per lap versus about 36 seconds for the RNNs, but the winning model was selected using that same track.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Mini Monaco was used both to select the 3D CNN variant (ablation, Table II) and to report its headline advantage (Table I), so the comparison is a selection artifact unless a separate held-out track/laps confirm it.","rationale":"Agree with reader's weakest_assumption: the most load-bearing flaw is using Mini Monaco both to select and to evaluate the 3D CNN. The RNN baselines were not provided the same model-selection opportunity on the test track, so the comparison is asymmetric. If the selection were done on a validation set and the final comparison on a held-out set, we could know whether the advantage is real. The Generated Track result already suggests the advantage is track-specific. We do not see a more fundamental problem: the architecture and experiments are described clearly enough to be re-run once the selection protocol is fixed. Therefore the reader's CONDITIONAL verdict stands unchanged.","tokens_in":7164,"tokens_out":7386,"duration_ms":70083,"concrete_test":"Re-run the Mini Monaco experiments with a strict split of the 30 lap trials: use the first 20 laps of each model for the ablation/selection, then evaluate the selected 3D CNN variant on the remaining 10 held-out laps and compare to the RNNs' held-out laps. If the selected 3D CNN's held-out mean is not faster than the RNNs by more than the standard error of the difference, the headline advantage is a selection artifact. (Additionally, reconcile Table I vs Table II by recomputing lap times from raw simulator logs for all variants.)","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim—3D CNN is faster than RNNs on the Mini Monaco task—rests on the Mini Monaco lap times in Table I. But Section IV-C states that the compared model was selected 'based on the results' of an ablation study run on this same track: 'we selected 3D CNN Modified version with one 3D convolutional layer less which was able to outperform the other structures.' The RNN models were not similarly selected. Thus the reported comparison is between a 3D CNN whose architecture was chosen by optimizing on the very test condition used for the claim and off-the-shelf RNNs. The advantage (34.10/35.10 vs 36.24/36.56) is therefore not an unbiased estimator of the model class's performance; selecting the best of three variants on the evaluation track inflates its apparent speed. This is not a hypothetical: on the other track (Generated Track), the same 3D CNN (14.58 s) is not faster than RNN Modified (14.45 s), so the claimed advantage does not generalize beyond the track used for selection. Additional inconsistencies (Table I 35.10 vs Table II 34.10/35.56; Section IV-A saying the 3D CNN Modified 'underperformed' on Mini Monaco) make it unclear even which model's time is reported, but the structural selection problem is independent of those numerical discrepancies.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The manuscript presents a behavior-cloning comparison between 3D CNN and RNN autopilot models in the DonkeyCar simulator, using two tracks (Mini Monaco and a simpler Generated Track). Standard and modified variants are trained; the modified 3D CNN adds residual blocks, LeakyReLU, and batch normalization, while the modified RNN replaces LSTM with GRU. Performance is measured as average lap time and standard deviation over 30 laps per model/track, with track deviations and response speed declared but not reported. The paper's headline claim is that the modified 3D CNN achieves the fastest average lap time on Mini Monaco (Table I: 35.10 s vs 36.24 s and 36.56 s for RNNs). An ablation study on Mini Monaco selects a 3D CNN variant with one fewer 3D convolutional layer as the final model.","tokens_in":7504,"tokens_out":3944,"duration_ms":40481,"significance":"If the result is robust, the paper would provide a useful, inexpensive comparison of spatiotemporal 3D convolutions versus recurrent temporal modeling for miniature autonomous-car control, with practical implications for embedded deployment. The study's strengths are its use of two visually distinct environments, 30 evaluation laps per model, explicit standard deviations, and an ablation of 3D CNN depth. However, the empirical support is currently compromised by model selection on the same track used for the headline comparison and by internal numerical contradictions. These issues are fixable with a clean held-out evaluation and consistent reporting; they do not invalidate the overall experimental design, but they must be resolved before the comparative claim can be accepted.","major_comments":[{"comment":"Inconsistent reporting of the winning model. Table I lists '3D CNN Modified' at 35.10 s on Mini Monaco; Table II lists '3D CNN Modified' at 35.56 s and the selected variant (one layer less) at 34.10 s. Section IV-A additionally states that the 3D CNN Modified 'underperformed on Mini Monaco.' The reader cannot tell which architecture produced the headline time, or which of 34.10/35.10/35.56 is the result of the final model. Please reconcile the tables and narrative and report the selected model's time unambiguously.","section":"§IV-C, Tables I and II"},{"comment":"Model selection on the evaluation track biases the comparison. The ablation that chose 'one 3D convolutional layer less' was conducted on Mini Monaco, and Table I then reports the 3D CNN advantage on Mini Monaco. This is not independent evidence: selecting the best of three variants on the same track inflates the apparent improvement. On Generated Track the same model is not faster than RNN Modified (14.58 vs 14.45 s). Confirm the result by evaluating the preselected architecture on a held-out track or independent laps, or by reporting all variants with a selection-aware comparison.","section":"§IV-C, Table I"},{"comment":"Two of the four declared metrics are absent. Track Deviations and Response Speed are defined but never reported; the abstract and conclusion appeal to 'driving consistency' without presenting those data. Moreover, lap-time differences of about 0.5–1.5 s with per-lap standard deviations of ~0.5 s over 30 laps are not shown to be statistically significant. Please provide the missing metric values and at least a paired test or per-lap distributions for the central comparison.","section":"§III-D, Table I"}],"minor_comments":[{"comment":"The paper says RGB-D information is used, but the camera description mentions RGB plus a depth channel; clarify whether depth is an input to the networks or only collected.","section":"Abstract, §III-C"},{"comment":"The sentence 'It was synchronized with the control cycle A total...' is incomplete; please revise.","section":"§III-C"},{"comment":"The batch-normalization equation is garbled in the PDF; check the math notation.","section":"§III-A, Eq. (6)"},{"comment":"'1 layers less' should be 'one layer less'; row labels for the selected variant should be consistent with Table I.","section":"Table II"},{"comment":"Figure references such as 'removing some layers 4, (12, c)' and '5, (12, d)' need correction.","section":"§IV-C"},{"comment":"The paper would benefit from reporting hyperparameters, random seeds, and code/data availability for reproducibility.","section":"General"}],"recommendation":"major_revision","confidential_remarks":"This is a work-in-progress empirical study. My main concern is the selection/evaluation overlap, not novelty. I support major revision rather than rejection because the authors can address the concern with additional experiments and a corrected report. Please ensure that the '3D CNN Modified underperformed on Mini Monaco' sentence in Section IV-A is resolved; as written, it contradicts the paper's central result."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nThe short version: this is a small, honest behavior-cloning comparison on the DonkeyCar simulator, and the direct 3D CNN versus RNN head-to-head is new. But the headline result doesn't hold up under the paper's own methodology: the best 3D CNN variant was selected by an ablation on Mini Monaco, then the paper claims that variant is fastest on Mini Monaco. That is test-set model selection. The numbers are also muddled: Table I reports 35.10 s for the modified 3D CNN on Mini Monaco, Table II reports 35.56 s for the same model, and the selected one-layer-less variant is 34.10 s. Section IV-A even says the modified 3D CNN 'underperformed' on Mini Monaco. So it is unclear which exact model is being claimed as best.\n\nWhat's genuinely there: a clean setup of two tracks, 30 laps per model, standard deviations reported, and an ablation that adds/removes one 3D layer. That is a useful datapoint showing more layers can hurt. The paper also correctly notes that the modified 3D CNN overfits on the simpler Generated Track, while the RNNs are more stable there. Those observations are plausible and worth reporting.\n\nThe soft spots beyond selection: no significance tests (a 1 s difference over 30 laps could be noise), no code or data release, and two declared metrics—track deviations and response speed—are never shown. The authors cite the relevant prior work and keep the abstract's claim to 'promising results,' which is fair.\n\nThis is a workshop-grade empirical study, not a major advance. The comparison is new in a narrow sense, but the central claim is not established. It could be fixed by reserving a held-out track or lap set for final evaluation, or by reporting all variants without selecting the best after the fact. As it stands, I'd treat the 3D CNN advantage as unproven.\n\nWho gets value: people working on miniature autonomous cars who want a quick survey of what happens when you swap LSTM for GRU or add/remove 3D layers. The ablation result alone might be worth a footnote.\n\nMy recommendation: send it to peer review, because a serious referee can force the authors to address the selection issue and the numeric inconsistencies. If the advantage holds on a track not used for selection, this becomes a solid small contribution. I would not cite it until that is fixed.","headline":"A useful ablation, but the headline 3D CNN advantage is compromised by selecting the best variant on the same Mini Monaco track used for the claim.","tokens_in":8010,"tokens_out":2837,"would_cite":false,"duration_ms":24646,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A behavior-cloning 3D CNN with residual blocks and batch normalization completes Mini Monaco in 35.10 s on average, beating both an LSTM-based RNN (36.24 s) and a GRU-based RNN (36.56 s).","keywords":["autonomous driving","mini autonomous car","3D convolutional neural network","recurrent neural network","behavior cloning","RGB-D perception","lap-time evaluation","ablation study"],"falsifier":"Run the default RNN, modified RNN, and the selected 3D CNN on 30 fresh Mini Monaco laps that were never used when choosing the 3D CNN's layer count, then compare mean lap times with their standard deviations; if the 3D CNN mean is not lower than both RNN means, the reported superiority is an artifact of selection.","tokens_in":7077,"feed_emoji":"🏎️","tokens_out":10567,"duration_ms":100098,"temperature":0.7,"pith_summary":"The paper claims that a behavior-cloned 3D convolutional neural network can drive a miniature autonomous car around a visually complex simulated track faster than recurrent neural networks. Using front-camera color-plus-depth image sequences, the authors train default and modified versions of each architecture family and compare average lap times over 30 laps on two tracks. Their headline result is 35.10 seconds per lap on Mini Monaco for the modified 3D CNN, against 36.24 seconds for the LSTM-based RNN and 36.56 seconds for the GRU-based RNN. An ablation then shows that removing one 3D convolutional layer improves the same model to 34.10 seconds, while adding a layer causes overfitting. The paper argues that temporal convolutions with residual blocks and batch normalization are a viable alternative to recurrent policies, especially on tracks with rich visual detail.","feed_headline":"3D CNN beats both RNNs on Mini Monaco: 35.1 s vs 36.2/36.6 s","feed_subtitle":"A residual-block 3D CNN with batch norm posts 35.10 s per lap; the one-layer-leaner version hits 34.10 s.","key_machinery":"The load-bearing object is the modified 3D CNN: three-dimensional convolutions slide over both space and time, so a short stack of front-camera color-and-depth frames is mapped directly to steering and throttle commands without a recurrent state. Two modifications carry the argument. The residual identity y = F(x, {W_i}) + x lets gradients skip layers during training, and batch normalization standardizes intermediate activations, together making the deeper temporal stack trainable on datasets of about ten thousand images. The comparison mechanism is behavior cloning: recorded joystick commands paired with camera frames train the networks, and 30-lap average lap time is the deciding metric. T","core_discovery":"The central claim is that a 3D CNN consuming short stacks of RGB-D frames can outperform RNN-based steering policies on a behavior-cloning task in a simulated miniature-car environment. The authors report that their modified 3D CNN, a standard 3D CNN augmented with residual blocks, LeakyReLU activations, and batch normalization, averaged 35.10 s per lap on Mini Monaco over 30 laps, beating the default LSTM-based RNN (36.24 s) and the GRU-based modified RNN (36.56 s). They further report an ablation in which the version with one fewer 3D convolutional layer is fastest on the same track (34.10 s), while the version with an extra layer overfits. On the simpler Generated Track, the GRU-based RNN","pith_inferences":["The authors leave implicit that the depth channel is never ablated, so whether the 3D CNN's edge comes from temporal structure or from depth information remains untested; an RGB-only version of the same model would isolate the source.","Because the leaner model won, even smaller temporal windows or factorized spatiotemporal convolutions might preserve the lap-time benefit while reducing computation for embedded deployment.","A practical decision rule suggested by the paper but not stated: use a recurrent policy on visually sparse tracks and a lean 3D CNN on visually cluttered ones.","The same methodology transfers naturally to physical miniature cars, since the simulator is designed for that transfer and a fixed-size temporal window avoids maintaining recurrent state during deployment."],"forward_implications":["A 3D CNN with residual connections and batch normalization can serve as a complete behavior-cloning autopilot, with no recurrent layer needed to achieve the fastest lap times on a visually complex track.","On such tracks, a smaller 3D CNN can beat a larger one: the one-layer-leaner variant was fastest and the one-layer-heavier variant overfit.","The architecture ranking is environment-dependent: on the simpler track the GRU-based RNN was slightly faster, so the 3D CNN advantage does not generalize to low-complexity tracks.","Lap-time variability stayed small across 30 laps for all models, so the observed speed differences are not large relative to run-to-run consistency."],"supporting_citations":[{"why":"Supplies the simulator environment used to generate the RGB-D training data and run the 30-lap evaluations.","marker":"[8]"},{"why":"Precedent applying 3D CNNs to miniature autonomous car racing in simulation, which the proposed method builds on.","marker":"[13]"},{"why":"Provides the recurrent-image-steering design that the RNN baseline is adapted from.","marker":"[14]"},{"why":"Defines the LSTM gating mechanism used in the default RNN baseline.","marker":"[15]"},{"why":"Defines the GRU update and reset gates used in the modified RNN baseline.","marker":"[16]"},{"why":"Supplies the residual-block identity used to modify the 3D CNN.","marker":"[18]"},{"why":"Supplies the batch-normalization transform used to stabilize the modified 3D CNN.","marker":"[19]"},{"why":"Provides the open-source driving platform whose model scaffolds and simulator configuration the experiments use.","marker":"[20]"}],"fun_headline_variants":["3D CNN outdrives RNNs in mini racing: 35.1s vs 36.2s","Modified 3D CNN hits 34.1s lap on Mini Monaco","3D CNN beats LSTM and GRU in simulated car race","RGB-D and 3D CNNs steer mini cars faster than RNNs","Mini car race: 3D CNN bests recurrent nets in lap times"],"cache_read_input_tokens":2688,"weakest_assumption_plain":"The model labeled fastest was chosen because of its performance on the same Mini Monaco laps where it is then reported as the winner, so its lap-time advantage over the RNNs is not measured under independent model selection.","fun_headline_variants_meta":{"raw":{"variants":["3D CNN outdrives RNNs in mini racing: 35.1s vs 36.2s","Modified 3D CNN hits 34.1s lap on Mini Monaco","3D CNN beats LSTM and GRU in simulated car race","RGB-D and 3D CNNs steer mini cars faster than RNNs","Mini car race: 3D CNN bests recurrent nets in lap times"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000339,"raw_usage":{"total_tokens":1725,"prompt_tokens":781,"completion_tokens":944,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":525,"completion_tokens_details":{"reasoning_tokens":835}},"tokens_in":525,"tokens_out":944,"duration_ms":7132,"temperature":1.0,"reasoning_tokens":835,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-05T14:25:04.844295+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the default RNN, modified RNN, and the selected 3D CNN on 30 fresh Mini Monaco laps that were never used when choosing the 3D CNN's layer count, then compare mean lap times with their standard deviations; if the 3D CNN mean is not lower than both RNN means, the reported superiority is an artifact of selection.","supporting_citations":[{"cited_title":"gym-donkeycar: A simulator for donkey car with openai gym interface,","cited_arxiv_id":null,"evidence_quote":"Supplies the simulator environment used to generate the RGB-D training data and run the 30-lap evaluations."},{"cited_title":"Sim-To-Real Transfer for Miniature Autonomous Car Racing","cited_arxiv_id":"2011.05617","evidence_quote":"Precedent applying 3D CNNs to miniature autonomous car racing in simulation, which the proposed method builds on."},{"cited_title":"How hard is it to cross the room? -- Training (Recurrent) Neural Networks to steer a UAV","cited_arxiv_id":"1702.07600","evidence_quote":"Provides the recurrent-image-steering design that the RNN baseline is adapted from."},{"cited_title":"Long short-term memory,","cited_arxiv_id":null,"evidence_quote":"Defines the LSTM gating mechanism used in the default RNN baseline."},{"cited_title":"Donkeycar: An open source diy self-driving platform,","cited_arxiv_id":null,"evidence_quote":"Provides the open-source driving platform whose model scaffolds and simulator configuration the experiments use."}],"review_version":1}