{"id":"272f623d-5563-4ffc-ab57-2fd064343e6c","arxiv_id":"2507.07745","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"A custom GPT-4-turbo configuration classifies and segments fruit-picking motion time series into primitive actions, but per-segment accuracy stays between 14% and 44% depending on prompting and feedback.","lead":"This paper tests whether a custom GPT-4-turbo configuration can classify and segment robot-captured fruit-picking motions into primitive actions such as twist, tilt, and pull. It compares explanation-only, few-shot example, and combined prompting, finding all variants far from reliable but with a small feedback step improving segmentation on a 5-sequence subset.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Evaluation may be measuring mismatch between hand-coded kinematic rules and the data, not an LLM capability; ground-truth boundaries also lack independent validation.","rationale":"The reader's weakest assumption already identifies the dependency of the evaluation on the correctness of Table I and the button-press ground truth. My stress-test concurs: this is the most load-bearing link in the argument. A concrete validation of the kinematic signatures and boundary timestamps would either substantiate the reported accuracy numbers or reveal that they reflect rule-schema mismatch. The paper's other weaknesses—lack of baselines, no released code or prompts, small feedback subset—are real but secondary; they affect interpretability and generalizability, whereas the ground-truth and schema validity affect whether the central measurement means anything at all. Since the reader's CONDITIONAL verdict already flags this as an open condition, my assessment does not change the verdict.","tokens_in":6129,"tokens_out":6769,"duration_ms":73347,"concrete_test":"Re-analyze the raw velocity data: for each of the 56 ground-truth primitive segments, compute the mean absolute value of each linear and angular velocity component and verify that the dominant axes match Table I (e.g., all 'tilt' segments should have minimal translation and dominant angular velocity about y; all 'pull' segments should have dominant translational velocity along x with little rotation), and implement a simple threshold-based segmenter that starts a new segment at the first sample where the dominant velocity axis changes by more than a noise floor, then compare its boundaries to the button-press timestamps to check whether the ground-truth boundaries align with the kinematic signatures.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim that an LLM can classify and segment fruit-picking motions is only as strong as the evaluation it rests on. That evaluation assumes two things without independent support: (i) the five primitives are correctly and completely characterized by the dominant-axis kinematic signatures in Table I, and (ii) the button-press timestamps accurately mark the true segment boundaries. If the captured motions do not actually exhibit those clean dominant-axis patterns, or if the button presses are systematically early or late relative to the onset of the primitive, then the reported accuracies (19%, 14%, 28%, 44%) quantify the fit between a hand-coded rule schema and the data, not the LLM's learned capability. The paper provides no validation that each ground-truth segment's velocity profile matches its Table I signature, and no independent check of the button-press boundaries (e.g., by an algorithmic detector or a second annotator). Without such validation, the segmentation errors in Fig. 3 and the classification counts in Table II are uninterpretable as evidence about LLM abilities.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The manuscript investigates whether a GPT-4-turbo-based Custom GPT can simultaneously classify and segment kinesthetically recorded fruit-picking motion time series into five predefined primitive actions (Pull, Slide, Swing, Tilt, Twist). Three in-context learning approaches are compared—rules-only (Approach A), examples-only (Approach B), and both (Approach C)—together with a feedback variant applied to five selected sequences. Using 20 complex test sequences containing 56 primitive segments, the paper reports per-segment classification accuracies of 19%, 14%, 28%, and 44% (feedback), with absolute segmentation errors summarized only as boxplots in Fig. 3.","tokens_in":6282,"tokens_out":5783,"duration_ms":66565,"significance":"The paper addresses a relevant and timely problem in Learning from Demonstration: whether an LLM can perform motion segmentation without supervised model training or analytic expertise. The authors are to be credited for using genuinely kinesthetically captured robot data, for the explicit comparison of three prompting modes, and for attempting a feedback-based improvement loop; the accompanying Python helper script is a useful reproducibility artifact. If the evaluation were fully convincing, the results would constitute a preliminary, honest data point on the feasibility and current limitations of LLM-based action segmentation. However, the absence of a rule-based baseline, the lack of independent validation of the ground-truth boundaries, and the small uncontrolled feedback experiment severely constrain the strength of the capability claims, so the significance as presented is modest.","major_comments":[{"comment":"The evaluation conflates the LLM's reasoning capability with the quality of the author-defined kinematic rule schema. The model is explicitly given the five primitive definitions in Table I and the 'earliest significant change' heuristic, and it is then scored against the same taxonomy of primitives. Because no deterministic baseline that directly implements these same rules (e.g., a threshold-based detector of dominant-axis velocity changes) is reported, the low accuracies of 19%, 14%, and 28% cannot be attributed to deficiencies in LLM reasoning; they may simply reflect a mismatch between the hand-coded signatures and the actual captured motions. The authors should add a baseline that applies Table I programmatically and compare its classification and segmentation errors with those of the three LLM approaches.","section":"Section III (Table I) and Section II"},{"comment":"Ground-truth segment boundaries are taken from button-press timestamps, but no independent validation of those timestamps is reported. A button press may lead or lag the true kinematic onset of a primitive by an unknown and probably varying amount, which would directly bias both the absolute segmentation error in Fig. 3 and the segment labels used in Table II. The authors should validate a subset of boundaries against an independent annotation or an automatic change-point detector, and they should report the accuracy of the button-press timing. In addition, the 'Abs Segm. error' metric is never defined; please provide an exact formula and include the numeric values underlying Fig. 3 instead of boxplots alone.","section":"Section III.A and Fig. 3"},{"comment":"The feedback experiment is not a controlled or statistically interpretable comparison. Only 5 of the 20 sequences (marked with ⋆) are used for feedback, but the selection protocol is not stated, the number of feedback iterations is not given, and the reported 44% accuracy is computed over 43 primitives rather than the 56 used for the other approaches. Consequently, the 28%-to-44% improvement cannot be attributed to feedback as a general mechanism. The authors should specify the feedback procedure, apply it to a pre-registered held-out set or to the full 20-sequence set, and report confidence intervals or significance tests appropriate for the small sample size.","section":"Section IV and Table II"}],"minor_comments":[{"comment":"The term 'fine-tuning' is used throughout (abstract, Sections I, III, IV) to describe the three approaches, but the paper actually uses a Custom GPT with in-context prompting, not gradient-based fine-tuning of model weights. Please replace 'fine-tuning' with 'in-context learning' or 'prompting'.","section":"Title and Section III"},{"comment":"The preprocessing relies on Nadaraya–Watson interpolation, but the Gaussian kernel variance σ² in Eq. (1) is never reported, and the number of kernels per second (20) is stated without justification. Please provide the parameter values and a brief rationale, as they control the smoothness of the velocity signals shown to the model.","section":"Section III.B"},{"comment":"The phrase 'earliest significant change' is used as a segmentation rule but is never operationally defined (e.g., threshold on derivative magnitude, relative change, or rise above a noise floor). Without a precise definition, the rule set is under-specified and the model's behavior is hard to interpret or reproduce.","section":"Section III.C"},{"comment":"The manuscript does not clarify how the time series are presented to the LLM: as numeric text, as plots, or as a combination. The text in Section III mentions 'plots' and normalization of 'all plots,' but the prompt format is never described. This is essential for reproducibility and should be stated explicitly.","section":"Sections III and IV"},{"comment":"The legend in Table II uses ● symbols to indicate which approach matched, but a per-class confusion matrix or a clearer breakdown would be more informative, especially because Approach B's overall accuracy (14%) is below the 20% chance level. Fig. 3 should also show numeric axis values and sample sizes.","section":"Table II and Fig. 3"},{"comment":"There are several typographical errors, e.g., 'primarly' in the Introduction, 'is relied' in Section III.C, and inconsistent punctuation in the Abstract. A careful proofreading pass is needed.","section":"Introduction and text"}],"recommendation":"major_revision","confidential_remarks":"This is a late-breaking poster submission, so the depth of experimentation is understandably limited; however, the key weaknesses are load-bearing for the central capability claim. The most important fix is a rule-based baseline control that directly implements Table I, since without it the low accuracies are uninterpretable as evidence about LLM capabilities. The feedback result is also presented too strongly relative to the uncontrolled 5-sequence subset. These issues are addressable within a revision, so major_revision seems appropriate rather than rejection."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: this is a thin but genuine pilot. The comparison of explanation-only, examples-only, and combined prompting for LLM-based segmentation of fruit-picking motions is new relative to the cited work, and the paper is honest about its low hit rates. The soft spots are just as the reader says: no baselines, no error bars, no prompt/data release, and the feedback protocol is under-specified.\n\nThe stress-test concern about the hand-coded rules is real but slightly overstated: the ground-truth boundaries come from button presses, which are independent of the rule schema, so the errors do tell you something about the rules-plus-LLM pipeline. What is missing is a control that applies Table I rules via a simple threshold detector; without that, you can't separate the LLM's contribution from the rule schema's quality. I also want to flag that \"fine-tuning\" is the wrong word: the paper never updates weights, it just changes the prompt context. That's not fatal but it should be fixed.\n\nThe feedback experiment is the most confusing part: the text says five sequences were used for corrective feedback, but the reported 19/43 covers 43 primitives, which matches the non-starred sequences. If the evaluation is on the held-out 15 sequences, that's a reasonable generalization test; the paper should say so explicitly. As is, the numbers are low (28% best without feedback) and the claim \"LLMs can segment\" is only weakly supported.\n\nEven so, the direction is worth exploring, and the paper is clearly written and honest. It deserves a shot at peer review, not as a definitive result but as a pilot that needs baselines, statistics, and a proper protocol. I'd send it to a workshop or a late-breaking track.","headline":"A thin but honest pilot that compares LLM prompting styles for motion segmentation; the low numbers and missing baselines make it a starting point, not a proof.","tokens_in":6862,"tokens_out":3837,"would_cite":false,"duration_ms":43064,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper tests whether a commercially available LLM, configured with kinematic rules and a few example traces, can simultaneously classify and segment fruit-picking motion time series into five primitives, and reports that accuracy is…","keywords":["large language models","motion primitive segmentation","learning by demonstration","fruit picking","time series classification","kinesthetic teaching","GPT-4","action segmentation"],"falsifier":"Take a fresh set of fruit-picking demonstrations with a deliberately compliant stem so that a 'pull' includes substantial rotation, and run the rules-plus-examples configuration: if the model still marks the pull boundary at the point of the first significant x-translation rather than at the actual detachment moment, it shows the rigid rule schema—not the model's reasoning—is fixing the boundary.","tokens_in":5905,"feed_emoji":"🍑","tokens_out":6656,"duration_ms":72880,"temperature":0.7,"pith_summary":"The paper investigates whether a large language model can take on a classic robot-learning task: splitting a complex demonstrated motion into a sequence of primitive actions. Using GPT-4-turbo configured through a custom chat interface, the authors test three ways of teaching the model the five fruit-picking primitives (Pull, Slide, Swing, Tilt, Twist): written kinematic rules, a few example motion traces, and a combination of both. On 20 complex sequences containing 56 primitive segments, the per-segment classification accuracy was 19% with rules only, 14% with examples only, and 28% with both; providing corrective feedback on five of the sequences raised accuracy to 44% on those segments. The authors present this as evidence that the approach is feasible and worth refining, not as a ready-to-deploy system.","feed_headline":"GPT-4 segments fruit-picking motions at only 28 percent accuracy","feed_subtitle":"Rules plus examples beat either alone; feedback lifts accuracy to 44 percent on five sequences.","key_machinery":"The carrying mechanism is a 'custom GPT' whose instructions embed Table I's hand-defined kinematic signatures—each primitive is assigned a dominant translational or angular velocity axis (Pull: translation along x without significant rotation; Twist: rotation about x without translation, and so on)—together with a rule that each primitive begins at the earliest significant change in the relevant velocity component. The model is also given downsampled traces produced by Nadaraya-Watson kernel regression (20 kernels per second) and a plotting helper that normalizes velocity profiles; it reads these and must return a chronological primitive list with start/end indices.","core_discovery":"The central discovery is an empirical result: a GPT-4-turbo configured through a Custom GPT chat interface can take a kinesthetically recorded velocity time series of a complex fruit-picking motion and output a structured chronological list of primitive actions with start and end indices (for instance, 'twist (Index 0–62), tilt (Index 63–112), pull (Index 113–170)'). On 20 complex sequences containing 56 primitive segments, the per-segment classification accuracy was 19% when the model was given only written kinematic rules, 14% when given only five example traces per primitive, and 28% when given both. After corrective feedback on five validation sequences, accuracy on the 43 segments in those sequences rose to 44%. The paper frames this as evidence that LLM-based simultaneous classification and segmentation is feasible and worth pursuing, not as a deployment-ready system.","pith_inferences":["The hand-written rule table is a potential confound: if real picking motions often deviate from the dominant-axis patterns, the low accuracy partly reflects the schema rather than the LLM, and a redesigned schema with tolerance for mixed axes is a natural next test.","Ground-truth boundaries came from button presses with no reported inter-rater check; an independent coding study would show how much of the apparent model error is actually annotation noise.","A cheap experiment would feed the model numeric velocity arrays instead of plotted time series; a large accuracy change would point to the visual-plot interface as a bottleneck."],"forward_implications":["A demonstration-based teaching interface could be configured for a new task by writing kinematic rules in natural language and showing a handful of examples, bypassing per-task neural network training.","Rules and examples carry complementary information: combining them lifted classification from 19%/14% to 28%, more than either alone.","Providing corrective feedback on a few sequences raised accuracy from 28% to 44% on those sequences, indicating that an interactive refinement loop is a promising direction."],"supporting_citations":[{"why":"Supplies the Transformer architecture that the tested large language model is built upon.","marker":"[5]"},{"why":"Describes the generative pre-training method underlying the GPT model used in the experiments.","marker":"[7]"},{"why":"Introduces the Nadaraya-Watson kernel regression used to downsample the 500 Hz motion data to 20 Hz.","marker":"[14]"},{"why":"Provides the companion kernel regression estimator also cited for the downsampling step.","marker":"[15]"},{"why":"Is the reference for the Gaussian kernel weighting used in the interpolation.","marker":"[16]"},{"why":"Supplies the unit-quaternion algebra used to compute angular velocity from orientation data.","marker":"[18]"}],"fun_headline_variants":["LLMs segment fruit-picking motions at 28% accuracy","GPT-4-turbo segments fruit-picking primitives at 28%","Rules plus examples lift LLM fruit-picking accuracy to 44%","LLM fruit-picking segmentation: 28% accuracy, 44% with feedback","LLMs classify fruit-picking motions at 28% accuracy"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that each fruit-picking primitive is correctly and completely characterized by its dominant translational or angular velocity axis and that each primitive starts at the earliest significant change in that signal; if real motions do not follow those clean patterns, the reported errors measure the mismatch between the hand-written rules and the data, not the model's ability.","fun_headline_variants_meta":{"raw":{"variants":["LLMs segment fruit-picking motions at 28% accuracy","GPT-4-turbo segments fruit-picking primitives at 28%","Rules plus examples lift LLM fruit-picking accuracy to 44%","LLM fruit-picking segmentation: 28% accuracy, 44% with feedback","LLMs classify fruit-picking motions at 28% accuracy"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000918,"raw_usage":{"total_tokens":3916,"prompt_tokens":900,"completion_tokens":3016,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":516,"completion_tokens_details":{"reasoning_tokens":2933}},"tokens_in":516,"tokens_out":3016,"duration_ms":21663,"temperature":1.0,"reasoning_tokens":2933,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T18:32:47.483160+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a fresh set of fruit-picking demonstrations with a deliberately compliant stem so that a 'pull' includes substantial rotation, and run the rules-plus-examples configuration: if the model still marks the pull boundary at the point of the first significant x-translation rather than at the actual detachment moment, it shows the rigid rule schema—not the model's reasoning—is fixing the boundary.","supporting_citations":[{"cited_title":"Smooth regression analysis,","cited_arxiv_id":null,"evidence_quote":"Provides the companion kernel regression estimator also cited for the downsampling step."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Is the reference for the Gaussian kernel weighting used in the interpolation."},{"cited_title":"Exponential stability of an attitude trajectory tracking controller utilizing unit quaternions,","cited_arxiv_id":null,"evidence_quote":"Supplies the unit-quaternion algebra used to compute angular velocity from orientation data."}],"review_version":1}