On the capabilities of LLMs for classifying and segmenting time series of fruit picking motions into primitive actions

Dimitrios Papageorgiou; Eleni Konstantinidou; Nikolaos Efstathopoulos; Nikolaos Kounalakis

REVIEW 3 major objections 6 minor 19 references

On the capabilities of LLMs for classifying and segmenting time series of fruit picking motions into primitive actions

T0 review · 3 major / 6 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read This paper tests whether a commercially available LLM, configured with kinematic rules and a few example traces, can simultaneously classify and segment fruit-picking motion time series into five primitives, and reports that accuracy is…

desk verdict A thin but honest pilot that compares LLM prompting styles for motion segmentation; the low numbers and missing baselines make it a starting point, not a proof. read the letter →

arxiv 2507.07745 v1 pith:MF3Y6KU5 submitted 2025-07-10 cs.RO

Eleni Konstantinidou , Nikolaos Kounalakis , Nikolaos Efstathopoulos , Dimitrios Papageorgiou This is my paper · ORCID

classification cs.RO

keywords largelanguagemodelsmotionprimitivesegmentationlearningbydemonstrationfruitpickingtimeseriesclassificationkinestheticteachingGPT-4action

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper investigates whether a large language model can take on a classic robot-learning task: splitting a complex demonstrated motion into a sequence of primitive actions. Using GPT-4-turbo configured through a custom chat interface, the authors test three ways of teaching the model the five fruit-picking primitives (Pull, Slide, Swing, Tilt, Twist): written kinematic rules, a few example motion traces, and a combination of both. On 20 complex sequences containing 56 primitive segments, the per-segment classification accuracy was 19% with rules only, 14% with examples only, and 28% with both; providing corrective feedback on five of the sequences raised accuracy to 44% on those segments. The authors present this as evidence that the approach is feasible and worth refining, not as a ready-to-deploy system.

What carries the argument

The carrying mechanism is a 'custom GPT' whose instructions embed Table I's hand-defined kinematic signatures—each primitive is assigned a dominant translational or angular velocity axis (Pull: translation along x without significant rotation; Twist: rotation about x without translation, and so on)—together with a rule that each primitive begins at the earliest significant change in the relevant velocity component. The model is also given downsampled traces produced by Nadaraya-Watson kernel regression (20 kernels per second) and a plotting helper that normalizes velocity profiles; it reads these and must return a chronological primitive list with start/end indices.

What would settle it

Take a fresh set of fruit-picking demonstrations with a deliberately compliant stem so that a 'pull' includes substantial rotation, and run the rules-plus-examples configuration: if the model still marks the pull boundary at the point of the first significant x-translation rather than at the actual detachment moment, it shows the rigid rule schema—not the model's reasoning—is fixing the boundary.

Watch

Extended reading notes

Core claim

The central discovery is an empirical result: a GPT-4-turbo configured through a Custom GPT chat interface can take a kinesthetically recorded velocity time series of a complex fruit-picking motion and output a structured chronological list of primitive actions with start and end indices (for instance, 'twist (Index 0–62), tilt (Index 63–112), pull (Index 113–170)'). On 20 complex sequences containing 56 primitive segments, the per-segment classification accuracy was 19% when the model was given only written kinematic rules, 14% when given only five example traces per primitive, and 28% when given both. After corrective feedback on five validation sequences, accuracy on the 43 segments in those sequences rose to 44%. The paper frames this as evidence that LLM-based simultaneous classification and segmentation is feasible and worth pursuing, not as a deployment-ready system.

Load-bearing premise

The load-bearing premise is that each fruit-picking primitive is correctly and completely characterized by its dominant translational or angular velocity axis and that each primitive starts at the earliest significant change in that signal; if real motions do not follow those clean patterns, the reported errors measure the mismatch between the hand-written rules and the data, not the model's ability.

Editorial extensions

If this is right

A demonstration-based teaching interface could be configured for a new task by writing kinematic rules in natural language and showing a handful of examples, bypassing per-task neural network training.
Rules and examples carry complementary information: combining them lifted classification from 19%/14% to 28%, more than either alone.
Providing corrective feedback on a few sequences raised accuracy from 28% to 44% on those sequences, indicating that an interactive refinement loop is a promising direction.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

The hand-written rule table is a potential confound: if real picking motions often deviate from the dominant-axis patterns, the low accuracy partly reflects the schema rather than the LLM, and a redesigned schema with tolerance for mixed axes is a natural next test.
Ground-truth boundaries came from button presses with no reported inter-rater check; an independent coding study would show how much of the apparent model error is actually annotation noise.
A cheap experiment would feed the model numeric velocity arrays instead of plotted time series; a large accuracy change would point to the visual-plot interface as a bottleneck.

Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Short version: this is a thin but genuine pilot. The comparison of explanation-only, examples-only, and combined prompting for LLM-based segmentation of fruit-picking motions is new relative to the cited work, and the paper is honest about its low hit rates. The soft spots are just as the reader says: no baselines, no error bars, no prompt/data release, and the feedback protocol is under-specified.

The stress-test concern about the hand-coded rules is real but slightly overstated: the ground-truth boundaries come from button presses, which are independent of the rule schema, so the errors do tell you something about the rules-plus-LLM pipeline. What is missing is a control that applies Table I rules via a simple threshold detector; without that, you can't separate the LLM's contribution from the rule schema's quality. I also want to flag that "fine-tuning" is the wrong word: the paper never updates weights, it just changes the prompt context. That's not fatal but it should be fixed.

The feedback experiment is the most confusing part: the text says five sequences were used for corrective feedback, but the reported 19/43 covers 43 primitives, which matches the non-starred sequences. If the evaluation is on the held-out 15 sequences, that's a reasonable generalization test; the paper should say so explicitly. As is, the numbers are low (28% best without feedback) and the claim "LLMs can segment" is only weakly supported.

Even so, the direction is worth exploring, and the paper is clearly written and honest. It deserves a shot at peer review, not as a definitive result but as a pilot that needs baselines, statistics, and a proper protocol. I'd send it to a workshop or a late-breaking track.

Referee Report

3 major / 6 minor

Summary. The manuscript investigates whether a GPT-4-turbo-based Custom GPT can simultaneously classify and segment kinesthetically recorded fruit-picking motion time series into five predefined primitive actions (Pull, Slide, Swing, Tilt, Twist). Three in-context learning approaches are compared—rules-only (Approach A), examples-only (Approach B), and both (Approach C)—together with a feedback variant applied to five selected sequences. Using 20 complex test sequences containing 56 primitive segments, the paper reports per-segment classification accuracies of 19%, 14%, 28%, and 44% (feedback), with absolute segmentation errors summarized only as boxplots in Fig. 3.

Significance. The paper addresses a relevant and timely problem in Learning from Demonstration: whether an LLM can perform motion segmentation without supervised model training or analytic expertise. The authors are to be credited for using genuinely kinesthetically captured robot data, for the explicit comparison of three prompting modes, and for attempting a feedback-based improvement loop; the accompanying Python helper script is a useful reproducibility artifact. If the evaluation were fully convincing, the results would constitute a preliminary, honest data point on the feasibility and current limitations of LLM-based action segmentation. However, the absence of a rule-based baseline, the lack of independent validation of the ground-truth boundaries, and the small uncontrolled feedback experiment severely constrain the strength of the capability claims, so the significance as presented is modest.

major comments (3)

[Section III (Table I) and Section II] The evaluation conflates the LLM's reasoning capability with the quality of the author-defined kinematic rule schema. The model is explicitly given the five primitive definitions in Table I and the 'earliest significant change' heuristic, and it is then scored against the same taxonomy of primitives. Because no deterministic baseline that directly implements these same rules (e.g., a threshold-based detector of dominant-axis velocity changes) is reported, the low accuracies of 19%, 14%, and 28% cannot be attributed to deficiencies in LLM reasoning; they may simply reflect a mismatch between the hand-coded signatures and the actual captured motions. The authors should add a baseline that applies Table I programmatically and compare its classification and segmentation errors with those of the three LLM approaches.
[Section III.A and Fig. 3] Ground-truth segment boundaries are taken from button-press timestamps, but no independent validation of those timestamps is reported. A button press may lead or lag the true kinematic onset of a primitive by an unknown and probably varying amount, which would directly bias both the absolute segmentation error in Fig. 3 and the segment labels used in Table II. The authors should validate a subset of boundaries against an independent annotation or an automatic change-point detector, and they should report the accuracy of the button-press timing. In addition, the 'Abs Segm. error' metric is never defined; please provide an exact formula and include the numeric values underlying Fig. 3 instead of boxplots alone.
[Section IV and Table II] The feedback experiment is not a controlled or statistically interpretable comparison. Only 5 of the 20 sequences (marked with ⋆) are used for feedback, but the selection protocol is not stated, the number of feedback iterations is not given, and the reported 44% accuracy is computed over 43 primitives rather than the 56 used for the other approaches. Consequently, the 28%-to-44% improvement cannot be attributed to feedback as a general mechanism. The authors should specify the feedback procedure, apply it to a pre-registered held-out set or to the full 20-sequence set, and report confidence intervals or significance tests appropriate for the small sample size.

minor comments (6)

[Title and Section III] The term 'fine-tuning' is used throughout (abstract, Sections I, III, IV) to describe the three approaches, but the paper actually uses a Custom GPT with in-context prompting, not gradient-based fine-tuning of model weights. Please replace 'fine-tuning' with 'in-context learning' or 'prompting'.
[Section III.B] The preprocessing relies on Nadaraya–Watson interpolation, but the Gaussian kernel variance σ² in Eq. (1) is never reported, and the number of kernels per second (20) is stated without justification. Please provide the parameter values and a brief rationale, as they control the smoothness of the velocity signals shown to the model.
[Section III.C] The phrase 'earliest significant change' is used as a segmentation rule but is never operationally defined (e.g., threshold on derivative magnitude, relative change, or rise above a noise floor). Without a precise definition, the rule set is under-specified and the model's behavior is hard to interpret or reproduce.
[Sections III and IV] The manuscript does not clarify how the time series are presented to the LLM: as numeric text, as plots, or as a combination. The text in Section III mentions 'plots' and normalization of 'all plots,' but the prompt format is never described. This is essential for reproducibility and should be stated explicitly.
[Table II and Fig. 3] The legend in Table II uses ● symbols to indicate which approach matched, but a per-class confusion matrix or a clearer breakdown would be more informative, especially because Approach B's overall accuracy (14%) is below the 20% chance level. Fig. 3 should also show numeric axis values and sample sizes.
[Introduction and text] There are several typographical errors, e.g., 'primarly' in the Introduction, 'is relied' in Section III.C, and inconsistent punctuation in the Abstract. A careful proofreading pass is needed.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the evaluation is an empirical benchmark against independent button-press ground truth, with no fitted parameters or load-bearing self-citations.

full rationale

The paper's central claim is empirical: whether GPT-4-turbo, configured as a Custom GPT, can classify and segment fruit-picking motion time series into five predefined primitives. The model is given either linguistic rules from Table I, example motions, both, or corrective feedback, and its outputs are compared against ground-truth segment labels and button-press timestamps. The ground truth is collected independently of the model via a push-button attached to the end-effector, and no parameter is fitted from the testing sequences. Approach A uses only the predefined kinematic descriptions, Approach B uses only example motions, and Approach C combines both; none of these reduce to the evaluation metric by construction. The classification counts in Table II and segmentation errors in Fig. 3 are externally measurable outcomes, not identities derived from the inputs. There are no self-citations, no invoked uniqueness theorems, and no ansatz smuggled in through prior author work. The feedback approach is evaluated on the same five sequences used for feedback, which is a validity limitation because those numbers are not an independent generalization result, but the paper discloses this and does not present the feedback numbers as an unseen-data prediction. The concern that the hand-coded Table I signatures may not match real motions, or that button-press boundaries lack independent validation, is a correctness or measurement-risk issue, not circularity. No step in the derivation chain is equivalent to its own input by construction. Therefore the circularity score is 0.

Assumptions & free parameters 5 free parameters · 4 assumptions · 0 invented entities

The central result depends on a hand-made action taxonomy, unvalidated ground-truth markers, manual prompt rules, and arbitrary data splits. No external benchmark or released artifact anchors the numbers, so the empirical claims rest almost entirely on the authors' own setup.

free parameters (5)

Gaussian kernel variance sigma^2 = Not reported
Used in Nadaraya-Watson interpolation of the 500 Hz pose data (Eq. 1); the value affects the smoothed input time series and is not given.
Interpolation kernel count = 20 kernels per second
Chosen manually to reduce data size for GPT input; no justification or sensitivity analysis is provided.
Segment start threshold = "Earliest significant change" (unspecified)
The prompt rule for finding primitive boundaries is qualitative; no quantitative threshold is defined.
Few-shot example count = 5 examples per primitive
The training set size for Approaches B and C is an arbitrary choice; no ablation is reported.
Feedback validation set size = 5 of 20 sequences
The subset used for corrective feedback is selected without a stated random split or justification.

assumptions (4)

domain assumption The five primitive actions (Pull, Slide, Swing, Tilt, Twist) and their kinematic signatures in Table I are a valid and complete decomposition of fruit-picking detachment motions.
The entire evaluation labels data according to this hand-defined schema, introduced in Section II and Table I.
domain assumption Button-press timestamps provide reliable ground truth for primitive boundaries.
Used as ground truth in evaluation without independent validation; Section III-A.
standard math The standard pose representation, quaternion kinematics, and centered numerical differentiation formulas in Section III-B are correct.
Equations (1)-(3) are standard background from cited references.
domain assumption GPT-4-turbo can meaningfully process the provided velocity data and return parseable indices in the requested format.
No raw model outputs, parsing checks, or failure analysis are shown, so low accuracy could originate from formatting or parsing issues rather than semantic understanding.

how reviews work

0 comments

Cite this review

Pith. "Pith review of On the capabilities of LLMs for classifying and segmenting time series of fruit picking motions into primitive actions." pith.science (2026). https://pith.science/paper/MF3Y6KU5

@misc{pith2026250707745,
  author       = {Pith},
  title        = {Pith review of: On the capabilities of LLMs for classifying and segmenting time series of fruit picking motions into primitive actions},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/MF3Y6KU5}},
  note         = {Machine review of arXiv:2507.07745}
}

read the original abstract

Despite their recent introduction to human society, Large Language Models (LLMs) have significantly affected the way we tackle mental challenges in our everyday lives. From optimizing our linguistic communication to assisting us in making important decisions, LLMs, such as ChatGPT, are notably reducing our cognitive load by gradually taking on an increasing share of our mental activities. In the context of Learning by Demonstration (LbD), classifying and segmenting complex motions into primitive actions, such as pushing, pulling, twisting etc, is considered to be a key-step towards encoding a task. In this work, we investigate the capabilities of LLMs to undertake this task, considering a finite set of predefined primitive actions found in fruit picking operations. By utilizing LLMs instead of simple supervised learning or analytic methods, we aim at making the method easily applicable and deployable in a real-life scenario. Three different fine-tuning approaches are investigated, compared on datasets captured kinesthetically, using a UR10e robot, during a fruit-picking scenario.

Figures

Figures reproduced from arXiv: 2507.07745 by the authors.

**Figure 1.** The considered primitive actions To teach a fruit picking skill to a novice farmer, humans firstly utilize verbal communication, i.e. linguistic explanation. An example sentence provided by an experienced farmer to the novice one, to this aim, could be the following: ”To detach the fruit, you firstly have to twist it and then pull it from the branch.”. In most of the cases, the experienced farmer also provides an … view at source ↗

**Figure 1.** Notice that the velocity of the end-effector is considered, instead of its pose, in order to facilitate the segmentation by rejecting bias. In other words, by selecting the velocity instead of the pose, each action maintains its characteristics even if it follows after a displacement that could possibly have occurred due to the sequential nature of the complex motion considered. III. CONSIDERED APPROACHES AND COMPAR… view at source ↗

**Figure 2.** Image from the demonstration of the pull primitive. All experimental data employed in this work were organized into three datasets: training (provided to the model in Approaches B and C as examples), validation (used for assessing the performance of Approach C with feedback) and testing, each of which consisted of time-series motion recordings in the same format described above. The training dataset was made up of … view at source ↗

Figures from the paper (1 more)

**Figure 3.** Figure 3: Statistical comparison of segmentation error across [PITH_FULL_IMAGE:figures/full_fig_p004_3.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

19 extracted references · 14 canonical work pages

[1]

Artificial intelligence: A powerful paradigm for scientific research,

Y . Xu, X. Liu, X. Cao, and et al., “Artificial intelligence: A powerful paradigm for scientific research,” Innovation (Cambridge) , vol. 2, no. 4, p. 100179, 2021, © 2021 The Author(s). Published by Elsevier Inc

work page 2021
[2]

A compact guide to learn large language models,

A. Suruj, “A compact guide to learn large language models,” 11 2024

work page 2024
[3]

Wulff, M

P. Wulff, M. Kubsch, and C. Krist, Natural Language Processing and Large Language Models. Cham: Springer Nature Switzerland, 2025, pp. 117–142

work page 2025
[4]

Bisong, Building Machine Learning and Deep Learning Models on Google Cloud Platform: A Comprehensive Guide for Beginners , 01 2019

E. Bisong, Building Machine Learning and Deep Learning Models on Google Cloud Platform: A Comprehensive Guide for Beginners , 01 2019

work page 2019
[5]

Attention is all you need,

A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. Kaiser, and I. Polosukhin, “Attention is all you need,”
[6]

Bert: Pre-training of deep bidirectional transformers for language understanding,

J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova, “Bert: Pre-training of deep bidirectional transformers for language understanding,” 10 2018

work page 2018
[7]

Improving language understanding by generative pre-training,

A. Radford, K. Narasimhan, T. Salimans, and I. Sutskever, “Improving language understanding by generative pre-training,” 2018

2018
[8]

Recent progress in semantic image segmentation,

X. Liu, Z. Deng, and Y . Yang, “Recent progress in semantic image segmentation,” Artificial Intelligence Review, vol. 52, no. 2, pp. 1089– 1106, 2019

work page 2019

Show all 19 references

[9]

A dnn-based semantic segmentation for detecting weed and crop,

J. You, W. Liu, and J. Lee, “A dnn-based semantic segmentation for detecting weed and crop,” Computers and Electronics in Agriculture , vol. 178, p. 105750, 2020

2020
[10]

Unifying motion segmentation, estimation, and tracking for complex dynamic scenes,

K. Judd, “Unifying motion segmentation, estimation, and tracking for complex dynamic scenes,” Ph.D. dissertation, University of Oxford, 2019

2019
[11]

An organizing principle for a class of voluntary move- ments,

N. Hogan, “An organizing principle for a class of voluntary move- ments,” Journal of Neuroscience, vol. 4, no. 11, pp. 2745–2754, 1984

1984
[12]

Temporal convolutional networks for action segmentation and detection,

C. Lea, M. D. Flynn, R. Vidal, A. Reiter, and G. D. Hager, “Temporal convolutional networks for action segmentation and detection,” 2016

2016
[13]

Temporal convolutional networks: A unified approach to action segmentation,

C. Lea, R. Vidal, A. Reiter, and G. D. Hager, “Temporal convolutional networks: A unified approach to action segmentation,” in Computer Vision – ECCV 2016 Workshops , G. Hua and H. J ´egou, Eds. Cham: Springer International Publishing, 2016, pp. 47–54

2016
[14]

On estimating regression,

E. A. Nadaraya, “On estimating regression,” Theory of Probability & Its Applications, vol. 9, no. 1, pp. 141–142, 1964

1964
[15]

Smooth regression analysis,

G. S. Watson, “Smooth regression analysis,” Sankhy¯a: The Indian Journal of Statistics, Series A (1961–2002) , vol. 26, no. 4, pp. 359– 372, 1964

1961
[16]

C. E. Rasmussen and C. K. I. Williams, Gaussian Processes for Machine Learning. Cambridge, Massachusetts: The MIT Press, 2006

2006
[17]

R. L. Burden and J. D. Faires, Numerical Analysis, 9th ed. Boston: Brooks/Cole, Cengage Learning, 2010

2010
[18]

Exponential stability of an attitude trajectory tracking controller utilizing unit quaternions,

L. Koutras and Z. Doulgeri, “Exponential stability of an attitude trajectory tracking controller utilizing unit quaternions,” 06 2021, pp. 1126–1131

2021
[2023]

Available: https://arxiv.org/abs/1706.03762

[Online]. Available: https://arxiv.org/abs/1706.03762

arXiv

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.

[1] [1]

Artificial intelligence: A powerful paradigm for scientific research,

Y . Xu, X. Liu, X. Cao, and et al., “Artificial intelligence: A powerful paradigm for scientific research,” Innovation (Cambridge) , vol. 2, no. 4, p. 100179, 2021, © 2021 The Author(s). Published by Elsevier Inc

work page 2021

[2] [2]

A compact guide to learn large language models,

A. Suruj, “A compact guide to learn large language models,” 11 2024

work page 2024

[3] [3]

Wulff, M

P. Wulff, M. Kubsch, and C. Krist, Natural Language Processing and Large Language Models. Cham: Springer Nature Switzerland, 2025, pp. 117–142

work page 2025

[4] [4]

Bisong, Building Machine Learning and Deep Learning Models on Google Cloud Platform: A Comprehensive Guide for Beginners , 01 2019

E. Bisong, Building Machine Learning and Deep Learning Models on Google Cloud Platform: A Comprehensive Guide for Beginners , 01 2019

work page 2019

[5] [5]

Attention is all you need,

A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. Kaiser, and I. Polosukhin, “Attention is all you need,”

[6] [6]

Bert: Pre-training of deep bidirectional transformers for language understanding,

J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova, “Bert: Pre-training of deep bidirectional transformers for language understanding,” 10 2018

work page 2018

[7] [7]

Improving language understanding by generative pre-training,

A. Radford, K. Narasimhan, T. Salimans, and I. Sutskever, “Improving language understanding by generative pre-training,” 2018

2018

[8] [8]

Recent progress in semantic image segmentation,

X. Liu, Z. Deng, and Y . Yang, “Recent progress in semantic image segmentation,” Artificial Intelligence Review, vol. 52, no. 2, pp. 1089– 1106, 2019

work page 2019