{"id":"6b7b5f50-f057-49bf-bd82-73da80e0d748","arxiv_id":"2607.26631","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"RAG-HAR+ adapts retrieval features offline via an LLM agent and routes only ambiguous windows to the LLM, cutting online LLM cost by ~90–99.9% with competitive accuracy on six HAR benchmarks.","lead":"RAG-HAR+ is a retrieval-first system for recognizing human activities from wearable sensors that uses an LLM offline to pick better features and calls the LLM online only for ambiguous cases. It reports competitive accuracy on six HAR benchmarks while cutting online LLM tokens by 89–99.9% and per-sample latency by roughly 6–44×.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Validation-tuned retrieval design and single-split evaluation leave the 89.3–99.9% token-reduction claim unquantified; repeated-split evidence is needed before the accuracy–cost trade-off can be trusted.","rationale":"The reader's weakest assumption identifies the core issue: validation-based selection of feature groups, weights, K, q, and prompt representation is assumed to transfer from validation to test and across datasets. I sharpen this: because the fallback rate and hence the 89.3–99.9% token reductions are directly determined by the validation-selected retrieval configuration, the cost claim inherits any instability in that selection. The absence of repeated splits and error bars means the reported point estimates cannot be distinguished from selection noise. A secondary red flag is internal: Table III reports only ~22k offline tokens per dataset, while Section III-B2 describes multi-round iterative improvement prompts with per-class feedback; unless only one round was run, the offline cost is understated. This reinforces the need for released logs but does not replace the primary concern about transfer. The verdict remains CONDITIONAL: the experiments are suggestive, but the central accuracy–cost trade-off is not yet robustly established.","tokens_in":19828,"tokens_out":12584,"duration_ms":146950,"concrete_test":"Run the complete RAG-HAR+ offline-design plus inference pipeline on five non-overlapping training/validation/test splits of at least USC-HAD and MHEALTH, using subject-disjoint splits and contiguous time-based splits for single-subject datasets such as Skoda. Report mean ± std of final F1 and online token reduction across splits. If the mean accuracy delta versus RAG-HAR is within noise or the fallback rate varies materially across splits, the headline reductions are not robust. Additionally, release the round count and per-round prompt-token logs to verify the Section III-B2 iterative design against the offline costs in Table III.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim—competitive accuracy with 89.3–99.9% fewer online tokens—rests on the retrieval representation produced by the offline Retrieval Designer Agent. Section III-B2 describes an iterative process in which candidate feature groups are evaluated on a validation split and the best configuration across rounds is kept. Section IV-E additionally selects K=3, q=10, feature-group weights, and the Ambiguity Resolver prompt representation from validation ablations, mostly on USC-HAD. The reported test accuracy and the fallback rate (which directly determines the token reductions in Table IV) are therefore properties of the configuration that maximized validation performance over an unreported number of candidates. The paper reports one test split per dataset, no confidence intervals, no significance tests, and no accounting for the number of design rounds searched. For datasets where two subjects form the validation split, and Skoda uses a 20% window-level split, the selection signal is weak; if the validation-tuned groups, weights, K, and q do not transfer to the test split or to other datasets, the claimed cost reduction and 'competitive' accuracy are not a stable property of RAG-HAR+ but an artifact of the selected configuration. This is especially salient because two of six datasets (PAMAP2, GOTOV) already show accuracy below RAG-HAR; the 'competitive' interpretation is doing work. No code, data, or prompt logs are released, so the number of evaluated candidates and the actual offline rounds are not auditable.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"RAG-HAR+ extends the authors' prior RAG-HAR framework for training-free, LLM-based human activity recognition. It introduces an offline Retrieval Designer Agent that iteratively selects three dataset-specific feature groups from a motion-descriptor pool using validation-set feedback, followed by online multi-vector retrieval, weighted re-ranking, and majority-vote classification over the top-q neighbors. Only tie cases are sent to an LLM Ambiguity Resolver Agent. The paper evaluates on six public HAR datasets and reports competitive or improved accuracy relative to RAG-HAR while reducing online LLM token usage by 89.3–99.9% and per-sample latency by 5.7–44.5×, together with a smartphone prototype demonstrating the feasibility of the approach in a mobile setting.","tokens_in":20149,"tokens_out":5222,"duration_ms":61499,"significance":"If the reported accuracy–cost trade-offs are stable, this is a meaningful step toward practical LLM-based HAR: it shifts the LLM from a per-sample online classifier to a one-time offline retrieval designer plus a selective fallback, and it provides a careful decomposition of the factors that make the retriever effective. The paper's strengths include a broad ablation study (feature-group selection, number of groups, group assignment, group weights, retrieval depth q, and prompt representation), a clear separation of offline and online costs, and an actual mobile prototype with real network round-trip latency. These strengths are undermined by the absence of repeated evaluations, error bars, significance tests, and released code/data/prompt logs; as presented, the central cost-reduction claim is not yet shown to be a stable property of the method rather than of the specific validation-selected configuration.","major_comments":[{"comment":"The headline 89.3–99.9% token-reduction claim is not yet stable because the retrieval configuration is selected on validation splits and then evaluated on a single test split. The Retrieval Designer Agent iteratively proposes feature groups and keeps the best validation configuration; weights (w1,w2,w3) are grid-searched on validation; K=3 and q=10 are chosen from USC-HAD validation; and the Ambiguity Resolver prompt representation is also chosen on USC-HAD validation. Since the fallback rate is exactly the rate at which top-q voting ties, these choices directly determine the token reductions in Table IV. The validation signals are weak for several datasets (two held-out subjects; a 20% window-level split for Skoda), and no number of design rounds or candidate configurations is reported. Please report the search effort, repeat the full protocol over multiple subject splits or bootstrap r","section":"§III-B2, §IV-E1d, Table IV"},{"comment":"The claim that RAG-HAR+ maintains 'competitive or improved' performance is not statistically supported on the two datasets where it is below RAG-HAR. Table II reports GOTOV F1 of 75.97 versus 79.92 for RAG-HAR, and PAMAP2 F1 of 90.60 versus 91.12, with no error bars, no confidence intervals, and no significance tests. Because each dataset is evaluated once, and the LLM fallback is stochastic, the observed gaps could be within run-to-run noise. The abstract's 'competitive' interpretation needs explicit support for these datasets, for example repeated runs with different seeds, paired significance tests, or at least confidence intervals around the reported point estimates.","section":"§IV-A, Table II"},{"comment":"The amortization argument for the offline design cost is under-specified. Table III reports only the final total of 132,176 tokens across six datasets, but not how many Retrieval Designer Agent rounds were executed, how many candidate feature-group configurations were evaluated, or the stopping criterion used. This matters both for the 'one-time cost' claim and for the overfitting concern: the number of validation-maximizing candidates is the effective number of comparisons behind the selected G*. Please report the per-dataset round count, the validation-score trajectory, and the number of candidates considered, and incorporate those numbers into the cost comparison.","section":"§IV-B, Table III"},{"comment":"No code, data, exact prompt templates, prompt logs, or design-round traces are released. Because the Retrieval Designer Agent's outputs drive every downstream result, the paper should at least release the exact prompts, the per-round feature-group proposals, the selected feature groups, and the grid-search implementation, or provide a detailed reproducibility appendix. Without these artifacts, the validation-selection process and the reported token reductions cannot be independently recomputed or audited.","section":"Reproducibility"}],"minor_comments":[{"comment":"Typo: 'Ambuiguity' should be 'Ambiguity'.","section":"§I, final bullet"},{"comment":"The heading 'F . Discussion' has an unwanted space after the initial letter.","section":"§IV-F"},{"comment":"The validation-split protocol differs across datasets: most use two held-out subjects, while Skoda uses a random 20% window-level split. The paper should justify this difference, since window-level splits can create subject or temporal leakage that subject-level splits avoid.","section":"§IV-E1d"},{"comment":"The token counts are reported for gpt-5-mini, but the paper does not state whether these counts include the fixed system-prompt overhead or only the sample-dependent content. A clear statement of what is included in 'tokens per sample' would make the comparison with RAG-HAR easier to interpret.","section":"Table IV"}],"recommendation":"major_revision","confidential_remarks":"The stress-test concern lands: the paper's strongest quantitative claims rest on a single test split of a validation-tuned configuration, and the search effort is not reported. I would ask for repeated-split evidence and release of the design-round artifacts before considering acceptance. The work is not fatally flawed; the reported approach is plausible and the ablations are useful, but the current evidence does not yet support the strength of the abstract's cost-reduction claim."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nQuick take: this is a solid engineering contribution that moves the RAG-HAR line forward, and the cost numbers are eye-catching. The core idea—split the LLM into an offline retrieval designer and an online fallback for ambiguous cases—is genuinely new and well motivated. The offline designer iteratively picks three feature groups from a pool, and the online voter handles most samples, so online token use drops by 89–99.9% and latency by 5.7–44.5× across six benchmarks. The ablations are thoughtful: they isolate feature selection from grouping, show the grouping itself helps over random partitions, and search over the number of groups, retrieval depth, and prompt representation. The mobile prototype is a nice extra.\n\nWhere I would push back: the evaluation is too thin to support the strongest version of the claim. One test split per dataset, no error bars, no significance tests, no code or data release. That matters because almost every design choice—feature groups, weights, K, q, prompt format—is tuned on validation, mostly on USC-HAD, and then applied to the other five datasets. The stress-test note is right: the token reductions and accuracy are properties of a configuration selected from an unquantified number of validation candidates, and we cannot tell how much that selection pressure affects the test numbers. Also, on PAMAP2 and GOTOV, RAG-HAR+ is actually a few points below RAG-HAR, so \"competitive\" is doing a bit of work there.\n\nThe central argument is not circular—the test sets are held out and the routing rule is parameter-free—but the generalizability claim is under-supported. This is fixable: rerun with multiple splits, report variance, release the prompt logs and feature-group selections, and maybe show how robust the chosen groups are across validation splits.\n\nWho is this for? Anyone working on retrieval-augmented inference, edge AI, or HAR will find it a useful reference. The cost breakdown and the idea of amortizing LLM design cost offline are worth taking seriously.\n\nRecommendation: send it to peer review. The idea is new enough and the empirical work is careful enough that a referee can help the authors tighten the evaluation. I would not desk-reject this; I would ask for repeated-split results and artifacts before accepting.\n\nBest.","headline":"A practical, clever cost-reduction extension of RAG-HAR with a strong cost story, but the validation-tuned design and single-split evaluation mean the headline numbers need more evidence before I'd trust them across datasets.","tokens_in":20707,"tokens_out":1131,"would_cite":true,"duration_ms":15910,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"RAG-HAR+ shows that LLM-based activity recognition can keep accuracy while cutting online LLM calls by 89.3–99.9% and per-sample latency by 5.7–44.5×, by moving the LLM into an offline design stage and a small online fallback role.","keywords":["Human Activity Recognition","Retrieval-Augmented Generation","Large Language Models","Edge Deployment","Cost Optimization","Feature Group Selection","Majority Voting","Wearable Sensors"],"falsifier":"A decisive check is to re-run the Retrieval Designer Agent with a different validation split (or hold out entire subjects not used anywhere in design) and compare the resulting feature groups, online LLM call rate, and F1 to the reported numbers; if the 89.3–99.9% token reductions and competitive F1 scores collapse when the validation split changes, the cost-efficiency result is an artifact of validation tuning rather than a property of the retrieval-first design.","tokens_in":19702,"feed_emoji":"📱","tokens_out":2568,"duration_ms":27964,"temperature":0.7,"pith_summary":"RAG-HAR+ is a training-free extension of retrieval-augmented human activity recognition that tries to make LLM-based HAR cheap enough for continuous edge deployment. Its central idea is to stop calling an LLM for every sensor window. Instead, an offline Retrieval Designer Agent uses the LLM once per dataset to choose compact, dataset-specific feature groups for similarity search, and online the system classifies most windows directly by majority vote over retrieved labeled neighbors, routing only undecided cases to an Ambiguity Resolver Agent. Across six HAR benchmarks the paper claims competitive or improved recognition accuracy while reducing online LLM tokens by 89.3–99.9% and per-sample latency by 5.7–44.5×, with a smartphone prototype showing similar gains in a live mobile setting. A sympathetic reader would see this as a practical route to retrieval-first, LLM-assisted HAR that scales with ambiguity rather than data volume.","feed_headline":"LLM calls drop up to 99.9% in retrieval-first HAR","feed_subtitle":"A training-free activity recognizer routes only ambiguous sensor windows to the LLM, keeping accuracy while slashing latency.","key_machinery":"The load-bearing mechanism is the Retrieval Designer Agent, an offline LLM prompted iteratively with validation feedback to design three compact, non-overlapping feature groups from a 67-feature candidate pool per dataset. These groups define a multi-vector index where each window is represented by three normalized numerical vectors; during inference, weighted multi-vector search (with weights grid-searched on validation) merges and re-ranks candidates, and a deterministic majority-voting routing rule decides whether a query is answered directly from retrieval or deferred to the Ambiguity Resolver Agent. The separation of a numerical-vector retriever from a segment-wise textual prompt repres","core_discovery":"The paper claims that the dominant barrier to deploying LLM-based HAR is not accuracy but the per-sample online LLM call, and that this barrier can be removed by changing when and where the LLM is used. RAG-HAR+ moves the LLM upstream into a one-time offline Retrieval Designer Agent that selects three complementary feature groups from a pool of statistical, temporal, spectral, and signal-shape descriptors, tuned on a validation split. Online, it indexes training windows as multi-vector records, retrieves top-q neighbors under validation-tuned group weights, and classifies by parameter-free majority voting whenever one label has a unique plurality; only genuinely ambiguous ties are sent to th","pith_inferences":["The same two-stage pattern—offline LLM design of a retrieval representation, online majority-vote routing with selective fallback—could transfer to other time-series classification tasks beyond HAR, such as gesture recognition, sleep staging, or anomaly detection, whenever labeled exemplars are plentiful.","A natural extension the paper leaves implicit is replacing the tie-based routing rule with a soft uncertainty threshold (e.g., vote-margin or label entropy), which could reduce fallback calls further or recover accuracy on datasets where ties are rare but near-miss majorities are common.","Since the Retrieval Designer Agent's groups and weights are chosen on a validation split of each dataset, the approach would only be truly 'zero-shot' for a new dataset if those design choices transfer without re-running the offline stage; testing this transfer directly would clarify how much per-dataset design cost remains.","The privacy framing is deliberately scoped out: routing only ambiguous windows to a cloud LLM reduces raw sensor data exposure compared with sending every window, but the ambiguous windows still carry sensitive motion information, so quantifying that residual exposure would be a valuable follow-up."],"forward_implications":["Online inference cost under RAG-HAR+ scales with the number of ambiguous windows rather than the total number of test samples, making continuous streaming HAR practical on phones and edge gateways.","Because the offline design cost is amortized over all later inference, new datasets can be handled without training a classifier or fine-tuning an LLM, requiring only a one-time feature-group design pass.","The reported per-sample latency of 0.41–2.04 s across benchmarks is compatible with near-real-time activity recognition on mobile devices, as demonstrated by the smartphone prototype.","The framework keeps a small but useful role for LLM reasoning: the Ambiguity Resolver Agent adds 0.1–3.3 percentage points of F1 on top of retrieval voting, justifying its retention for confusing activity pairs.","The design suggests that retrieval quality, not LLM reasoning, is the primary levers for both accuracy and cost in retrieval-augmented sensor classification."],"fun_headline_variants":["Retrieval-first HAR cuts 99.9% of LLM calls, stays accurate","Training-free HAR: LLM only for uncertain windows","RAG-HAR+ slashes LLM cost 99.9% for edge HAR","Edge HAR with 99.9% fewer LLM calls, no training needed","HAR on edge: retrieval decides, LLM only when ambiguous"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"The claimed accuracy and cost savings depend on the assumption that the feature groups, group weights, K=3, q=10, and routing rule chosen on the validation split of one dataset — mostly USC-HAD — transfer to the test split and to the other five datasets without overfitting or dataset-specific bias.","fun_headline_variants_meta":{"raw":{"variants":["Retrieval-first HAR cuts 99.9% of LLM calls, stays accurate","Training-free HAR: LLM only for uncertain windows","RAG-HAR+ slashes LLM cost 99.9% for edge HAR","Edge HAR with 99.9% fewer LLM calls, no training needed","HAR on edge: retrieval decides, LLM only when ambiguous"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000258,"raw_usage":{"total_tokens":1433,"prompt_tokens":774,"completion_tokens":659,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":518,"completion_tokens_details":{"reasoning_tokens":558}},"tokens_in":518,"tokens_out":659,"duration_ms":6912,"temperature":1.0,"reasoning_tokens":558,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-01T11:54:46.543480+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"A decisive check is to re-run the Retrieval Designer Agent with a different validation split (or hold out entire subjects not used anywhere in design) and compare the resulting feature groups, online LLM call rate, and F1 to the reported numbers; if the 89.3–99.9% token reductions and competitive F1 scores collapse when the validation split changes, the cost-efficiency result is an artifact of validation tuning rather than a property of the retrieval-first design.","supporting_citations":[],"review_version":1}