{"id":"115e02f3-d754-4807-ac93-2bc023f6228d","arxiv_id":"2602.14612","paper_version":5,"verdict":"REJECT","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"high","formal_verification":"none","parameter_count":4,"one_line_summary":"LA-RAG stores timestamped audio events from a grounding model in SQL and uses intent-aware retrieval plus an LLM to answer long-audio questions, claiming 76.88% accuracy on synthetic home audio.","lead":"This paper introduces LA-RAG, a system that indexes hours of audio as timestamped sound events in a SQL database and answers natural-language questions through retrieval and an LLM. It reports strong accuracy and low latency on synthetic benchmarks, but the evaluation is circular because the question-answer pairs are built from the system's own event detector.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"QA evaluation is circular: Simple-QA pairs are generated from the AGM's own event annotations (Sec. 3.1), so reported accuracies may measure LLM reading of the AGM log rather than audio understanding; abstract's CASTELLA/LALM 11-17% F1 claims are absent from the body.","rationale":"The reader's weakest assumption correctly identifies the circularity in QA generation as the load-bearing issue. Section 3.1 makes the AGM output the source of ground truth for the evaluation, while Sections 3.2 and 4 use the same AGM output as the system's knowledge base; therefore the end-to-end evaluation cannot measure audio understanding independently. The DESED result (74.8 F1) provides partial external validation of the AGM but does not close the loop because the QA benchmark never penalizes AGM detection errors. The additional mismatch between the abstract's CASTELLA-QA/LALM F1 claims and the full text is a separate, objective inconsistency that further weakens the paper's stated contributions. Given the reader already recommended REJECT, my stress-test does not change the verdict; it sharpens the reason: the reported accuracy numbers are consistent with a tautological benchmark rather than evidence of long-audio understanding. A direct oracle-label re-run is the minimal check that would settle the issue; if the authors can show high accuracy on true-label-generated questions, the central architectural claim would be credible despite the current presentation.","tokens_in":11157,"tokens_out":3902,"duration_ms":38589,"concrete_test":"Regenerate evaluation QA pairs directly from the ground-truth event labels used to synthesize the two 24-hour recordings (event class, start, end), rather than from AGM's detections, keeping the question templates and counts fixed. Run the LA-RAG pipeline of Section 3.2 on this new test set and compare with Table 1. If detection accuracy drops materially below 90.67%/90.07% and overall accuracy below 76.88%/71.10%, the reported results are inflated by circular QA generation; if the numbers hold, the circularity concern is substantially mitigated.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim that structured event-level retrieval yields 76.88%/71.10% audio QA accuracy is not independently supported. In Section 3.1, Simple-QA pairs are created from 'ground-truth event annotations produced by the Audio Grounding Model (AGM) module.' The same AGM log is inserted into the SQL database queried by LA-RAG. Consequently the test questions are generated from the very event records the system must retrieve and reason over. If AGM misses an event, no question about it exists; if AGM inserts a false event, the 'correct' answer is the false detection. Detection accuracy near 90% in Table 1 may therefore reflect exact string matching over the AGM log, not faithful audio understanding. The AGM's external DESED F1 of 74.8 (Appendix C) shows the detector is imperfect, but that measure is disconnected from QA. Second, the abstract promises CASTELLA-QA augmentation and 11-17% temporal F1 gains for LALMs, yet the full text contains no CASTELLA-QA experiment, no LALM table, and no F1-gain numbers. These claims are unsupported. Reproducibility is further limited by no released code/data and the statement that the system is proprietary. These issues together undermine the strongest claim as stated.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper presents LA-RAG, a hybrid system for natural-language QA over multi-hour audio. An Audio Grounding Model (AGM) converts audio into timestamped event records; these are stored in a SQLite database. At query time, the system performs query rephrasing, temporal reference resolution, intent classification, embedding/top-k retrieval, and LLM-based answer generation. The authors construct synthetic 24-hour Home-IoT and Industrial-IoT benchmarks with template-based QA pairs, compare against RAG and text-to-SQL baselines, and report overall accuracies of 76.88% and 71.10% with sub-0.6s latency. They also claim that structured retrieval improves accuracy, that CASTELLA was augmented with QA pairs, and that LALMs achieve temporal F1 gains of 11-17% when augmented with their retrieval metadata.","tokens_in":11500,"tokens_out":4991,"duration_ms":54021,"significance":"If independently validated, the core idea is practically significant: event-level structured logs combined with LLM reasoning could offer a low-latency, cost-effective alternative to long-context audio-language models. The paper has concrete strengths: a modular edge-cloud deployment with latency measurements, a sweep across LLM model scales, an external SED evaluation of the AGM on the AudioMarathon/DESED benchmark (74.8 F1), a small human evaluation, and a reasonably detailed system description. However, the evaluation is currently circular, some headline results advertised in the abstract are absent from the body, and the benchmark is synthetic and closed-set. The significance is therefore conditional on substantially stronger evidence than the paper currently provides.","major_comments":[{"comment":"The QA evaluation is circular. Simple-QA pairs are 'created deterministically from ground-truth event annotations produced by the Audio Grounding Model module.' The same AGM event log is the content of the SQL database that LA-RAG retrieves from. Thus a missed AGM event cannot be probed by any question, and an AGM false positive becomes the 'correct' answer. The reported 76.88% / 68.92% accuracies in Table 1 measure the LLM's ability to read the AGM's own event log, not independent audio understanding. The AGM's external DESED F1 of 74.8 (Appendix C) is not connected to QA accuracy. Appendix E's acknowledged reliance on AGM detection quality does not mitigate this design. This undermines the paper's central empirical claim.","section":"Section 3.1, Table 1"},{"comment":"The abstract advertises two results that do not appear anywhere in the body: (i) augmentation of CASTELLA with QA pairs, and (ii) LALM temporal-detection F1 gains of 11-17% when augmented with structured retrieval metadata. No section, table, or appendix reports a CASTELLA-QA experiment or an LALM F1 comparison. These are load-bearing advertised results, not peripheral. Either the experiments must be added with full evaluation details, or the claims must be removed from the abstract.","section":"Abstract vs. full text"},{"comment":"The headline accuracy is ambiguous. The abstract says LA-RAG achieves 71.10% on Industrial-IoT, but Table 1 reports 68.92% for AGM LA-RAG under what appears to be the default Phi-4-mini configuration (Sec. 4.9). The 71.10% value appears only in Table 2 as the Phi-3.5-MoE result. The paper should specify which model configuration underlies the abstract's headline and why Table 1 and Table 2 report different Industrial-IoT numbers for 'LA-RAG.' As written, the reader cannot tell which result is being advertised.","section":"Tables 1 and 2, Abstract"},{"comment":"The synthetic benchmark further limits the evidence. The acoustic classes are restricted to predefined lists, the same closed set is passed to the downstream prompts, and the QA pairs are generated from the same annotation process that produces the database records. This creates a favorable closed-world setting: the LLM is told the answer vocabulary and the database was built from the same event definitions. This may substantially inflate accuracy relative to open-vocabulary, real-world long-audio QA. The authors acknowledge in Appendix E that the benchmark 'does not fully capture the diversity of real-world acoustic environments,' but the evaluation design does not address this concern.","section":"Section 4.1"}],"minor_comments":[{"comment":"The time-resolution evaluation uses only 45 QA pairs, and the hard-condition accuracy is 30% for the combined system. Report more instances, confidence intervals, and an error breakdown by temporal-expression type.","section":"Section 4.7 / Table 3"},{"comment":"The AGM decision threshold (0.8) and median-filter window (0.3s) are held constant across all experiments. A sensitivity analysis would help assess how robust the downstream QA accuracy is to these choices.","section":"Section 4.4"},{"comment":"The RAG baseline uses a fixed k=5, and the AF3-based variant is prompted to produce structured outputs that 'fail to produce reliably structured outputs.' No retrieval-recall numbers are reported, so it is unclear whether the RAG baselines are strong or artificially weak.","section":"Section 4.3"},{"comment":"The human evaluation uses only 11 summarization questions, making the category mean unstable. Report per-item variance or confidence intervals, and consider increasing the sample for the summary category.","section":"Appendix D"},{"comment":"Reproducibility is limited: the code, data, and benchmark are not released, and the system is described as proprietary. Contribution (ii) claims 'a complete and reproducible implementation stack,' but no code or data is provided. The first-person note in Section 1 ('We cannot provide a demo link...') is also unusually informal for a journal submission.","section":"Sections 1, 4.8, F, G"},{"comment":"Figure 1 is referenced without a descriptive caption; Figures 3 and 4 are screenshots that add little. Some citations are inconsistent (e.g., 'Arah A. 2024' for Phi-3). These are presentation issues only.","section":"General"}],"recommendation":"reject","confidential_remarks":"The most serious problem is not merely missing polish but a mismatch between the abstract and the body: the CASTELLA-QA and LALM F1-gain results are absent, and the QA evaluation is circular by construction. The synthetic closed-set benchmark would not, by itself, necessarily be disqualifying for a systems paper, but combined with the circularity it cannot support the strong claims made. If the authors were to include the missing experiments and re-evaluate on independently annotated audio events, a revised submission could be worth reconsidering; I would also recommend reframing the system as an event-log QA architecture rather than as an audio-understanding benchmark."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague — quick take on arXiv:2602.14612. The system is plausible; the evaluation is not. The headline accuracy numbers (76.88% / 71.10%) come from questions built from the AGM's own event annotations, so they measure how well the LLM reads the AGM's log, not how well the system understands audio. That circularity is the load-bearing flaw, and the reader's report has it right.\n\nWhat the paper does well: it puts together a sensible stack — open-vocabulary SED (WSTAG-style), SQL event store, time-expression resolution, intent routing, LLM generation — and shows it running on edge hardware with low latency. The relative comparison of query strategies (LA-RAG vs RAG vs Text2SQL) is informative because all baselines share the same event log; it tells you structured retrieval helps downstream reasoning. The model-scaling table and time-resolution ablation are useful. The DESED F1 of 74.8 gives some independent evidence the detector is not garbage. The limitations section is honest about propagation of AGM errors.\n\nSoft spots in proportion: the circular QA generation is severe. Complex-QA pairs exist but the headline table only reports Simple-QA. The abstract promises CASTELLA-QA and 11-17% F1 gains for LALMs, and these are nowhere in the text — that is an unacceptable abstract-body mismatch. No code or data, and the system is proprietary, so others cannot re-run it. The human eval is small (113 pairs) with moderate GPT-4o correlation. Still, the core architecture could be validated with independent ground truth (e.g., human-annotated events or a public SED dataset re-purposed for QA). It's fixable.\n\nWho gets value: researchers and engineers working on practical long-audio QA, and anyone teaching evaluation design — this is a good case of ground truth contamination. It deserves a serious referee, but the referee should be instructed to check the abstract against the body and demand either independent QA labels or a reframing of the accuracy numbers as 'event-log QA accuracy.' My own verdict is skeptical, but not because the idea is bad — because the evidence as presented does not support the claims as stated.","headline":"A sensible long-audio QA architecture whose headline accuracy numbers are circular — the abstract also claims results the body never reports.","tokens_in":12013,"tokens_out":2305,"would_cite":false,"duration_ms":24308,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Answering questions over multi-hour audio can be done accurately and in under a second by first converting the audio into timestamped event logs and retrieving only the events a query needs.","keywords":["long-audio QA","audio grounding model","retrieval-augmented generation","event detection","temporal grounding","text-to-SQL","structured retrieval","edge-cloud deployment"],"falsifier":"Take a 24-hour recording with an independent human-annotated set of all events (not derived from the AGM), ask the system questions about events the AGM missed, and check whether it answers them correctly; if it fails on those questions, the accuracy numbers do not reflect true audio understanding. A simpler version: compute LA-RAG's recall of human-annotated events and see whether it matches the detection accuracy reported.","tokens_in":11022,"feed_emoji":"🎧","tokens_out":6872,"duration_ms":57809,"temperature":0.7,"pith_summary":"The paper tries to show that natural-language question answering over multi-hour audio does not require feeding the whole recording to a large audio-language model. Instead, the system converts audio into timestamped event records using an open-vocabulary grounding model, stores them in a SQL database, and answers each query by resolving time references, classifying intent, retrieving only the relevant events, and letting a compact LLM generate a grounded response. On synthetic 24-hour Home and Industrial IoT benchmarks, this structured pipeline reaches 76.88% and 71.10% overall accuracy with average query latencies below 0.6 seconds, substantially outperforming vanilla retrieval-augmented generation (RAG) and text-to-SQL baselines. The paper also shows that feeding this structured event metadata to existing audio-language models improves their temporal detection F1 by 11 to 17 percentage points. The significance is a practical, deployment-oriented route to long-audio QA that is cheaper, faster, and more temporally precise than purely generative approaches.","feed_headline":"Indexed events let LLMs answer 24-hour audio in 0.6 s","feed_subtitle":"Convert audio to timestamped events, retrieve relevant ones, and a small LLM answers — beats RAG and text-to-SQL.","key_machinery":"The Audio Grounding Model (AGM) — an open-vocabulary text-to-audio grounding network that outputs timestamped event records (event name, start/end times, confidence, loudness) for free-form sound queries. These logs feed a SQL event database; at query time, a pipeline of rephrasing, time resolution, intent classification, and embedding-based top-k retrieval selects the evidence that the LLM must use to answer.","core_discovery":"The central claim is that explicit timestamped grounding via structured event logs, followed by intent-aware retrieval and LLM generation, yields large accuracy gains over both retrieval-augmented generation on raw event text and text-to-SQL over the database. Detection, counting, and summary tasks all improve, and the system stays fast because audio grounding runs once at index time on edge hardware, while only filtered event records are sent to a cloud-hosted LLM. The paper attributes these gains to three design choices: the AGM produces well-structured event logs; a time-resolution module maps varied natural-language expressions (12-hour, 24-hour, shift references, before/after, durations","pith_inferences":["Because the Simple-QA ground truth is generated from the AGM's own event detections, the reported accuracy measures consistency with the AGM's event log, not independent acoustic understanding; the AGM's detection quality is therefore the true ceiling of the whole system.","The evaluation uses a predefined closed set of sound classes, so open-vocabulary performance in unrestricted acoustic environments remains untested; real-world deployments with novel sounds could degrade.","The synthetic benchmark concatenates short labeled clips with template-generated questions, which may yield higher accuracy than naturally continuous audio with human-annotated events; a real-world corpus would be a stronger test.","The same 'convert raw signal to structured events first, then reason' pattern could extend to other long modalities such as video or sensor streams, where context-length limits similarly block direct modeling."],"forward_implications":["Long-audio QA can be answered accurately without ingesting raw audio into the model, sidestepping context-length limits of current audio-language models.","Sub-second query latency is achievable even for 24-hour recordings when event extraction happens offline at index time.","Mid-sized LLMs (roughly 7–14B active parameters) are sufficient when given structured, timestamped evidence; very large models are not required.","Adding structured retrieval metadata improves the temporal detection F1 of existing audio-language models by 11–17%, showing event logs complement generative models.","A hybrid edge–cloud deployment keeps raw audio local for privacy while transmitting only compact event metadata to the cloud LLM."],"fun_headline_variants":["Timestamped event logs unlock 24-hour audio QA in 0.6s","Structured retrieval beats RAG and text-to-SQL for long audio QA","Event grounding lifts temporal F1 by 11–17% on audio QA","Offline indexing makes 24-hour audio question answering fast","LA-RAG: Convert audio to events, retrieve, answer in 0.6s"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"The evaluation's ground truth for Simple-QA pairs comes from the AGM's own event annotations, so questions only exist about events the detector found; any event the AGM misses cannot be probed, and the reported accuracy is not an independent measure of audio understanding.","fun_headline_variants_meta":{"raw":{"variants":["Timestamped event logs unlock 24-hour audio QA in 0.6s","Structured retrieval beats RAG and text-to-SQL for long audio QA","Event grounding lifts temporal F1 by 11–17% on audio QA","Offline indexing makes 24-hour audio question answering fast","LA-RAG: Convert audio to events, retrieve, answer in 0.6s"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000317,"raw_usage":{"total_tokens":1667,"prompt_tokens":821,"completion_tokens":846,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":565,"completion_tokens_details":{"reasoning_tokens":745}},"tokens_in":565,"tokens_out":846,"duration_ms":8132,"temperature":1.0,"reasoning_tokens":745,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-02T23:08:08.545044+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a 24-hour recording with an independent human-annotated set of all events (not derived from the AGM), ask the system questions about events the AGM missed, and check whether it answers them correctly; if it fails on those questions, the accuracy numbers do not reflect true audio understanding. A simpler version: compute LA-RAG's recall of human-annotated events and see whether it matches the detection accuracy reported.","supporting_citations":[],"review_version":1}