{"id":"cbfa5422-313a-4092-b8ab-bd8dd0adb61e","arxiv_id":"2412.13799","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"An ontology-based German rhetorical figure annotation web app with RAG, where basic chunking at 2048 tokens yields the best Ragas scores.","lead":"The authors built a web application called “Find your Figure” that guides users through identifying rhetorical figures in German texts using a restructured ontology and a chatbot. It is an early practical combination of rhetorical-figure ontologies and retrieval-augmented generation, aimed at closing the gap in non-English annotated data.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The claim that untrained users can annotate via the reified property dropdowns is untested; Section 7 admits failure for metaphors, so the data-collection pipeline lacks its key validation.","rationale":"The reader's weakest assumption pinpoints the lay-user mapping, and I agree that this is the load-bearing condition. The paper's own Section 7 is an explicit admission that the workflow fails for transferred-meaning figures; this is the strongest in-text evidence. The reification in Section 3 creates a three-dimensional property space whose cognitive demands on non-experts are never evaluated. The RAG evaluation weaknesses (single-run metrics, same-ontology ground truth, no no-RAG baseline) are real but secondary: even if the optimal RAG configuration were perfectly identified, the annotation pipeline's value would still depend on users' ability to select the correct properties. Conversely, if a user study showed non-experts can reliably annotate lexical-repetition figures, the app would meet its stated goal for a meaningful subset of German rhetorical figures, and the RAG claims could be strengthened with a more rigorous evaluation later. I therefore keep the verdict at CONDITIONAL, adding an explicit condition: a user study demonstrating reliable annotation by the target population. I do not see internal inconsistency; the paper is honest about limitations and provides code and data links. The concern is absence of evidence for a central assumption, not a demonstrated contradiction.","tokens_in":12725,"tokens_out":9122,"duration_ms":79600,"concrete_test":"Recruit 20-30 German-speaking participants from the intended population (adults or school students without linguistic training) and have them each annotate 40 expert-validated sentences using the deployed 'Find your Figure' app: 20 sentences containing lexical-repetition figures (e.g., anaphora, epiphora, alliteration) and 20 containing transferred-meaning figures (e.g., metaphor, metonymy, irony). For each sentence, record whether the property dropdown selections yield a SPARQL result list containing the gold figure and whether the participant selects it. Compute Cohen's kappa or macro-F1 against expert labels, separately for the two figure classes. If kappa is below 0.6, or if the lexical-figure success rate is not significantly above chance, the central assumption fails and the app's output cannot be used as training data without per-item expert verification.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The load-bearing assumption is that users without linguistic knowledge can reliably map a text onto the reified property dimensions introduced in Section 3—operation, affected element, and operational form—and that the SPARQL query built from those selections in Section 4.1 returns the intended figure. The paper provides no evidence for this: there is no user study, no inter-annotator agreement, and no pilot test of the FyF.html dropdown workflow. Section 4.2 admits that verifying assigned figures is 'highly challenging' and still depends on manual administrator checks via the is_verified column, so the app cannot guarantee annotation quality at scale. Most tellingly, Section 7 concedes that the application is better suited to figures with obvious lexical patterns than to figures relying on transferred meaning, such as metaphors; the authors frame this as a user-side limitation, but it is exactly a failure of the core annotation workflow for a major class of figures. The reified dimensions are also internally inconsistent (Section 3 uses 'operationalForm', Table 1 uses 'hasOperationForm'), which underlines the lack of a validated property model. Since the paper's stated goal (Section 4) is to collect reliable annotated instances to improve detection, an unvalidated user mapping would compromise the collected data and defeat the central contribution.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper presents \"Find your Figure,\" a Flask-based web application for annotating rhetorical figures in German, built on a reified version of the GRhOOT ontology. Users select properties such as operation, affected element, and operational form from dropdowns; the backend translates these into a SPARQL query and returns candidate figures. A separate chat page integrates a retrieval-augmented generation (RAG) pipeline using gpt-3.5-turbo, bge-m3 embeddings, and a reranker. The authors evaluate several chunk sizes and retrieval settings with the Ragas framework on 70 competency questions derived from the ontology, and they conclude that basic chunking with chunk size 2048 and top-12 retrieval reranked to top-6 is the optimal configuration. They also describe input verification mechanisms, limitations for figures based on transferred meaning, and ethical considerations around intellectual property.","tokens_in":12971,"tokens_out":3918,"duration_ms":36626,"significance":"The paper addresses a real bottleneck in computational rhetoric: the scarcity of annotated data for non-English and non-metaphor rhetorical figures. The ontology restructuring, the openly available code, and the integration of RAG with a rhetorical ontology are useful contributions if the system works as described. The significance is currently limited, however, because the annotation workflow is not validated with users, and the RAG evaluation does not compare against a no-RAG baseline or provide error bars. The claim of identifying the optimal RAG configuration is therefore not yet supported, and the central data-collection goal rests on an untested assumption about untrained annotators.","major_comments":[{"comment":"The RAG evaluation is circular with respect to the ontology. The 70 competency questions are either template questions asking for properties of figures or questions generated by gpt-4 from formalizations of the same ontology; the ground-truth answers are obtained by running SPARQL queries on that ontology, and the context is manually extracted from that same ontology. Since the retrieval source and the answer source are identical, high context recall and high faithfulness partly measure whether the retriever finds the exact passage from which the answer was derived, not whether the pipeline produces correct answers on an independent task. Please either reframe the evaluation as a retrieval-coverage check on a single knowledge source or construct ground truth from an external corpus (e.g., dictionary definitions or expert-verified examples).","section":"§5.1"},{"comment":"The choice of basic chunking 2048 with top-12/6 as the optimal setting is not consistent with the stated priority on answer metrics. In Table 2, this row has answer correctness 0.7355 and answer similarity 0.8655, while the AMR 512/256/128 top-6/3 row has answer correctness 0.8619 and the basic 2048 top-6/3 row has answer similarity 0.9673. If answer correctness and answer similarity are the most important metrics, as claimed at the end of §5.1, the selected configuration is not the best on those metrics. Please specify the exact selection criterion (e.g., a weighted combination or a threshold) and justify why the chosen setting is preferred over rows that score higher on answer correctness or answer similarity.","section":"§5.2, Table 2"},{"comment":"The core annotation workflow rests on an untested assumption: that users without linguistic knowledge can map a text onto the reified property dimensions (operation, affected element, operational form) and that the resulting SPARQL query reliably surfaces the intended rhetorical figure. The paper provides no user study, no inter-annotator agreement, and no pilot test of the FyF.html dropdown workflow. Section 7 concedes that the application is better suited to figures with obvious lexical patterns than to figures relying on transferred meaning, such as metaphors. That is not merely a user-side limitation; it directly limits the kind of data the application can collect. Please report at least a small pilot evaluation with untrained users, including agreement or accuracy against expert labels, or explicitly scope the application's data-collection claim to lexical-pattern figures.","section":"§4.1, §7"},{"comment":"The evaluation compares only RAG configurations and never includes a no-RAG baseline, such as prompting the same LLM without retrieved ontology context or a retrieval-only baseline. Without such a baseline, the claim that RAG improves answer quality over a plain LLM is not supported. In addition, Table 2 reports a single run per configuration without standard deviations or repeated trials; even at temperature 0.1, LLM output is stochastic and the post-processing step (described in §5.1) could introduce run-to-run variation. Please add a no-RAG baseline and repeated runs with variance reporting, or explicitly weaken the conclusion to a comparison among the tested RAG settings rather than an identification of the globally optimal configuration.","section":"§5"}],"minor_comments":[{"comment":"The property name is inconsistent: Section 3 writes `:operationalForm :SameForm`, while Table 1 uses `hasOperationForm`. Please standardize the property names in the text, the table, and the ontology itself.","section":"§3, Table 1"},{"comment":"The sentence \"The answers of the LLM still require post-processing before we can use them in the Ragas framework\" is clear, but the paper does not explain whether the same post-processing is applied consistently to all configurations or whether it could interact with the chunking settings. Please clarify in the evaluation description.","section":"§5.1"},{"comment":"The text says \"we will flag the example in the database in the columnis_invalid,\" which appears to be a missing-space typo for `is_invalid`. Please fix the formatting.","section":"§4.2"},{"comment":"The phrase \"We notice deviations in answer correctness and answer similarity ,\" contains an extra space before the comma. Please correct the typo.","section":"§5.2"}],"recommendation":"major_revision","confidential_remarks":"This appears to be a system/demo-oriented paper. The web application and ontology reification are described in enough detail to be credible, and the authors are honest about several limitations. However, the central contributions—an annotation workflow for untrained users and the identification of optimal RAG settings—are not yet supported by the evidence. The evaluation circularity, the absence of a no-RAG baseline, and the lack of any user validation are load-bearing issues that require additional experiments before the paper can be accepted. If the authors can add a small user pilot and a no-RAG baseline, with error bars, the paper could become a useful resource for the community."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The paper delivers a real, working system: a reified version of the German GRhOOT ontology, a web application called “Find your Figure” for annotating rhetorical figures, and a RAG pipeline tuned over chunking and retrieval settings. The engineering is credible and the code is on GitHub. The reification of compound relations into operation/affectedElement/operationalForm is a sensible simplification for querying, and converting figures from individuals to classes is a defensible design choice. The related-work claim that no existing rhetorical ontology has been applied in a practical annotation scenario holds up against the cited literature. Credit where due: this is a useful building block for German rhetorical-figure annotation.\n\nThe soft spots are real and mostly where the reader put them. The RAG evaluation is the weakest section. Ground-truth answers for Ragas are generated from the same ontology that serves as the retrieval source, so high faithfulness and context recall partly measure self-consistency, not external correctness. There is no no-RAG baseline, no repeated runs, no error bars. The “optimal setting” claim rests on single numbers, and the answer metrics actually point in different directions across configurations (answer correctness favors AMR top-6/3, answer relevance favors basic top-12/6). The authors note wordiness hurts correctness, but that is an artifact of the metric, so comparative claims are shaky.\n\nThe deeper issue, which the stress-test note correctly flags, is that the paper’s stated goal—collecting reliable annotations from lay users—is untested. There is no user study, no inter-annotator agreement, no pilot of the dropdown workflow. The core assumption, that users can map a sentence onto operation/affectedElement/operationalForm, is asserted in Section 4.1 (“we do not see any drawbacks”) but never demonstrated. Section 7 concedes the app suits lexical patterns better than transferred-meaning figures like metaphor, which is exactly the class where annotation help is most needed. There is also a minor terminology inconsistency: Section 3 uses “operationalForm” while Table 1 uses “hasOperationForm.” Minor, but it suggests the property model is not fully settled.\n\nWho is this for? Researchers working on rhetorical-figure detection in German, and people building ontology-grounded RAG systems. It is a systems paper, not a results paper. It deserves a serious referee, but the authors need to add a non-RAG baseline, report variance, and validate the annotation workflow with a small user study. As is, the claims outrun the evidence.","headline":"A genuine engineering contribution—reified GRhOOT ontology, a German annotation web app, and a RAG pipeline—whose evaluation is too weak to support the claims as written.","tokens_in":13508,"tokens_out":2047,"would_cite":true,"duration_ms":20169,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that combining a reified rhetorical-figure ontology with retrieval-augmented generation lets untrained users annotate German rhetorical figures, with basic chunking (size 2048, top-12 reranked to top-6) as the best RAG…","keywords":["rhetorical figures","ontology reification","GRhOOT ontology","retrieval-augmented generation","annotation tool","German language","data scarcity","RAG evaluation"],"falsifier":"Give the application to a group of untrained, non-expert German speakers with a set of sentences that contain figures based on transferred meaning, such as metaphors, and measure how often their property selections lead to the correct figure name. A success rate near chance for those figures would show that the property-based flow does not support untrained annotation for the very class of figures the paper admits are hardest.","tokens_in":12492,"feed_emoji":"🗣️","tokens_out":9826,"duration_ms":78774,"temperature":0.7,"pith_summary":"The paper builds a web application, “Find your Figure,” that lets people without linguistic training identify and annotate rhetorical figures in German by selecting descriptive properties of a text instead of knowing figure names. The German rhetorical ontology GRhOOT is restructured, or reified, so that compound relations are split into three small, queryable property dimensions: operation, affected element, and operational form. To make interaction more natural, the app adds a chat feature powered by retrieval-augmented generation, in which the reified ontology is chunked, embedded, and retrieved to ground a large language model's answers. The paper identifies the best RAG configuration for this application as basic chunking with a chunk size of 2048 and top-12 retrieval reranked to top-6. The authors state this is one of the first practical uses of a rhetorical ontology for figure annotation and that it shows promising results, toward easing the scarcity of annotated German rhetorical figures.","feed_headline":"RAG-guided app lets novices tag German rhetorical figures","feed_subtitle":"Untrained users get candidate figure names via property dropdowns and a grounded chat, tackling scarce German data.","key_machinery":"The reified GRhOOT ontology is the central object. In the original ontology, a figure like epiphora was described by compound relations such as 'isRepeatableElementOfSameForm'; reification splits these into three fine-grained properties, hasOperation, affectedElement, and operationalForm, on which users can filter. This decomposition is what lets a non-expert describe a figure's pattern instead of knowing its name. The second load-bearing mechanism is the RAG pipeline: the ontology is chunked and embedded into a vector store, a user question is embedded and matched against chunks, the top chunks are reranked, and a language model answers with that retrieved context. The paper evaluates the pipeline using ontology competency questions and an automated RAG evaluation suite, comparing chunk sizes, chunking methods, and retrieval depths to select the best setting.","core_discovery":"The paper's central claim is that an ontology of rhetorical figures can be made practically useful for annotation by reifying its relations into user-facing property dimensions and by grounding a language-model chat in retrieval from that ontology. A user who spots a pattern, such as repetition of a word at the same position, selects the matching properties from dropdowns; the app translates the selection into a SPARQL query and returns candidate figures with definitions and examples. The same reified ontology serves as the retrieval source for a RAG-powered chat, and the paper reports that a basic chunking setting (chunk size 2048, retrieve top-12 then rerank to top-6) yields the best answer-oriented metrics among the configurations tested. The approach is described as one of the first to combine a rhetorical ontology with RAG for practical annotation, and the paper claims it shows promising results, while noting that figures based on transferred meaning, such as metaphor, remain harder for this property-selection flow.","pith_inferences":["The same reification pattern could transfer to other low-resource annotation domains: any domain ontology whose compound relations can be split into user-facing property dimensions could power a similar property-selection interface.","The RAG result was tuned on a small, compact ontology; larger or more heterogeneous knowledge bases might not favor basic chunking at 2048, so the optimal setting is likely scale-dependent.","Because the LLM's wordy answers and modern paraphrasing lowered semantic-similarity scores, future evaluations should separate factual correctness from phrasing, for example by also scoring exact figure-name matches.","A head-to-head comparison of the dropdown property flow against the RAG chat flow, using the same sentences and annotators, would show which interface yields more accurate annotations and where the 'No idea' fallback is needed."],"forward_implications":["Users without linguistic training can submit German examples and receive candidate figure names with definitions, which lowers the barrier to creating annotated data.","Because figures can be queried by operation, affected element, or operational form, the same ontology supports flexible search paths rather than requiring users to know figure names in advance.","The RAG configuration identified in the paper, basic chunking with a chunk size of 2048 and top-12 to top-6 reranking, can serve as a default starting point for other ontology-based RAG applications.","Questions that require aggregating information across multiple ontology chunks (for example, listing all figures with a given property) remain a weak spot for the RAG chat, so the app is better suited to direct look-ups.","Once user-submitted examples are verified, they can be added back to the ontology and the vector store, giving a concrete route to grow the sparse German rhetorical-figure data."],"supporting_citations":[{"why":"This work supplies the GRhOOT ontology that the paper reifies and later uses as the retrieval source.","marker":"Kühn et al., 2022"},{"why":"This work introduces retrieval-augmented generation, the core mechanism for grounding the language model in the ontology.","marker":"Lewis et al., 2020"},{"why":"This work provides the automated evaluation suite used to score the RAG pipeline on faithfulness, correctness, and related metrics.","marker":"Es et al., 2023"},{"why":"This work introduces competency questions, which form the ground-truth basis for the RAG evaluation.","marker":"Grüninger and Fox, 1995"},{"why":"This work demonstrates that language models can generate competency questions for existing ontologies, which the paper uses to expand the small set of ontology questions.","marker":"Alharbi et al., 2023"},{"why":"This work supplies evidence that content in the middle of long contexts is harder for language models to recall, motivating the chunk-size comparison.","marker":"Liu et al., 2024"},{"why":"This work identifies data scarcity and annotation variability as central challenges that the application is designed to address.","marker":"Kühn and Mitrović, 2024a"}],"fun_headline_variants":["Ontology + RAG app spots German rhetorical figures","Property dropdowns and RAG chat tag German figures","RAG-grounded ontology guides German figure annotation","Tool pairs ontology and RAG for German rhetoric","German rhetorical figure annotation via ontology + RAG"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The entire annotation flow assumes that a user without linguistic training can reliably map a text to the three property dimensions of operation, affected element, and operational form; if users cannot do that, the app returns the wrong candidates and the collected data lose quality.","fun_headline_variants_meta":{"raw":{"variants":["Ontology + RAG app spots German rhetorical figures","Property dropdowns and RAG chat tag German figures","RAG-grounded ontology guides German figure annotation","Tool pairs ontology and RAG for German rhetoric","German rhetorical figure annotation via ontology + RAG"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000288,"raw_usage":{"total_tokens":1704,"prompt_tokens":977,"completion_tokens":727,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":593,"completion_tokens_details":{"reasoning_tokens":654}},"tokens_in":593,"tokens_out":727,"duration_ms":7588,"temperature":1.0,"reasoning_tokens":654,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T12:46:53.596551+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Give the application to a group of untrained, non-expert German speakers with a set of sentences that contain figures based on transferred meaning, such as metaphors, and measure how often their property selections lead to the correct figure name. A success rate near chance for those figures would show that the property-based flow does not support untrained annotation for the very class of figures the paper admits are hardest.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"This work introduces competency questions, which form the ground-truth basis for the RAG evaluation."},{"cited_title":"An Experiment in Retrofitting Competency Questions for Existing Ontologies","cited_arxiv_id":"2311.05662","evidence_quote":"This work demonstrates that language models can generate competency questions for existing ontologies, which the paper uses to expand the small set of ontology questions."}],"review_version":1}