{"id":"8d03a460-e964-48db-89cd-b729d62d8c95","arxiv_id":"2504.21475","paper_version":1,"verdict":"REJECT","confidence":"MODERATE","novelty_score":3.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"An Arabic reverse dictionary using a semi-encoder network reports a best ranking score of 0.0644 with ARBERTv2, alongside qualitative dataset standards.","lead":"This paper builds an Arabic reverse dictionary, a tool that finds a word from a description of its meaning, by training a neural network to map definitions to word embeddings. It reports the best retrieval ranking with the ARBERTv2 language model and proposes eight rules for writing better dictionary definitions.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Rank Eqs. (5)-(7) do not define a retrieval rank; without the official KSAA-CAD protocol the headline 0.0644 is uninterpretable.","rationale":"The paper's stated purpose is to establish a transformer-based Arabic reverse dictionary with a state-of-the-art rank. For that claim to hold, the rank metric must be well-defined and correctly computed. Section 3.2.1, the only formal account of the metric, is internally incoherent: Eqs. (5)-(6) sum indicators over the same set M_w and mix distinct evaluation metrics, so they do not define a retrieval rank. No candidate vocabulary V or official protocol is specified, making Table 3 non-reproducible from the paper alone. The reader's weakest_assumption identifies exactly this point, and I agree. Secondary issues (undisclosed 84k-sample augmentation, the identical 0.0644 value in the authors' prior paper, absence of error bars) reinforce the concern but are not needed to establish it. The dataset-quality standards and the released RDTL library are useful contributions, but they do not validate the quantitative claim. A single targeted re-implementation with the official evaluation script would settle the matter; until then the central claim is unverified. Since the reader already recommended REJECT and this analysis supports that recommendation, the verdict is unchanged.","tokens_in":14076,"tokens_out":8218,"duration_ms":82653,"concrete_test":"Run the released RDTL code on the official KSAA-CAD 2024 test split with ARBERTv2 and the semi-encoder, and compute rank with the official shared-task evaluation script (Alshammari et al., 2024). Check whether the code's ranking path implements Eqs. (5)-(7) or the official protocol, and whether the resulting rank equals 0.0644. If the official evaluation gives a different number, or the code implements Eqs. (5)-(7), the headline result is unsupported.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is the rank score 0.0644 in Table 3. Section 3.2.1 is the paper's only formal specification of rank, and it is incoherent: M_w is defined as the set of scores of a single word under k different metrics (MSE, cosine similarity, rank), and Eqs. (5)-(6) sum indicator functions over that same set. A retrieval rank must instead compare each candidate's score with scores of all other candidates in a specified candidate vocabulary V, e.g. R_w = sum_{v in V} 1(S_v >= S_w). As written, Eqs. (5)-(7) cannot produce a retrieval rank at all, and V is never defined. If the implementation follows these equations, the headline number is meaningless; if it follows the official KSAA-CAD evaluation, the paper's formal definition is wrong and the result cannot be verified from the text. The abstract's 'state-of-the-art' claim therefore rests on an unverifiable metric. This is compounded by the fact that the identical value 0.0644 appears in the authors' prior KSAA-CAD paper (Sibaee et al., 2024), so an independent computation is essential.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper presents an Arabic reverse dictionary (RD) system built around a proposed \"semi-encoder\" feed-forward network with geometrically decreasing hidden layers, trained to map definition embeddings to target word embeddings. The authors evaluate several pre-trained transformer encoders, report ARBERTv2 as the best model with a ranking score of 0.0644, and claim state-of-the-art performance. The paper also proposes eight standards for Arabic lexicographic definition quality, presents a qualitative dataset analysis, and releases a Python library called RDTL.","tokens_in":14298,"tokens_out":8495,"duration_ms":87440,"significance":"If the reported result were substantiated, the paper would offer a useful benchmark for Arabic reverse dictionary research and a practical open-source library. The breadth of compared encoders and the qualitative dataset-quality analysis are valuable, and releasing RDTL is a constructive step toward reproducibility. However, the central quantitative claim is not verifiable as written: the ranking metric is defined incoherently, the source of the 84,000 augmented samples is unspecified, no error bars or significance tests are provided, and the headline number 0.0644 coincides exactly with a value the authors reported in their own prior KSAA-CAD publication. The claimed state-of-the-art result therefore cannot be accepted on the evidence in this manuscript.","major_comments":[{"comment":"The definition of the ranking metric is incoherent. M_w is defined as the set of metric scores for a single word w, and Eqs. (5)-(6) sum indicator functions over that same set; this does not compare a candidate's score with the scores of other candidates. A retrieval rank requires a comparison between S_w and the scores of all candidates in a defined vocabulary V, e.g., R_w = sum_{v in V} 1(S_v >= S_w). The vocabulary V is never specified beyond its appearance in Eq. (4), and the official KSAA-CAD evaluation protocol is not cited. Consequently, the rank column of Table 3 and the headline 0.0644 cannot be interpreted or reproduced from the text.","section":"Section 3.2.1, Eqs. (5)-(7)"},{"comment":"The approximately 84,000 additional training samples are said to come from an \"external source\" with no citation, description, or construction details. This is load-bearing because the augmented set is larger than the original 31,372 training samples and could dominate training; without knowing its provenance, domain, and overlap with the test set, the experimental results are not reproducible.","section":"Section 3.1"},{"comment":"All results are reported as single runs with no standard deviations, confidence intervals, or significance tests. The text repeatedly uses terms such as \"significant performance gap\" and \"significantly impact\" without statistical support. Given the small rank differences among several models (e.g., ARBERTv2 at 0.0644 versus OpenAI large at 0.077), the claim that ARBERTv2 is the best model is not established.","section":"Section 5.1 and Table 3"},{"comment":"The reported best rank of 0.0644 is identical to the rank the authors report for their previous system in Sibaee et al. (2024), described in Section 2.3 as achieving \"the best rank and an MSE score of 0.0644 and 0.059.\" The manuscript provides no explanation for this coincidence and no independent evidence that the current ARBERTv2 configuration was evaluated with a correct ranking protocol. Because the abstract's state-of-the-art claim depends directly on this number, the relationship between the current result and the prior publication must be clarified with an independen computation or an explicit statement of what is new.","section":"Section 2.3 and abstract"},{"comment":"The manuscript does not specify how each transformer encoder is applied to definitions, including pooling strategy, maximum sequence length, and whether the encoder is fine-tuned or frozen. The claim that the architecture is consistent across all experiments therefore cannot be checked, and the released library is not documented with the evaluation script used to compute the rank column. These details are necessary for reproducibility of the central experiments.","section":"Section 3.2 and Figure 1"}],"minor_comments":[{"comment":"There are typographical errors in \"this architicture is chosen after empirical expirements from Sibaee et al. (2023)\"; the sentence should also clarify why the authors' own prior experience is the only basis for the architecture choice.","section":"Section 3.2"},{"comment":"The annotation procedure for the 200+ randomly selected samples is not described: the number of annotators, inter-annotator agreement, and sample selection method are missing. Figure 2's axes are labeled \"Class\" and \"Count of accuracy,\" which does not make clear whether the scores are 1-5 quality ratings or accuracy classes.","section":"Section 5.2 and Figure 2"},{"comment":"The model \"baselineM_elct\" is never defined in the text; the baseline should be clearly described in the methodology so readers know what it represents.","section":"Section 4.1 and Table 3"},{"comment":"The notation R_i,w is used before it is defined, and the phrase \"aggregated across users\" suggests a user study that is not presented anywhere in the paper.","section":"Section 3.2.1, Eq. (7)"},{"comment":"The sentence describing Sibaee et al. (2024) is grammatically ambiguous: \"the best rank and an MSE score of 0.0644 and 0.059\" should be rephrased as \"a rank of 0.0644 and an MSE of 0.059.\"","section":"Section 2.3"},{"comment":"The paper is described as proposing a \"Transformer-Based Approach,\" but the actual novelty is a fully connected semi-encoder network on top of existing transformer embeddings; the contribution boundary should be stated more precisely in the title and abstract.","section":"Title and abstract"}],"recommendation":"reject","confidential_remarks":"The exact match between the reported rank (0.0644) and the value in the authors' prior KSAA-CAD paper, together with the incoherent formal definition of the rank metric, makes the central empirical claim unverifiable in its current form. I recommend that the editor ask the authors for the official evaluation script and detailed provenance of the augmented dataset if a revised version is ever considered; without these, the headline result cannot be independently checked."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Two things up front: the headline rank (0.0644) is the same number the authors reported in their own KSAA-CAD 2024 paper, and Section 3.2.1's equations do not define a retrieval rank as written. That makes the 'state-of-the-art' claim hard to verify from this text.\n\nWhat is genuinely new is the broader encoder comparison (OpenAI, GATE, ATMv2, Nomic, and several SBERT variants) and the qualitative analysis of definition quality in the KSAA-CAD data. The eight dataset-writing standards are sensible and could be useful to anyone building Arabic RD resources. Releasing RDTL is also a practical plus, though the paper does not evaluate the library.\n\nThe problems are not minor. Equations (5)-(7) define M_w as a set of metric scores for a single word and then sum indicators over that same set. A rank must compare each candidate's score with scores over a candidate vocabulary V, and V is never defined. If the code follows the official KSAA-CAD protocol, the paper should say so; if it follows the equations, the 0.0644 value is meaningless. Either way the result is unverifiable from the text. The 84k augmented samples come from an unnamed external source. There are no error bars or significance tests, and the best score matches a prior publication without explicit discussion, so the new empirical content is unclear. Minor: the GATE model is cited to a HuggingFace repo, not an archival paper.\n\nWho is this for? Researchers working on Arabic reverse dictionaries, who may find the quality standards and the encoder comparison useful as a starting point. I would not cite the numerical results. The paper deserves a serious referee only if the authors are pushed to fix the metric definition, disclose the augmentation source, and reframe this as an extension of their prior work. Absent that, I would reject it in review. I would not desk-reject it outright, because the dataset-guidelines portion has real value.","headline":"Headline rank repeats their own 2024 result, the rank equations are incoherent as written, and the dataset-quality guidelines are the only clearly new value.","tokens_in":14867,"tokens_out":6183,"would_cite":false,"duration_ms":61205,"reading_group":"maybe","serious_thinker":"no","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper claims ARBERTv2, paired with a shrinking-layer semi-encoder network, attains the best Arabic reverse-dictionary rank (0.0644).","keywords":["Arabic reverse dictionary","transformer models","ARBERTv2","semantic search","semi-encoder network","Arabic lexicographic definitions","dataset construction standards","embedding-based retrieval"],"falsifier":"Run the released RDTL pipeline on the official test split and evaluate with the official rank metric; if the average rank of the true word differs materially from 0.0644, or if applying Equations (5)-(6) literally assigns the same rank to every candidate, the headline result does not hold.","tokens_in":13877,"feed_emoji":"📚","tokens_out":9250,"duration_ms":87151,"temperature":0.7,"pith_summary":"This paper tries to establish that a simple transformer-plus-regression pipeline can retrieve Arabic words from plain definitions, and that the choice of Arabic-specific definition encoder matters more than architectural complexity. The authors claim that ARBERTv2, paired with a semi-encoder network whose hidden layers shrink geometrically, achieves the best ranking score of 0.0644 among the tested models on the Arabic reverse-dictionary task. They also argue that the training data itself is the main bottleneck and propose eight standards for writing definitions so future datasets support better retrieval. If the claim holds, Arabic speakers gain a practical tool for tip-of-the-tongue searches and dataset builders get a concrete checklist.","feed_headline":"ARBERTv2 hits best Arabic reverse-dictionary rank: 0.0644","feed_subtitle":"Arabic-specific text models outperform multilingual ones at retrieving a word from its definition; best rank is 0.0644","key_machinery":"The load-bearing object is the learned map $m: \\mathbb{R}^d \\to \\mathbb{R}^b$ from definition-embedding space to word-embedding space, realized as a semi-encoder: a four-layer feedforward network with hidden widths $8s, 4s, 2s, s$, where $s=d=256$, GELU activations, dropout between 0.2 and 0.4, and the AdamW optimizer at learning rate $10^{-4}$. The network is trained with mean squared error loss on pairs $(f(\\text{def}_i), e(\\text{word}_i))$, and a definition is judged correct when its mapped vector has top cosine similarity to the true word embedding among all candidates. This geometric alignment is what carries the retrieval; no external lexicon or morphological rules are used at inference time.","core_discovery":"The paper's central claim is that an Arabic-specific pretrained encoder is the decisive ingredient in Arabic reverse-dictionary retrieval, and that ARBERTv2 preserves the relative semantic distances needed for ranking better than any tested alternative. With a semi-encoder network whose hidden layers decrease geometrically from $8s$ to $s$ ($s=256$), trained against 256-dimensional AraELECTRA word embeddings using mean squared error loss, ARBERTv2 achieves the best ranking score of $0.0644$ on the shared-task test set, ahead of the next-best OpenAI large embedding at $0.077$ and above the baseline score of $0.84$. The paper also claims that inspecting more than 200 randomly selected definitions exposes recurring quality problems, and it condenses those findings into eight standards for constructing Arabic reverse-dictionary datasets.","pith_inferences":["A direct extension the paper does not run is to rebuild a small gold-standard Arabic definition set under its eight standards and retrain the same architecture on it; fixed-capacity improvement over the raw dataset would isolate how much of the ranking gains come from data quality rather than encoder choice.","Because the printed rank equations are internally inconsistent, an editor's check would be to re-implement the official evaluator and recompute Table 3; that would confirm whether the 0.0644 figure and the baseline comparisons are directly comparable.","The superiority of Arabic-specific encoders is demonstrated on Arabic definitions and Arabic word embeddings; whether it persists under cross-lingual or dialectal queries is not tested."],"forward_implications":["With ARBERTv2 as the definition encoder, a new Arabic reverse-dictionary deployment can start from the released RDTL pipeline and expect the best tested rank of 0.0644 rather than searching across multilingual embeddings.","If the eight definition-writing standards are adopted, future dataset builders can avoid morphology-only entries, ambiguous pronouns, and field-specific definitions that the analysis found in more than 30 percent of sampled cases.","The formal abstraction of reverse dictionary as a learned map between definition and word embedding spaces gives future system builders a single training and evaluation target.","The modular library lets researchers swap encoders and retrain, so the same architecture can be tested on new Arabic datasets without re-implementing the pipeline."],"supporting_citations":[{"why":"Defines the Arabic reverse dictionary shared task and its official ranking evaluation, the protocol the headline score relies on.","marker":"Al-Matham et al. (2023)"},{"why":"Provides the KSAA-CAD dataset and the baselineM_elct baseline whose rank (0.84) the system is compared against.","marker":"Alshammari et al. (2024)"},{"why":"Supplies ARBERTv2 and MARBERTv2, the Arabic-specific encoders whose relative performance is the paper's main result.","marker":"Abdul-Mageed et al. (2021)"},{"why":"Supplies AraELECTRA, the model that produces the 256-dimensional word embeddings used as regression targets.","marker":"Antoun et al. (2021)"},{"why":"Introduces the semi-encoder structure for reverse dictionary tasks that this paper adopts and extends.","marker":"Sibaee et al. (2024)"},{"why":"Documents the earlier two-stage SBERT-plus-decoder architecture from which this paper draws its optimizer and architecture lessons.","marker":"Sibaee et al. (2023)"},{"why":"Provides the multilingual sentence-transformer baselines (mpnet, L12, LaBSE, CasedV1) compared in the experiments.","marker":"Reimers and Gurevych (2020)"},{"why":"Contributes the GATE Arabic embedding model evaluated as another language-specific baseline.","marker":"Nacar et al. (2025)"}],"fun_headline_variants":["Arabic reverse dictionary: ARBERTv2 hits best rank 0.0644","ARBERTv2 sets new best for Arabic reverse dictionary","Transformer model improves Arabic reverse dictionary ranking","Arabic reverse dictionary gets dataset standards and ARBERTv2 top score"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the reported ranking score was computed with the official shared-task rank procedure, not literally with the paper's own rank equations, since those equations compare each score to itself and would not discriminate between candidates.","fun_headline_variants_meta":{"raw":{"variants":["Arabic reverse dictionary: ARBERTv2 hits best rank 0.0644","ARBERTv2 sets new best for Arabic reverse dictionary","Transformer model improves Arabic reverse dictionary ranking","Arabic reverse dictionary gets dataset standards and ARBERTv2 top score"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000515,"raw_usage":{"total_tokens":2476,"prompt_tokens":898,"completion_tokens":1578,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":514,"completion_tokens_details":{"reasoning_tokens":1508}},"tokens_in":514,"tokens_out":1578,"duration_ms":11937,"temperature":1.0,"reasoning_tokens":1508,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-16T05:02:14.847243+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the released RDTL pipeline on the official test split and evaluate with the official rank metric; if the average rank of the true word differs materially from 0.0644, or if applying Equations (5)-(6) literally assigns the same rank to every candidate, the headline result does not hold.","supporting_citations":[],"review_version":1}