{"id":"26d28d8a-29cb-4af7-9967-61fdd22b8b85","arxiv_id":"2412.07213","paper_version":1,"verdict":"REJECT","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"high","formal_verification":"none","parameter_count":4,"one_line_summary":"IntellectSeeker combines a fine-tuned GPT-3.5-turbo term translator and a probabilistic relevance filter for personalized academic search, reporting BLEU 0.93 and ROUGE-1 0.94 on a self-generated corpus.","lead":"IntellectSeeker is a literature-search platform that uses a fine-tuned GPT-3.5-turbo to turn everyday questions into academic search terms and a probabilistic filter to personalize each user's paper database. The paper describes the system and reports high scores on its own synthetic translation benchmark, but provides no end-to-end evaluation against existing tools.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The fine-tuned LLM's near-perfect BLEU/ROUGE scores are not credible evidence of search precision because the validation set is ChatGPT-generated from the same prompts that created the training set; the undefined similarity function S in Eq. (1) compounds this reproducibility gap.","rationale":"The reader's weakest_assumption focuses on the undefined similarity function S in Eq. (1). That is a genuine and important reproducibility flaw: without knowing how keyword features are converted to codes and how S is computed, the probabilistic data-scraping model cannot be independently implemented or evaluated. I agree this concern is valid. However, I find a more load-bearing problem in the LLM evaluation. Section 4.1 states that ChatGPT generated the question-answer pairs from manually written prompts, and the training/validation split was drawn from that single synthetic corpus. Thus, the near-perfect BLEU, ROUGE, and METEOR scores in Table 1 do not establish that the fine-tuned model converts everyday language to academic terms accurately for real users. They only establish that the model reproduces the formatting conventions of its own synthetic training distribution. Because the abstract's 'exceptional search precision and matching capabilities' is the paper's headline contribution, this circular evaluation directly weakens the central claim. The undefined S is a separate, independent flaw that makes the probabilistic filtering component untestable; both issues support the REJECT verdict. I would not change the reader's verdict, but I would reframe the primary objection as validation-set circularity rather than the undefined similarity function, with the undefined S as a secondary but still serious reproducibility gap. A concrete test with an independently authored held-out set would settle whether the reported scores reflect real generalization or only synthetic-data overfitting.","tokens_in":9083,"tokens_out":2541,"duration_ms":28486,"concrete_test":"Retrieve the linked GitHub repository and inspect the corpus construction script and the exact 10% validation split. Then construct an independent held-out test set of 200 to 500 human-written everyday-language queries with verified academic-term targets, sourced from academic writing guides or domain experts, and ensure that no prompt template from Section 4.1 is used. Re-run the fine-tuned GPT-3.5-turbo model and the Table 1 baselines on this held-out set. If the fine-tuned model's BLEU/ROUGE/METEOR scores drop substantially (for example, BLEU below 0.6) or its ranking changes, the reported scores are an artifact of validation-set leakage and the central search-precision claim has no quantitative support.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central claim of 'exceptional search precision and matching capabilities' rests on Table 1, where a fine-tuned GPT-3.5-turbo achieves BLEU 0.9269, ROUGE-1 0.9413, and METEOR 0.9553. Section 4.1 describes the corpus construction: manually written prompts were used to guide ChatGPT to generate question-answer pairs 'in the same format defined manually,' and the resulting dataset was split 9:1 into training and validation sets. The validation set is therefore not independent human ground truth; it is synthetic text produced by the same model family and the same prompt format used to generate the training data. High overlap metrics under this protocol largely measure format consistency and template memorization, not real-world ability to map everyday language to academic terminology. This is the most load-bearing concern because it invalidates the only quantitative evidence supporting the LLM component's contribution to search precision. The probabilistic filter adds a second, independent reproducibility failure: Eq. (1) defines the importance score I via a similarity function S, but S is never specified beyond 'all features K will be converted into codes, then the similarity will be calculated.' No algorithm, distance measure, or code reference is given for S, so the personalized filtering mechanism cannot be reimplemented or tested. A system-level comparison against Google Scholar or other baselines is also absent, but the circular evaluation alone is enough to undermine the claimed quantitative superiority.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper presents IntellectSeeker, a literature management platform that combines a probabilistic data-scraping filter with a fine-tuned GPT-3.5-turbo model. The probabilistic filter computes an importance score I = wp*S(Ka,Ku) + wi*S(Ka,Ki) to decide whether an article enters the user's personalized database, with a fixed threshold of 0.75 and a random inclusion probability P=0.05. The language-model component is fine-tuned to convert everyday language into academic terminology, and Table 1 reports near-perfect BLEU/ROUGE/METEOR scores for the fine-tuned GPT-3.5-turbo. The paper also describes recommendation, summarization, and word-cloud features, and it claims 'exceptional search precision and matching capabilities.' The core quantitative evidence for this claim is the LLM evaluation in Section 4.1, together with the equation-based probabilistic filter in Section 3.2.","tokens_in":9388,"tokens_out":3597,"duration_ms":37774,"significance":"If rigorously validated, the idea of an LLM-based query-to-academic-terminology conversion and a behavior-driven probabilistic filter would be useful for novice researchers. The paper also ships a system description and a GitHub link, which is a positive step for reproducibility. However, the current manuscript does not provide credible evidence for its central claims: the similarity function S in Eq. (1) is never defined, the filtering thresholds are asserted without calibration, and the LLM evaluation is circular because both training and validation data are generated by ChatGPT using the same manually defined output format. As a result, the reported high metrics mostly demonstrate format and template reproduction rather than improved search precision, and the paper's core personalization mechanism is not reproducible or testable.","major_comments":[{"comment":"The similarity function S is never specified. The text states only that 'all features K will be converted into codes, then the similarity will be calculated,' but no encoding scheme, distance measure, or implementation detail is given for S, nor is the feature representation of Ka, Ku, or Ki defined. Since Eq. (1) is the entire formal basis for the personalized filtering claim, this omission makes the model untestable and non-reproducible. A reader cannot compute I, cannot verify the threshold behavior, and cannot reimplement the system without guessing the semantics of S.","section":"Section 3.2, Eq. (1)"},{"comment":"The validation set is not independent human ground truth. The authors write that they 'manually wrote several prompts' and then used those prompts 'to guide ChatGPT in generating more question-and-answer pairs in the same format defined manually.' The resulting dataset is split 9:1 into training and validation sets, so both sets come from the same ChatGPT generation process and the same fixed output format. The BLEU, ROUGE, and METEOR scores in Table 1 therefore largely measure whether the fine-tuned model reproduces the prompt template and formatting conventions, not whether it improves real-world search precision. The abstract's claim of 'exceptional search precision and matching capabilities' needs support from an independent human-annotated evaluation or a retrieval experiment with real queries and relevance judgments.","section":"Section 4.1, Table 1"},{"comment":"The inclusion threshold of 0.75 and the random inclusion probability of P=0.05 are stated without any calibration, sensitivity analysis, or justification. Because the distribution of the importance score I is not characterized and S is undefined, these numerical choices cannot be interpreted or evaluated. The paper provides no evidence about how varying these parameters affects precision, recall, or database personalization, despite these parameters being load-bearing for the filtering mechanism.","section":"Section 3.2"},{"comment":"No end-to-end evaluation of the complete IntellectSeeker system is provided. The only reported quantitative results are component-level LLM text-conversion metrics on the synthetic validation set. There are no retrieval precision/recall measurements, no user study, no comparison against Google Scholar or other literature management tools, and no evaluation of the recommendation or probabilistic-filtering components. Consequently, the central claim that IntellectSeeker provides a 'highly customizable literature management solution with exceptional search precision' is unsupported at the system level.","section":"Sections 3.3, 3.4, and 4"}],"minor_comments":[{"comment":"The keyword 'few-short learning' should be 'few-shot learning.'","section":"Keywords"},{"comment":"The heading contains a stray space: 'T rain Semantic Enhancement Model' should be 'Train Semantic Enhancement Model.'","section":"Section 4.1 heading"},{"comment":"The model name is misspelled as 'LaMDAMDA'; it should be 'LaMDA.'","section":"Reference [19]"},{"comment":"The sentence 'For the common unrelated questions, we allowed ChatGPT to generate them randomly' is unclear; it should be reworded, for example as 'For unrelated common questions, we allowed ChatGPT to generate responses randomly.'","section":"Section 4.1"},{"comment":"The text mentions 'Gemma and Mixtral with the highest MT-bench score,' but Table 1 lists Gemma2-2B and Mistral-7B rather than Mixtral; please reconcile the model names.","section":"Section 4.1 and Table 1"},{"comment":"The corpus size is given only as '0.266k rows'; please report the exact numbers of training and validation examples so readers can assess the scale of the fine-tuning experiment.","section":"Section 4.1"}],"recommendation":"reject","confidential_remarks":"The manuscript appears to be a system description with a promising high-level idea, but the two load-bearing pillars—the probabilistic filtering model and the LLM evaluation—are not sufficiently specified or validly evaluated. The circular ChatGPT-generated validation set and the undefined similarity function are not local presentation issues; they directly undermine the abstract's central claim. I would not rule out a future submission if the authors provide a fully specified similarity model, a human-annotated or otherwise independent evaluation, and an end-to-end comparison with existing tools, but in its current form the paper does not meet the bar for publication."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague — here's my take on IntellectSeeker. The one useful thing: it's a concrete, integrated system with public code, and the idea of fine-tuning GPT-3.5-turbo to render everyday language into academic terms is a legitimate and reasonably novel application. The authors compare several LLMs under zero/few-shot and fine-tuning, and the finding that fine-tuning beats prompting for this task is plausible. That part deserves credit.\n\nThe soft spots are exactly where the reader puts them, and they're load-bearing. Eq. (1) defines the importance score with an unspecified similarity function S—'features K will be converted into codes, then the similarity will be calculated' is not an algorithm. The filter cannot be reimplemented or tested. The evaluation of the LLM is circular: both training and validation are ChatGPT outputs in the same manually defined format. Near-perfect BLEU/ROUGE/METEOR under that protocol largely measures format consistency, not real translation quality. No human judgment, no real user queries, no comparison against Google Scholar or any baseline search. The threshold 0.75 and P=0.05 appear without sensitivity analysis. Table 1 also has a minor oddity: Llama2-7B fine-tune BLEU=0.0 and the text says Mixtral while the table says Mistral-7B.\n\nThe conclusion 'exceptional search precision' is not supported by the evidence in the paper. That said, the paper is not nonsense—it's an honest student-project write-up with a working demo and a clear direction. The flaws are fixable with a proper human evaluation and a defined similarity function. I would not cite it as a result, but I'd send it to a serious referee: the system is real, the idea is timely, and a revision that fixes the evaluation could become a useful workshop or short paper. Reading group: maybe, as a case study in how LLM evaluation can go circular.","headline":"A sincere system demo with an interesting LLM application, but the evaluation is circular and the probabilistic filter is underspecified, so the precision claims don't hold.","tokens_in":9942,"tokens_out":2366,"would_cite":false,"duration_ms":22963,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A fine-tuned chatbot that turns everyday questions into academic search terms.","keywords":["literature management","large language model","fine-tuning","GPT-3.5-turbo","probabilistic model","personalized search","term matching","recommendation system"],"falsifier":"Run the full pipeline on a fixed set of 100 genuine literature queries from novices and measure retrieval precision against a manually curated gold standard. If the fine-tuned chatbot's term conversions do not improve hit rate over a plain keyword search, the central claim fails. Also, check the reported BLEU on an external dataset not generated with the same ChatGPT prompts.","tokens_in":8877,"feed_emoji":"🔍","tokens_out":1278,"duration_ms":14609,"temperature":0.7,"pith_summary":"The paper introduces IntellectSeeker, a literature management platform that combines a probabilistic data-scraping filter with a fine-tuned GPT-3.5-turbo chatbot. The central claim is that this combination closes the gap between casual language and academic terminology, improving search precision and personalization. The authors report that their fine-tuned model outperforms several open-source and prompt-based baselines on translation-style metrics (BLEU 0.9269, ROUGE-1 0.9413, ROUGE-L 0.9355, METEOR 0.9553). If the approach holds, it would give newcomers a more forgiving way to find relevant literature without knowing field-specific vocabulary.","feed_headline":"A fine-tuned chatbot turns plain questions into academic search terms","feed_subtitle":"IntellectSeeker pairs the chatbot with a probabilistic filter to personalize literature search.","key_machinery":"The probabilistic model is a weighted similarity formula $I = w_p \\cdot S(K_a, K_u) + w_i \\cdot S(K_a, K_i)$, where $K_a$ are the article's features, $K_u$ the user's preference features, $K_i$ the manually inputted requirements, $w_p$ and $w_i$ weights, and $S$ the similarity function measuring overlap between keyword sets. All features are converted into codes before similarity is computed. This formula decides whether an article is added to the user's database, and the weights and threshold (0.75) are user-adjustable. The language model component is a fine-tuned GPT-3.5-turbo that translates everyday expressions into academic terms using a manually constructed corpus with a 9:1 train-validation split.","core_discovery":"The paper claims that IntellectSeeker provides 'exceptional search precision and matching capabilities' by combining a probabilistic article-selection model with a fine-tuned large language model. The probabilistic model scores candidate articles by a weighted similarity between article features and both user preferences and explicit requirements, keeping articles above a 0.75 threshold (with a small random acceptance probability) to build a personalized database. The language model component, fine-tuned GPT-3.5-turbo on a corpus of 0.266k question-answer pairs, converts everyday language into academic terms, and the paper reports that this fine-tuned model 'outperforms others in all metrics' on BLEU, ROUGE-1, ROUGE-2, ROUGE-L, and METEOR in Table 1.","pith_inferences":["The paper's central comparison rests on a single validation set; the 0.92+ scores could reflect the shape of that set rather than generalizable performance. A sensible test would be to evaluate the fine-tuned model on an external academic-query dataset not generated by the same ChatGPT prompts.","The probabilistic filter's similarity function $S$ is not fully specified. If $S$ is merely keyword overlap on 'codes,' it may ignore semantics and synonymy, limiting the claimed personalization to exact-match scenarios.","The design implies a workflow where users see one-line summaries and word clouds; a controlled user study could determine whether these visualizations actually speed up relevance judgment.","The claimed 'exceptional search precision' is asserted rather than measured; a direct comparison with Google Scholar or Semantic Scholar on a fixed query set would settle this more convincingly."],"forward_implications":["If the fine-tuned model reliably converts casual queries into academic terms, literature search becomes easier for novices and non-native speakers.","The probabilistic filter could make each user's database gradually more personalized with use, since implicit preferences accumulate from clicks and bookmarks.","The reporting of translation-style metrics (BLEU, ROUGE, METEOR) suggests the authors regard term conversion as a machine-translation task, which implies that better conversion quality should translate directly into better search recall.","If the threshold-and-random-acceptance design works, it would create a balance between strictly relevant results and serendipitous discoveries."],"supporting_citations":[{"why":"Supplies the academic word definitions used to construct the fine-tuning corpus.","marker":"[46]"},{"why":"The OpenAI GPT-4 technical report is used as background for GPT models, though the paper fine-tunes GPT-3.5-turbo.","marker":"[22]"},{"why":"Describes instruction-following training for GPT-3.5-turbo, the base model being fine-tuned.","marker":"[40]"},{"why":"Provides the Llama2 open-source model used as a baseline in the LLM comparison.","marker":"[21]"},{"why":"Surveys research paper recommender systems, framing the problem IntellectSeeker addresses.","marker":"[1]"},{"why":"Provides the probabilistic model concept used in the data-scraping stage.","marker":"[2]"}],"fun_headline_variants":["Fine-tuned chatbot plus probabilistic filter personalizes paper search","LLM and probability model make literature search personal","From everyday language to academic terms via AI and statistics","IntellectSeeker: combining LLM and probabilistic filter for personalized search","Personalized paper search powered by fine-tuned GPT-3.5"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The personalized-filtering claim depends on an unspecified similarity function $S$ that converts article and user features into codes and measures their overlap.","fun_headline_variants_meta":{"raw":{"variants":["Fine-tuned chatbot plus probabilistic filter personalizes paper search","LLM and probability model make literature search personal","From everyday language to academic terms via AI and statistics","IntellectSeeker: combining LLM and probabilistic filter for personalized search","Personalized paper search powered by fine-tuned GPT-3.5"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001224,"raw_usage":{"total_tokens":5020,"prompt_tokens":924,"completion_tokens":4096,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":540,"completion_tokens_details":{"reasoning_tokens":4022}},"tokens_in":540,"tokens_out":4096,"duration_ms":35247,"temperature":1.0,"reasoning_tokens":4022,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T18:55:33.618601+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the full pipeline on a fixed set of 100 genuine literature queries from novices and measure retrieval precision against a manually curated gold standard. If the fine-tuned chatbot's term conversions do not improve hit rate over a plain keyword search, the central claim fails. Also, check the reported BLEU on an external dataset not generated with the same ChatGPT prompts.","supporting_citations":[{"cited_title":"A new academic word list.TESOL quarterly, 34(2):213–238, 2000","cited_arxiv_id":null,"evidence_quote":"Supplies the academic word definitions used to construct the fine-tuning corpus."},{"cited_title":"Gpt-4 technical report.arXiv, pages 2303–08774, 2023","cited_arxiv_id":null,"evidence_quote":"The OpenAI GPT-4 technical report is used as background for GPT models, though the paper fine-tunes GPT-3.5-turbo."},{"cited_title":"Train- ing language models to follow instructions with human feedback","cited_arxiv_id":null,"evidence_quote":"Describes instruction-following training for GPT-3.5-turbo, the base model being fine-tuned."},{"cited_title":"An anatomization of research paper recommender system: Overview, approaches and challenges","cited_arxiv_id":null,"evidence_quote":"Surveys research paper recommender systems, framing the problem IntellectSeeker addresses."},{"cited_title":"Approximate data collection in sensor networks using probabilistic models","cited_arxiv_id":null,"evidence_quote":"Provides the probabilistic model concept used in the data-scraping stage."}],"review_version":1}