{"id":"3f39dfc7-9454-4749-9b49-16e49df4edc8","arxiv_id":"2411.12649","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"PseudoSeer uses Elasticsearch and weighted BM25 to let researchers search titles, abstracts, authors, references, and LaTeX pseudocode in arXiv papers.","lead":"This paper presents PseudoSeer, a search engine for finding pseudocode inside academic papers, built with Elasticsearch and a facet-based search interface. It is a system description with screenshots, but it offers no quantitative evaluation of search quality.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Ranking and indexing choices are never evaluated against any baseline or relevance measure; until a known-item or query-quality test is run, the claim that PseudoSeer 'facilitates efficient retrieval' of pseudocode papers is unsupported, not contradicted.","rationale":"I read the paper as a systems/artifact description rather than as a claim of state-of-the-art retrieval quality. The central claim is modest: if correct, the engine can retrieve arXiv papers using queries over titles, abstracts, authors, references, and LaTeX pseudocode fields. The mechanism is plausible because it is built from standard Elasticsearch components, and the paper openly discloses its limitations, including the lack of quantitative evaluation and the restriction to \\begin{algorithm} blocks. What would have to be true for the central claim to hold is that the returned results are relevant enough for actual use. That condition is weakest not because the components are flawed but because there is no evidence: no baseline, no known-item test, no user study, and no analysis of ranking quality. The reader's weakest assumption about hand-chosen weights is a concrete instance of this broader gap, and I agree with that identification. The concern does not move the verdict because the paper's own framing as a system description plus clearly stated future evaluation makes a conditional decision appropriate rather than outright rejection. The proposed test would give the authors a cheap way to either substantiate the claim or pinpoint the most needed improvement.","tokens_in":5459,"tokens_out":5746,"duration_ms":59234,"concrete_test":"Build a known-item retrieval test on a random sample of 100 papers from the indexed corpus. For each paper, issue one natural-language query derived from the abstract, one query from the reference text around the pseudocode, and one query using raw LaTeX tokens from the pseudocode. Record whether the source paper appears in the top 10 and compute MRR for the current configuration, for a configuration with LaTeX commands removed (plain-text pseudocode), and for equal field weights. If the source paper is not in top 10 for a sizable fraction of natural-language queries, or if the current configuration does not beat both baselines on MRR, the ranking and indexing choices are not supporting the central usability claim.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is that PseudoSeer is a pseudocode search engine that facilitates efficient retrieval over academic papers. For this to hold, a user's query must retrieve relevant pseudocode-bearing papers. That premise is not tested anywhere in the paper. Section 7 says evaluation is 'conducted by manually inspecting the results' of a handful of demo queries, and Section 8 concedes 'Enhanced evaluation will be a focus of future.' The two decisions that shape relevance—Section 5.1's choice to index raw LaTeX commands instead of converting to plain text, and Section 6's hand-set field weights (1 for LaTeX and authors, 2 for title and abstract)—are justified by assertion only. No comparison answers whether raw LaTeX tokens (e.g., \\For, \\If) are actually useful to users, who may phrase queries in natural language, nor whether the chosen weights improve ranked lists over equal weights or over a stripped-text baseline. Because ranking is the user-facing output of a search engine, an untested ranking function makes the central 'capable of efficient retrieval' claim unverified. This is a correctness-risk gap, not an internal contradiction.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper describes PseudoSeer, an Elasticsearch-based search engine that indexes arXiv papers containing LaTeX-style pseudocode and allows facet-based retrieval over titles, abstracts, authors, references, and pseudocode snippets. The authors describe data collection from the arXiv-pseudocode dataset of about 320,000 snippets from 2.2 million papers, a tokenization and indexing strategy that retains raw LaTeX commands for code fields, a weighted BM25 ranking scheme with hand-set field weights, and a web interface. The evaluation consists of manually inspecting a few example queries, and the paper states that enhanced evaluation is future work.","tokens_in":5708,"tokens_out":2701,"duration_ms":28385,"significance":"If the system's ranking were validated, this would be a useful tool for searching algorithm and pseudocode content in the academic literature, complementing prior work on formula search and structured-document retrieval. The paper's concrete description, the availability of the underlying arXiv-pseudocode dataset, and the demonstration of a working interface are strengths. However, the central usability claim—that PseudoSeer 'facilitates efficient retrieval'—is not supported by any quantitative or user-based evaluation, and the ranking weights and indexing choices are justified only by assertion. The significance is therefore conditional on future validation.","major_comments":[{"comment":"The paper's only evaluation is manual inspection of a few demo queries, and Section 7 explicitly states that 'Enhanced evaluation will be a focus of future.' Because the abstract and introduction claim that PseudoSeer 'facilitates efficient retrieval' of papers containing pseudocode, the absence of a relevance test is load-bearing. Please add a quantitative evaluation, such as known-item retrieval, a judged relevance set with precision/recall or nDCG, or at least a user study, and compare against a baseline such as an unweighted BM25 search over the same fields.","section":"Section 7 (Evaluation)"},{"comment":"The field weights (1 for LaTeX and authors, 2 for title and abstract) are chosen by assertion, described only as 'based on the potential relevance of the fields in real-world use cases.' No experiments, ablations, sensitivity analysis, or user data support these values. Since ranking is the user-facing output of a search engine, these weights are a load-bearing component of the 'efficient retrieval' claim. Please provide at least a simple ablation (equal weights vs. the chosen weights, or a scan over a small set of weight values) on a small set of queries with relevance judgments.","section":"Section 6 (Ranking)"},{"comment":"The decision to index raw LaTeX commands rather than converting pseudocode to plain text is justified by a plausible example (preserving \\For and \\If as structural cues), but no experiment compares the two indexing approaches. Users might phrase queries in natural language, and it is not self-evident that tokens such as \\For or \\EndIf improve ranked retrieval. Please evaluate both indexing strategies on a common query set, or otherwise justify the choice with evidence.","section":"Section 5.1 (Indexing Code)"},{"comment":"The paper notes that papers and pseudocode that could not be parsed and indexed without errors were omitted, and that some references could not be extracted. No statistics are given for how many items were dropped or how this omission may bias retrieval. This is not a fatal flaw, but it limits the scope of the claim that the engine searches the full arXiv pseudocode corpus; please quantify the omission rate and discuss the potential bias.","section":"Section 3 (Data Collection)"}],"minor_comments":[{"comment":"The text says 'radio buttons' but also says users 'can select multiple buttons simultaneously'; these are contradictory, so please clarify whether the fields are checkboxes or radio buttons with combined search enabled by a separate control.","section":"Section 4 (Search Interface)"},{"comment":"The sentence 'Dataset is carefully chosen' is missing an article, and there are several other grammatical issues throughout (e.g., 'broad range of search queries' should be 'a broad range'); a copy edit would improve readability.","section":"Section 3 (Data Collection)"},{"comment":"There is a typo in 'such as and Hu et al. (2005)', and the reference list contains 'Proceedings of the the 16th conference'; please correct these citation errors.","section":"Section 2 (Related Work)"},{"comment":"The text refers to 'the pseudoseer link' and 'the arxiv-pseudocode repository on GitHub' without giving URLs; please include the actual links in the final version.","section":"Section 4 (Search Interface)"},{"comment":"The paper does not state the values of the BM25 parameters k1 and b, even though they are part of the ranking function; please report the settings used in Elasticsearch.","section":"Section 6 (Ranking)"},{"comment":"Figures 1, 4, 5, and 6 are referenced in the text but not visible in the manuscript text supplied; please ensure the figures and captions are correctly embedded in the submission.","section":"Figures"}],"recommendation":"major_revision","confidential_remarks":"This is a system-description paper with a plausible and concrete implementation, but it currently lacks the evaluation needed to support its central usability claim. The requested additions are standard and feasible, so a major revision rather than a rejection seems appropriate. I would also note that the novelty relative to the broader search-engine literature is modest; the paper's contribution is mainly the domain-specific indexing and retrieval prototype."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Punchline: PseudoSeer is a concrete, new artifact — a search engine that indexes raw LaTeX pseudocode and lets users query across title, abstract, author, and code facets. Nothing else in the cited literature does exactly this, so it fills a real niche.\n\nCredit where due: the implementation is described clearly. Tokenization, Elasticsearch, BM25 with field weights, exact-match quoting, and screenshots give enough to reproduce the basics. The authors are also upfront that evaluation so far is manual inspection and that proper evaluation is future work. That honesty matters.\n\nSoft spots: the central claim that PseudoSeer “facilitates efficient retrieval” is unsupported. Section 7 reports no numbers, no baselines, no user study; Section 8 punts evaluation. The two choices that shape ranking — retaining raw LaTeX commands (Section 5.1) and hand-set weights of 1 for LaTeX/authors and 2 for title/abstract (Section 6) — are justified only by assertion. No test against a plain-text baseline or equal-weight baseline appears. So the effectiveness claim is a correctness-risk gap, not a contradiction. If the paper is meant as a system demonstration, the gap is minor; if it claims superiority over general search, it's central.\n\nOne more note: the paper leans on the authors' earlier pseudocode dataset (Toksoz et al., 2024). That is not circularity — the search engine is a new use of that dataset, and the dataset is cited openly. Related work covers math formula search but not pseudocode search, so the novelty claim holds.\n\nWho should read this: anyone building code-aware academic search tools, and researchers hunting for pseudocode in arXiv. It reads like a solid workshop paper, not a finished journal contribution.\n\nRecommendation: send it to peer review at a workshop or short-paper track, and ask the authors for a small known-item retrieval test and a baseline comparison. Without that, the effectiveness claim should be explicitly framed as preliminary.","headline":"A plausible, clearly-described pseudocode search engine with no quantitative evaluation — the artifact is new but the effectiveness claim remains unverified.","tokens_in":6168,"tokens_out":2548,"would_cite":false,"duration_ms":24779,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"PseudoSeer is a search engine that retrieves academic papers by indexing and ranking the pseudocode inside them, alongside title, abstract, author, and reference fields.","keywords":["pseudocode search","academic search engine","BM25 ranking","facet-based search","LaTeX pseudocode indexing","exact-match query","scholarly document retrieval"],"falsifier":"Take a set of, say, 50 papers with pseudocode and a list of queries with known relevant results; run the engine under three conditions—raw LaTeX tokens with the stated weights, stripped LaTeX with equal weights, and raw LaTeX with equal weights—and have raters score the top results. If the stated configuration does not rank known-relevant papers above the alternatives, the engine's ranking advantage is not supported.","tokens_in":5292,"feed_emoji":"🔍","tokens_out":8581,"duration_ms":81075,"temperature":0.7,"pith_summary":"This paper sets out to show that pseudocode inside academic papers can be made searchable: it presents PseudoSeer, a search engine that indexes the title, abstract, author, references, and LaTeX code sections of papers and retrieves them under BM25 ranking. Built on a corpus of roughly 320,000 pseudocode blocks extracted from scholarly papers, the system supports single-field, combined-field, and exact-match queries. The authors argue that keeping raw LaTeX commands like \\For and \\If in the index preserves structural cues that plain-text indexing would lose, and that giving title and abstract twice the weight of LaTeX and author fields makes combined searches more relevant. A careful reader would care because code-level retrieval is a missing piece in academic search: papers are often found by their prose, not by the algorithms they implement.","feed_headline":"PseudoSeer lets researchers search papers by their pseudocode","feed_subtitle":"The system ranks matches across title, abstract, authors, references, and LaTeX pseudocode.","key_machinery":"The load-bearing mechanism is a weighted BM25 ranking over an inverted-index search engine with five facets: title, abstract, authors, references, and LaTeX pseudocode. BM25 scores documents by query-term frequency with saturation, inverse document frequency, and length normalization; the engine multiplies field scores by hand-selected weights—2 for title, abstract, and references, 1 for LaTeX and authors—so a combined query ranks a paper by the sum of its per-field BM25 scores. The distinctive choice is tokenizing raw LaTeX as ordinary text, which keeps commands such as \\For and \\If in the index and preserves the structure of the pseudocode.","core_discovery":"The paper's central claim is that a pseudocode search engine can retrieve relevant academic documents by indexing and ranking the raw LaTeX of algorithm environments alongside standard bibliographic fields. The system stores pseudocode as tokenized text without stripping LaTeX commands, so structural markers such as \\For and \\If are searchable, and it indexes surrounding reference text for thematic queries. Retrieval uses BM25 with field weights: title, abstract, and references are weighted 2, while LaTeX pseudocode and authors are weighted 1. Combined facet searches sum these weighted scores, and quoted phrases trigger exact matching. The paper presents the design and shows results pages, treating manual inspection of those pages as the current evaluation.","pith_inferences":["The raw-LaTeX tokenization implies that queries phrased in natural language, such as \"a loop that checks a condition,\" will not match code tokens like \\If; a normalization or synonym layer would be needed, which the paper leaves for future work.","Because extraction depends on the \\begin{algorithm} environment, the engine's coverage is bounded by that convention; measuring recall against a hand-labeled sample of papers with pseudocode in other formats would quantify the gap.","The same facet-weighted BM25 recipe could transfer to other structured document content such as equations, tables, or figures, since the paper's real contribution is showing that structural markup can be indexed as text.","The field weights are chosen rather than learned or tested; a user study or offline relevance benchmark comparing the stated weights against equal weights would directly test their effect."],"forward_implications":["Users can locate papers by the algorithms they contain, for example by searching the LaTeX facet for a loop or conditional structure, not just by keywords in the title or abstract.","Combined facet searches let a researcher intersect an author name with a pseudocode construct, a capability ordinary academic search engines do not offer.","Quoted exact-match queries give a way to find papers containing a precise phrase such as \"bubble sort\" within a chosen field.","The weighted BM25 scheme provides a deterministic ranking for both single-field and multi-field queries over a corpus of about 320,000 pseudocode blocks."],"supporting_citations":[{"why":"Supplies the roughly 320,000 pseudocode samples, extracted from LaTeX source files of scholarly preprints, that form the engine's dataset.","marker":"Toksoz et al. (2024)"},{"why":"Provides a pseudocode-to-code dataset that is one of the small, hand-crafted collections the paper contrasts with its much larger corpus.","marker":"Kulal et al. (2019)"},{"why":"Provides a manually crafted pseudocode dataset for translation experiments, another baseline illustrating the scale gap the new corpus fills.","marker":"Oda et al. (2015)"},{"why":"Contributes a small program-synthesis pseudocode dataset, further motivating the need for a large search-oriented collection.","marker":"Zavershynskyi et al. (2018)"},{"why":"Describes an earlier search engine for structurally indexed mathematical formulas, the closest prior model for searching formatted scholarly content.","marker":"Kohlhase and Sucan (2006)"},{"why":"Extends that formula search engine to larger scale, serving as the related-work baseline for specialized document search.","marker":"Kohlhase et al. (2012)"}],"fun_headline_variants":["Search papers by their pseudocode","PseudoSeer lets you query papers by pseudocode","Find academic papers using pseudocode snippets","Search LaTeX pseudocode in academic papers"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the two hand-chosen indexing and ranking choices—keeping raw LaTeX commands as searchable tokens and giving title/abstract/references twice the weight of LaTeX/author fields—produce results users find relevant; the paper does not run an experiment that compares them to alternatives.","fun_headline_variants_meta":{"raw":{"variants":["Search papers by their pseudocode","PseudoSeer lets you query papers by pseudocode","Find academic papers using pseudocode snippets","Search LaTeX pseudocode in academic papers"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000352,"raw_usage":{"total_tokens":1866,"prompt_tokens":838,"completion_tokens":1028,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":454,"completion_tokens_details":{"reasoning_tokens":973}},"tokens_in":454,"tokens_out":1028,"duration_ms":10644,"temperature":1.0,"reasoning_tokens":973,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T17:17:37.611199+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a set of, say, 50 papers with pseudocode and a list of queries with known relevant results; run the engine under three conditions—raw LaTeX tokens with the stated weights, stripped LaTeX with equal weights, and raw LaTeX with equal weights—and have raters score the top results. If the stated configuration does not rank known-relevant papers above the alternatives, the engine's ranking advantage is not supported.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides a manually crafted pseudocode dataset for translation experiments, another baseline illustrating the scale gap the new corpus fills."},{"cited_title":"NAPS: Natural Program Synthesis Dataset","cited_arxiv_id":"1807.03168","evidence_quote":"Contributes a small program-synthesis pseudocode dataset, further motivating the need for a large search-oriented collection."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Describes an earlier search engine for structurally indexed mathematical formulas, the closest prior model for searching formatted scholarly content."},{"cited_title":"Matican, and Corneliu C","cited_arxiv_id":null,"evidence_quote":"Extends that formula search engine to larger scale, serving as the related-work baseline for specialized document search."}],"review_version":1}