{"id":"f1c97a0d-8717-44a9-a555-230e99f836e4","arxiv_id":"2505.04666","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"On a self-generated building code QA dataset, Elasticsearch outperforms dense retrievers, and LoRA fine-tuning improves LLM answer quality.","lead":"This paper tests six retrieval methods and six language models for question answering over the National Building Code of Canada, finding Elasticsearch best and LoRA fine-tuning helpful. It builds a 1,436-question dataset from the code using a vision-language model, but does not ship code or data.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Train/test split may leak same-page CQA triplets; a random triplet-level split likely places roughly 69% of test items with a same-page sibling in training, potentially inflating the reported fine-tuning gains.","rationale":"The reader's weakest assumption concerned the accuracy and representativeness of the machine-generated CQA dataset. That concern is valid, but the more decisive and more concretely testable weakness is the construction of the train/test split used for the fine-tuning experiments. The generation procedure in Section 3.2 and Figure 7 creates two triplets per page image, and the paper never states whether the 991/445 split was performed at the page level. A random triplet-level split would create widespread same-page overlap between training and test, and with 94.3% extractive answers, the automatic metrics would reward memorization of the exact context and answer text seen in training. This directly threatens the paper's central claim that LoRA fine-tuning on an NBCC-derived dataset improves LLM answer generation. The retriever comparison also lacks error bars and the ES advantage over TF-IDF/S-BERT is often in the third decimal place, which is a secondary concern, but the split-integrity issue is more fundamental because it can invalidate the larger and more impressive generation results. Since the concern is unconfirmed and can be settled by a page-level re-split and re-evaluation, the appropriate verdict remains CONDITIONAL, matching the reader's original verdict; hence verdict_should_be is UNCHANGED.","tokens_in":22788,"tokens_out":10892,"duration_ms":116583,"concrete_test":"Ask the authors to release the dataset with source-page/image identifiers (Data availability says it will be provided on request). Group the 1436 triplets by source page, compute how many of the 445 test triplets share a page with a training triplet, and measure lexical overlap between train and test contexts. Then rerun the LoRA fine-tuning using a page-level grouped split (e.g., GroupShuffleSplit or manually assigning all triplets from each page to one fold) and recompute Table 3. If the F1/BLEU/ROUGE improvements for Llama-3.1-8b and other models largely disappear or drop within noise, the fine-tuning claim fails; if the improvements persist under a page-separated split, the leakage concern is resolved.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's fine-tuning claim depends on the integrity of the 991/445 train/test split described in Section 3.2, but the split is reported only as counts. The dataset was generated by prompting InternVL2.5-8B to produce exactly two CQA triplets from each NBCC page image (prompt in Figure 7), and Section 3.3 reports that 1354 of 1436 answers are extractive, meaning answers are nearly verbatim spans of their contexts. If the split was made at the triplet level without grouping by source page, then for each page with two triplets there is a substantial probability that one triplet lands in training and its sibling lands in testing. A random triplet-level split of 1436 triplets would put roughly 307 of the 445 test triplets (about 69%) in this situation. Because the two triplets from a page share the same page text as context, the fine-tuned model may have seen nearly identical or heavily overlapping context and answer text during training. Under this leakage, the headline results for Llama-3.1-8b (F1 +58.59%, BLEU +101.14%) and the other large gains could reflect memorization of same-page content rather than domain-specific adaptation to the NBCC. This is not an accusation; it is a missing detail about the split construction that is directly checkable and, if confirmed, would invalidate the generation half of the central claim.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper presents an experimental study of retrieval-augmented generation (RAG) components for question answering over the National Building Code of Canada (NBCC). The authors generate a Context-Question-Answer (CQA) dataset from NBCC page images using InternVL2.5-8B, compare six retrievers (TF-IDF, BM25, Elasticsearch, DPR+FAISS, S-BERT+cosine, DPR+Annoy) using BERTScore precision/recall/F1, and evaluate six LLMs before and after LoRA fine-tuning on a train split of the dataset. They report that Elasticsearch is the most robust retriever and that fine-tuning improves generation, with the largest relative gains for Llama-3.1-8b (F1 +58.59%, BLEU +101.14%).","tokens_in":23050,"tokens_out":4903,"duration_ms":45073,"significance":"If the results are robust, the paper provides a useful domain-specific benchmark and practical guidance for building-code QA: the retriever comparison covers standard baselines, the dataset-construction prompt is specified, and the fine-tuning protocol is described in reasonable detail. The paper is less valuable as a scientific claim about retrieval or fine-tuning because the evaluation is entirely internal to the machine-generated dataset and lacks statistical uncertainty quantification. The main strengths are the breadth of compared methods and the transparency of the dataset-construction prompt; the main weaknesses are the absence of external validation, the lack of error bars, and the unresolved question of train/test leakage in the fine-tuning split.","major_comments":[{"comment":"The train/test split for fine-tuning is reported only as counts (991 for training, 445 for testing), but the dataset was created by prompting InternVL2.5-8B to generate exactly two CQA triplets per page image. If the split was performed at the triplet level without grouping by source page, a large fraction of test triplets (roughly 69% under random assignment) would share a source page and nearly all of its context with a training sibling, especially given that 1354 of 1436 answers are extractive (Section 3.3). In that case the reported gains (e.g., Llama-3.1-8b F1 +58.59%, BLEU +101.14%) could largely reflect memorization of near-identical context rather than domain adaptation. The authors must specify whether the split was page-exclusive and, if it was not, rerun the generation evaluation on a page-exclusive split.","section":"Section 3.2, Figure 7, Table 3"},{"comment":"The claim that Elasticsearch is the most robust retriever is based on absolute BERT F1 differences that are very small (e.g., 0.845 vs 0.840 for TF-IDF at top-1, and 0.849 vs 0.847 at top-3). No standard deviations, confidence intervals, or paired significance tests are reported, so these differences could be within noise. The authors should report per-query variability and a paired test (e.g., bootstrap or Wilcoxon) to support the ranking.","section":"Section 4.1, Table 2"},{"comment":"The CQA dataset is generated by InternVL2.5-8B and only 'manually filtered,' with no human verification of question naturalness or answer correctness reported in the paper. Because this dataset is the ground truth for both retrieval and generation evaluation, the reported absolute scores measure agreement with machine-generated references rather than actual correctness against the code. The authors should include a human-validated subset (even a few hundred triplets) or an external check against the source code text to confirm the dataset's accuracy and representativeness.","section":"Section 3.2, Section 4.2"},{"comment":"The retrieval evaluation uses the same CQA triplets to define both queries and ground-truth contexts; since the questions were generated from those contexts, the retrieval task is easier than open retrieval and the absolute BERT scores are inflated. This does not by itself invalidate the relative comparison among retrievers, but the paper should explicitly acknowledge this circularity and temper the language that describes Elasticsearch as 'the most robust retriever for the NBCC' without qualification.","section":"Section 4.1, Section 3.2"}],"minor_comments":[{"comment":"The BM25 formula appears to have a missing closing parenthesis in the denominator; please fix the typesetting so that the formula is unambiguous.","section":"Equation (4)"},{"comment":"Sentence Mover's Similarity is described as a similarity metric, but the paper notes that lower scores indicate greater similarity; please clarify that it is a distance-like measure (e.g., 'Sentence Mover's Distance') to avoid confusion.","section":"Section 2.4.4"},{"comment":"The terminology for 'F1 Score' and 'BERT F1 Score' is used inconsistently; please define both terms clearly and use the same names in the text and table so that the reported improvements are unambiguous.","section":"Table 3 and Section 4.2"},{"comment":"The TF-IDF row reports '0.84' for top-1 BERT F1 while other entries use three decimals; please standardize the decimal places.","section":"Table 2"},{"comment":"The sample context contains the phrase 'bylbaws' (likely a typo for 'bylaws') and 'sloping use' (likely 'for use'); please correct these transcription errors.","section":"Figure 8"}],"recommendation":"major_revision","confidential_remarks":"The key uncertainty is the construction of the train/test split. If the split is triplet-level rather than page-exclusive, the fine-tuning half of the central claim would need to be re-evaluated with a proper split. Given that the dataset is machine-generated, I recommend asking the authors for a small human-validated sample and for significance testing on the retriever ranking before resubmission. The retriever comparison may still be of interest, but the current evidence is too weak to support the strong 'most robust' conclusion."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"What you should know: this is an applied RAG paper, not a methods paper. The new piece is a 1,436-triplet CQA dataset derived from the National Building Code of Canada, plus benchmark numbers for six retrievers and six LLMs before and after LoRA fine-tuning. That dataset could genuinely save time for people building code-compliance QA systems; the retriever comparison is straightforward and honestly reported.\n\nThe retriever story is plausible but weaker than the paper claims. Elasticsearch comes out on top, but the gap over TF-IDF, BM25, and S-BERT is tiny (BERT F1 differences are roughly 0.005–0.010). Without error bars or significance tests, “most robust” is too strong. The dense retrievers (DPR-FAISS, DPR-AI) trail clearly, but that is likely a configuration issue as much as an inherent limitation.\n\nThe bigger problem is the fine-tuning half. The dataset was generated by prompting InternVL2.5-8B to produce exactly two CQA triplets per NBCC page image, and 94% of answers are extractive, nearly verbatim spans of their context. The paper reports only a 991/445 train/test count, not how the split was constructed. If triplets were split at the triplet level without grouping by source page, then roughly 69% of test items share a page with a training item, making the reported F1 and BLEU gains (58.59% and 101.14% for Llama-3.1-8b) potentially a memorization artifact rather than domain adaptation. This is directly checkable and must be addressed before the generation claims are taken seriously.\n\nThere are smaller soft spots. The dataset is machine-generated with no human verification of correctness beyond manual filtering, so the ground truth itself is untrusted. The retrieval evaluation is also self-referential: questions were generated from the same pages that define relevance, which inflates retrieval scores. No code or data is released, only “on request,” and no end-to-end RAG system is actually built or measured, despite the title and conclusion implying one.\n\nWho is this for? Practitioners in construction AI who want a building-code QA recipe, and researchers who might use the dataset as a benchmark. It deserves a serious referee, but the revision needs to document the split construction, rerun fine-tuning with page-level grouping, add significance testing, and release the data. If the split leakage is confirmed, the generation half collapses; if not, the paper is a modest but useful empirical contribution. I would send it to review, with the split issue as the main requested revision.","headline":"Useful NBCC dataset and retriever comparison; the fine-tuning gains may be inflated by same-page train/test leakage that the paper must rule out.","tokens_in":23566,"tokens_out":1657,"would_cite":false,"duration_ms":19495,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Elasticsearch is the most reliable retriever for building-code queries, and LoRA fine-tuning on code-derived data sharply improves LLM answers.","keywords":["retrieval-augmented generation","building codes","Elasticsearch","LoRA fine-tuning","question answering","BERTScore","National Building Code of Canada","parameter-efficient fine-tuning"],"falsifier":"Ask building-code experts to audit a random sample of the 445 test triplets against the NBCC text; if a sizable fraction of ground-truth answers are not supported by their paired contexts, the reported rankings lose their basis. A lighter check is to rerun the retriever comparison using human relevance judgments instead of BERTScore and see whether Elasticsearch still leads.","tokens_in":22606,"feed_emoji":"🏗️","tokens_out":9789,"duration_ms":79893,"temperature":0.7,"pith_summary":"The paper tries to settle two practical questions for building a question-answering system over the National Building Code of Canada: which retriever pulls out the right code passages, and whether fine-tuning makes a large language model better at turning those passages into answers. It reports that Elasticsearch, a lexical search engine built on BM25 ranking, beats all dense and sparse competitors on BERT F1 at every top-k setting. It also reports that lightweight LoRA fine-tuning on roughly a thousand code-derived context-question-answer triples improves every model's answer quality, with Llama-3.1-8b more than doubling its BLEU score. If these results hold, a workable recipe is a lexical retriever plus a lightly fine-tuned open model, which matters because building codes are long, technical, and frequently queried by practitioners.","feed_headline":"Elasticsearch wins for building-code QA; fine-tuning doubles BLEU","feed_subtitle":"Lexical search beats dense retrievers on Canada's building code, and LoRA fine-tuning lifts LLM BLEU by up to 101%.","key_machinery":"The load-bearing mechanism is the two-stage RAG pipeline. In the retrieval stage, Elasticsearch's inverted index preprocesses text through tokenization, lowercasing, stemming, and lemmatization, then ranks passages with the BM25 scoring formula, which proved better at capturing the exact technical wording of building-code clauses than dense vector search. In the generation stage, LoRA injects small trainable low-rank matrices into the attention and feed-forward projections (q_proj, k_proj, v_proj, o_proj, gate_proj, up_proj, down_proj) of a frozen, 4-bit-quantized transformer, so each model adapts to the NBCC while updating less than 1.2% of its parameters. The evaluation is carried out with BERTScore for retrieved passages and with F1, BLEU, ROUGE-1, SMS, METEOR, and BERT scores for generated answers.","core_discovery":"The paper's central claim is that for the National Building Code of Canada, retrieval works best when it is lexical rather than semantic: Elasticsearch posts the highest BERT F1 scores at every top-k setting (0.845, 0.849, 0.847, and 0.844 at top-1, top-3, top-5, and top-10), ahead of TF-IDF, BM25, DPR with FAISS, Sentence-BERT with cosine similarity, and DPR with Annoy. It further claims that LoRA fine-tuning on an NBCC-derived dataset improves LLM answer generation across all six models tested, with the largest relative gains in Llama-3.1-8b (F1 from 0.326 to 0.517, BLEU from 0.176 to 0.354) and the highest absolute scores in Mistral-Small-24b-Instruct-2501 (fine-tuned F1 0.688, BLEU 0.549). Taken together, the two findings imply that a RAG system for building codes can be optimized by pairing Elasticsearch with a lightly fine-tuned open LLM.","pith_inferences":["Because 1,354 of the 1,436 answers are extractive, much of the measured gain may reflect the model learning to copy relevant spans from context; abstractive reasoning over codes is a harder test the current metrics could miss.","A hybrid retriever that merges Elasticsearch's lexical hits with dense semantic scores is a natural next experiment; the paper does not test this, but its own data suggests the two signal types are complementary.","The same pipeline can likely be transferred to other regulatory corpora, but a test on a different building code or standards document would be needed to confirm that Elasticsearch's edge is not specific to the NBCC's layout and vocabulary.","A human-verified gold set would strengthen confidence: the current ground truth is generated by a vision-language model and manually filtered only for low relevance, not for correctness."],"forward_implications":["A RAG system for the NBCC can combine Elasticsearch with a LoRA-fine-tuned LLM and likely answer code queries more accurately than either component alone.","Retrieving only the top-3 or top-5 passages is enough; the paper's scores show top-10 retrieval adds no contextual benefit and can introduce irrelevant text.","Light fine-tuning, touching less than 1.2% of parameters, can produce large generative gains; Llama-3.1-8b's BLEU more than doubled after 10 epochs on 991 triples.","Models with stronger pretrained baselines benefit less from fine-tuning, as seen in Mistral-Small-24b-Instruct-2501, where gains were smaller but absolute performance was highest.","Dense retrievers appear less suitable than lexical ones for technical regulatory text when relevance depends on exact clause wording, at least on this dataset."],"supporting_citations":[{"why":"It supplies the National Building Code of Canada document used as the retrieval and fine-tuning corpus.","marker":"[40]"},{"why":"It defines the retrieval-augmented generation framework that motivates pairing retrievers with LLMs.","marker":"[41]"},{"why":"It provides the multimodal model that generated the 1,436 context-question-answer triplets from page images.","marker":"[62]"},{"why":"It introduces LoRA, the low-rank adaptation method applied to all six language models.","marker":"[37]"},{"why":"It defines BM25, the ranking function that drives Elasticsearch's retrieval performance.","marker":"[42]"},{"why":"It introduces Dense Passage Retrieval, one of the dense baselines compared in the retriever evaluation.","marker":"[44]"},{"why":"It introduces Sentence-BERT, the semantic retriever baseline compared in the retriever evaluation.","marker":"[47]"},{"why":"It introduces BERTScore, the metric used to judge the relevance of retrieved passages.","marker":"[61]"}],"fun_headline_variants":["Building-code QA: Elasticsearch wins, LoRA lifts LLM BLEU 101%","Elasticsearch beats dense retrievers; LoRA fine-tuning boosts LLM","Fine-tuned LLMs + Elasticsearch dominate building-code QA","Elasticsearch trumps dense retrievers; LoRA fine-tuning lifts BLEU","Building-code RAG: Elasticsearch retrieval + LoRA fine-tuning wins"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The evaluation stands or falls on the 1,436 context-question-answer triplets generated by a vision-language model from page images and filtered only for obvious problems; if those triplets contain wrong or misaligned answers, both the retriever ranking and the fine-tuning gains inherit the error.","fun_headline_variants_meta":{"raw":{"variants":["Building-code QA: Elasticsearch wins, LoRA lifts LLM BLEU 101%","Elasticsearch beats dense retrievers; LoRA fine-tuning boosts LLM","Fine-tuned LLMs + Elasticsearch dominate building-code QA","Elasticsearch trumps dense retrievers; LoRA fine-tuning lifts BLEU","Building-code RAG: Elasticsearch retrieval + LoRA fine-tuning wins"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001002,"raw_usage":{"total_tokens":4321,"prompt_tokens":1108,"completion_tokens":3213,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":724,"completion_tokens_details":{"reasoning_tokens":3110}},"tokens_in":724,"tokens_out":3213,"duration_ms":23606,"temperature":1.0,"reasoning_tokens":3110,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T23:35:40.194884+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Ask building-code experts to audit a random sample of the 445 test triplets against the NBCC text; if a sizable fraction of ground-truth answers are not supported by their paired contexts, the reported rankings lose their basis. A lighter check is to rerun the retriever comparison using human relevance judgments instead of BERTScore and see whether Elasticsearch still leads.","supporting_citations":[{"cited_title":"National Building Code of Canada 2020","cited_arxiv_id":null,"evidence_quote":"It supplies the National Building Code of Canada document used as the retrieval and fine-tuning corpus."},{"cited_title":"OpenGVLab/InternVL2_5-8B · Hugging Face","cited_arxiv_id":null,"evidence_quote":"It provides the multimodal model that generated the 1,436 context-question-answer triplets from page images."}],"review_version":1}