Pith. sign in

REVIEW 3 major objections 6 minor 15 references

Optimizing Retrieval-Augmented Generation for Electrical Engineering: A Case Study on ABB Circuit Breakers

T0 review · 3 major / 6 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Across nine RAG configurations on ABB breaker manuals, Claude with paragraph-per-page chunking leads on factual faithfulness, but all configurations land below engineering-grade reliability.

desk verdict Useful, honest applied RAG case study whose headline 80% reliability figure is not supported by the RAGAS metrics; the qualitative failure analysis is the real contribution. read the letter →

arxiv 2505.17520 v1 pith:UBSQYGDE submitted 2025-05-23 cs.AI

Salahuddin Alawadhi , Noorhan Abbas This is my paper
classification cs.AI
keywords Retrieval-AugmentedGenerationElectricalengineeringABBcircuitbreakersChunkingstrategiesEmbeddingmodelsRAGASevaluationFaithfulnesssafety
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper asks whether retrieval-augmented generation can be trusted for high-stakes electrical engineering questions, using ABB circuit-breaker documentation as the test bed. It builds a 12-document corpus on the ABB SACE Emax 2 breaker family, applies three chunking strategies (basic, paragraph-per-page, and by-title), and pairs each with three commercial large language models to create nine retrieval-augmented pipelines. The paper's central claim is that the best configuration is Claude with paragraph-per-page chunking, which reaches a RAGAS faithfulness score of 0.8556, and that despite strong retrieval precision, all nine setups fall short of the reliability an accuracy-intolerant engineering field needs. The conclusion the authors draw is that roughly 80 percent efficiency is inadequate, and that engineering use of RAG will require better chunking, reranking, knowledge-graph grounding, and validation.

What carries the argument

The evaluation machinery is a standard retrieval-augmented pipeline: PDFs are partitioned and chunked with the Unstructured library under three chunking rules, text is summarized with GPT4o-mini at temperature 0, embeddings are stored in Chroma, retrieval returns the top 10 chunks by cosine similarity, and the selected model generates answers at temperature 0. The decisive instrument is RAGAS, an automated evaluation suite producing four scores: faithfulness (claims in the answer supported by the retrieved context), context recall and context precision (whether the right chunks were retrieved), and answer relevancy (mean cosine similarity between the question and three synthetic reverse-engineered questions). The paper treats faithfulness as the primary reliability signal and uses a qualitative reading of sample answers to check the numbers against what a careful reader can verify in the manuals.

What would settle it

Give the same 31 questions and nine configurations to a panel of engineers who work with ABB breakers, with the answers anonymized, and ask them to rank by correctness and completeness. If their ranking does not place Claude with paragraph-per-page chunking first, or if their scores correlate only weakly with the RAGAS faithfulness numbers, the paper's central comparison and its conclusion that all configurations are inadequate would not hold.

Watch

Extended reading notes

Core claim

On a corpus of 12 publicly available ABB SACE Emax 2 documents (537,708 tokens), the authors compare nine configurations formed by crossing three large language models (OpenAI GPT4o, Cohere command-xlarge-nightly, and Anthropic Claude 3.5 Sonnet) with three chunking strategies. Using RAGAS metrics on a 31-question set, they find that Claude with Paragraph-per-Page chunking is the best overall configuration, with the highest faithfulness (0.8556) and perfect context precision (1.0000), while Cohere's Paragraph-per-Page configuration has the best context recall (0.7705) and GPT4o's By-Title configuration has the best answer relevancy (0.8021). They also report that every configuration is below the accuracy threshold they deem necessary for engineering, stating that 'approximately 80% efficiency is inadequate for a field intolerant of inaccuracies.'

Load-bearing premise

The ranking and the 'about 80% efficiency is inadequate' verdict assume that the automated RAGAS scores measure answer quality the way an electrical engineer would judge it, and those scores were never calibrated against independent human experts.

Editorial extensions

If this is right

  • If the comparison is right, teams building engineering RAG systems should prefer paragraph-per-page chunking over by-title or basic chunking for dense procedural manuals, because it gives the best balance of faithfulness and recall.
  • The result puts a numerical floor on current expectations: even the best configuration falls short of the higher faithfulness values needed for design and operational decisions, so RAG answers in this domain should be treated as draft references rather than instructions.
  • Because the prompt, temperature, and token limits were fixed, the score gaps are attributable mainly to model choice and chunking, making chunking a directly controllable lever for system improvement.
  • The 'I don't know' and wrong-detail failures seen in the qualitative questions show that aggregate scores can hide critical omissions, such as failing to give dip-switch positions or citing incorrect termination-resistor values.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A testable extension the paper does not run: re-run the same nine configurations on another manufacturer's breaker manuals to see whether the paragraph-per-page advantage is a general property of semi-structured engineering documentation or specific to this corpus.
  • The authors' own qualitative examples suggest a sharper test: questions engineered to require exact switch positions and part numbers, where partial answers may still score well on automated faithfulness because they are true but incomplete, would likely lower every score.
  • Because the reference answers were constructed from the same documents used for retrieval, a natural next step implied by the method is an independent expert-created gold set; if scores on that set were much lower, the 'about 80%' conclusion would be a conservative estimate of the gap.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. The paper presents a case study evaluating three RAG pipelines (OpenAI GPT-4o, Cohere command-xlarge-nightly, and Anthropic Claude 3.5 Sonnet) combined with three chunking strategies (Basic, Paragraph-per-Page, and By-Title) on a curated 12-document ABB circuit breaker corpus. The authors construct 31 question-answer pairs from these documents, measure performance with four RAGAS metrics (faithfulness, context recall, context precision, answer relevancy), and supplement the quantitative results with qualitative analyses of selected answers. They conclude that Claude with Paragraph-per-Page chunking is the best overall configuration (faithfulness 0.8556), and that the overall performance of approximately 80% efficiency is inadequate for high-stakes electrical engineering use. The paper also discusses limitations and proposes knowledge-graph and multimodal extensions for future work.

Significance. If the evaluation were valid, this would be a useful practical comparison of RAG configurations in a technical domain, with a clearly described corpus, hyperparameters, and qualitative failure examples. The authors deserve credit for documenting the full pipeline, using temperature zero for reproducibility, and including qualitative answers that expose concrete errors. However, the central claim that the best configuration is 'most accurate and reliable' and that 'approximately 80% efficiency is inadequate' rests entirely on RAGAS scores that have not been calibrated against engineering ground truth. The small self-constructed question set, single-run measurements with no confidence intervals, and lack of any human expert validation make the ranking and the reliability conclusion unsupported in their current form. The paper is better positioned as a descriptive comparison of RAGAS scores across pipelines, with the qualitative analysis as evidence of specific failure modes, than as a measurement of engineering reliability.

major comments (3)
  1. [§3.9, §4, Table 4] The RAGAS metrics do not measure engineering correctness, yet they are used to support the 'most accurate and reliable' ranking and the '80% efficiency' conclusion. Faithfulness as defined in §3.9 scores consistency between the generated answer and the retrieved context, not agreement with authoritative ABB documentation; context precision measures retrieval overlap, not answer quality. The paper's own qualitative results demonstrate this gap: in Question 1, several configurations (e.g., Cohere Basic, GPT4o Basic, Claude Basic) give wrong resistance values (220Ω or 390Ω instead of the reference 120Ω) but the corresponding faithfulness scores are moderate to high (e.g., Claude Basic 0.7930). In Question 3, Claude Basic states IEEE/ANSI standards without the specific C37.17 designation and invokes 'inferred' 0.001 increments, yet its faithfulness is high. The metrics cannot distinguish these errors, so the ranking and the reliability judgment are not supported by the reported numbers.
  2. [§5, Table 4] The 'approximately 80% efficiency' figure is an unweighted arithmetic mean of four heterogeneous metrics. For Claude Paragraph-per-Page, (1.0000 + 0.8556 + 0.7410 + 0.6913)/4 = 0.822, so the perfect context-precision score inflates the average and masks lower recall and answer relevance. Since context precision measures retrieval overlap rather than answer correctness, this average does not constitute a meaningful 'efficiency' measure for engineering reliability. The conclusion in §5 that 'approximately 80% efficiency is inadequate for a field intolerant of inaccuracies' therefore conflates a retrieval-side score with a holistic accuracy judgment.
  3. [§3.9, §3.1] The benchmark is not external to the system under test. The 31 reference answers were constructed by the authors reviewing the same ABB documents that constitute the retrieval corpus, and the answer-relevancy metric is computed from synthetic questions generated by an LLM from the model's own answer, compared via cosine similarity with N=3. With no human expert validation of the reference answers or the metric scores, and with single-run RAGAS estimates, there is no basis for the general claim in §5 about the system's adequacy in a 'field intolerant of inaccuracies' or for fine-grained distinctions such as the 0.8556 vs. 0.8342 faithfulness difference between Claude configurations. The descriptive ranking may stand as a report of RAGAS behavior, but the broader reliability conclusion is not supported.
minor comments (6)
  1. [§3.9] The text says 'The formulas for k and context precision are shown below:' but no equations are actually displayed in the manuscript; either include the formulas or delete the sentence.
  2. [§3.5] There are typos and formatting inconsistencies, for example 'withing' for 'within' and the mixed use of '4000' and '4,000' within the same paragraph.
  3. [§3.5] The OpenAI model URL is truncated ('https://platform.openai.com/docs/models/gp#gpt-4o'); it should point to the correct page or be removed.
  4. [§3.6] The phrase 'fetching up to 2k documents (default k=10)' is confusing; §3.7 states that the top 10 documents are ranked and used. Please clarify the relationship between k and the actual number of retrieved documents.
  5. [§4] Tables 2–4 report single-point estimates without any variance, confidence intervals, or number of runs; at minimum, state explicitly that these are single-run measurements and that differences may not be statistically meaningful.
  6. [§5] The term 'approximately 80% efficiency' is used without a precise definition; specify exactly which values are being averaged and why this average is interpreted as efficiency.

Circularity Check

0 steps flagged · score 0.0 of 10

No circular derivation chain: the RAG ranking and efficiency summary are observed metrics, not fitted quantities or self-citation-forced conclusions.

full rationale

The paper's central claims are empirical rankings of three RAG pipelines against RAGAS metrics on a 31-question ABB-specific benchmark, plus a summary judgment that roughly 80% average metric performance is inadequate for engineering use. No step in this chain reduces to its inputs by construction. The reference question-answer set was constructed from the same ABB documentation used for retrieval, but that is an appropriate benchmark design for a retrieval system rather than a definitional equivalence: the generated answers are produced by the LLM from retrieved context and scored against reference answers, not fitted to those answers and then re-predicted. The reported faithfulness, context precision, context recall, and answer-relevancy scores are observed outputs of off-the-shelf RAGAS measurements; the 'approximately 80% efficiency' figure is a summary of those observed scores, not a parameter fitted to a subset and then called a prediction. There is no load-bearing self-citation chain: the cited works are external background studies on RAG, embeddings, and vector databases, and no uniqueness theorem or prior-work ansatz is invoked to force the choice of model, chunking strategy, or embedding. The qualitative analysis even shows failures (e.g., wrong resistor values and standards citations) that the quantitative metrics do not penalize, which cuts against any claim that the quantitative result is forced by construction. The acknowledged limitation of focusing on a single breaker and the caveat that RAGAS faithfulness measures consistency with retrieved context rather than external engineering ground truth are external-validity and metric-interpretation concerns, not circularity.

Assumptions & free parameters 5 free parameters · 4 assumptions · 0 invented entities

The evaluation depends on four main assumptions: RAGAS metric validity, author-constructed ground truth, sufficiency of the 12-document corpus, and chunking fidelity. No new physical entities are introduced. The free parameters are hand-chosen chunking and context limits that shape the results.

free parameters (5)
  • Basic chunking hyperparameters = max_characters=1000, new_after_n_chars=800, combine_text_under_n_chars=500
    Chosen by hand in Section 3.2; the ranking of chunking methods depends on these cutoffs.
  • Paragraph-per-page chunking hyperparameters = max_characters=1500, new_after_n_chars=1200, combine_text_under_n_chars=700
    Chosen by hand in Section 3.2.
  • By-title chunking hyperparameters = max_characters=4000, new_after_n_chars=3800, combine_text_under_n_chars=2000
    Chosen by hand in Section 3.2.
  • max_context_tokens = 4000
    Selected after observing that a 2,000-token context produced incomplete responses (Section 3.5); it directly shapes output completeness.
  • top_k retrieval = 10
    Default setting in Section 3.7; retrieval breadth and context truncation depend on it.
assumptions (4)
  • domain assumption RAGAS metrics are valid proxies for RAG output quality in this domain.
    Section 3.9 uses faithfulness, context recall, context precision, and answer relevancy without calibration against expert human judgments in electrical engineering.
  • domain assumption The 31-question reference set is a correct and representative ground truth for ABB breaker queries.
    Described in Section 3.9 as 'carefully constructed by reviewing the documents'; the questions and answers are not published.
  • domain assumption The selected 12 ABB documents are sufficient to answer the reference questions and represent the knowledge needed for real engineering tasks.
    In Section 3.1 the authors deliberately use a subset of documents focused on one breaker series, which may not cover all relevant knowledge.
  • domain assumption Unstructured's partition_pdf and the chosen chunking parameters preserve the meaning of tables and nested technical content.
    Invoked in Section 3.2; no verification is provided that table structures and cross-references survive chunking without information loss.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Optimizing Retrieval-Augmented Generation for Electrical Engineering: A Case Study on ABB Circuit Breakers." pith.science (2026). https://pith.science/paper/UBSQYGDE

@misc{pith2026250517520,
  author       = {Pith},
  title        = {Pith review of: Optimizing Retrieval-Augmented Generation for Electrical Engineering: A Case Study on ABB Circuit Breakers},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/UBSQYGDE}},
  note         = {Machine review of arXiv:2505.17520}
}
read the original abstract

Integrating Retrieval Augmented Generation (RAG) with Large Language Models (LLMs) has shown the potential to provide precise, contextually relevant responses in knowledge intensive domains. This study investigates the ap-plication of RAG for ABB circuit breakers, focusing on accuracy, reliability, and contextual relevance in high-stakes engineering environments. By leveraging tailored datasets, advanced embedding models, and optimized chunking strategies, the research addresses challenges in data retrieval and contextual alignment unique to engineering documentation. Key contributions include the development of a domain-specific dataset for ABB circuit breakers and the evaluation of three RAG pipelines: OpenAI GPT4o, Cohere, and Anthropic Claude. Advanced chunking methods, such as paragraph-based and title-aware segmentation, are assessed for their impact on retrieval accuracy and response generation. Results demonstrate that while certain configurations achieve high precision and relevancy, limitations persist in ensuring factual faithfulness and completeness, critical in engineering contexts. This work underscores the need for iterative improvements in RAG systems to meet the stringent demands of electrical engineering tasks, including design, troubleshooting, and operational decision-making. The findings in this paper help advance research of AI in highly technical domains such as electrical engineering.

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

15 extracted references · 6 canonical work pages

  1. [1]

    Critical Infrastructures Vulnerability and Risk Analysis,

    E. Zio, “Critical Infrastructures Vulnerability and Risk Analysis,” European Journal for Securi ty Research, vol. 1, no. 2, pp. 97–114, Mar. 2016, doi: https://doi.org/10.1007/s41125-016-0004-2

    work page doi:10.1007/s41125-016-0004-2 2016

  2. [2]

    Failures In Electrical Systems, Equipments & Materials - Causes & Prevention,

    E. Technology, “Failures In Electrical Systems, Equipments & Materials - Causes & Prevention,” ELECTRICAL TECHNOLOGY, Aug. 05, 2018. https://www.electricaltechnology.org/2018/08/failures-in-electrical-systems-equipments- materials.html

    work page 2018

  3. [3]

    Improving Retrieval -Augmented Generation in Medicine with Iterative Follow -up Questions,

    G. Xiong, Q. Jin, X. Wang, M. Zhang, Z. Lu, and A. Zhang, “Improving Retrieval -Augmented Generation in Medicine with Iterative Follow -up Questions,” arXiv.org, 2024. https://arxiv.org/abs/2408.00727

    arXiv 2024

  4. [4]

    Blog - Retrieval-Augmented Generation: Engineering a Smarter AI Future,

    “Blog - Retrieval-Augmented Generation: Engineering a Smarter AI Future,” Redline Group, 2024. https://www.redlinegroup.com/insight-details/retrieval-augmented-generation-engineering-a-smarter- ai-future?

    work page 2024

  5. [5]

    Maximizing RAG efficiency: A comparative analysis of RAG methods,

    T. Şakar and H. Emekci, “Maximizing RAG efficiency: A comparative analysis of RAG methods,” Natural Language Processing, pp. 1–25, Oct. 2024, doi: https://doi.org/10.1017/nlp.2024.53

    doi:10.1017/nlp.2024.53 2024

  6. [6]

    Performance Evaluation of Vector Embeddings with Retrieval -Augmented Generation,

    S. Kukreja, T. Kumar, Vishal Bharate, A. Purohit, A. Dasgupta, and D. Gu ha, “Performance Evaluation of Vector Embeddings with Retrieval -Augmented Generation,” 2022 7th International Conference on Computer and Communication Systems (ICCCS), pp. 333 –340, Apr. 2024, doi: https://doi.org/10.1109/icccs61882.2024.10603291

    arXiv 2022

  7. [7]

    Vector Databases and Vector Embeddings-Review,

    S. Kukreja, T. Kumar, Vishal Bharate, A. Purohit, A. Dasgupta, and D. Guha, “Vector Databases and Vector Embeddings-Review,” Dec. 2023, doi: https://doi.org/10.1109/iwaiip58158.2023.10462847

    arXiv 2023

  8. [8]

    A Method for Parsing and Vectorization of Semi -structured Data used in Retrieval Augmented Generation,

    H. Yang et al., “A Method for Parsing and Vectorization of Semi -structured Data used in Retrieval Augmented Generation,” arXiv.org, 2024. https://arxiv.org/abs/2405.03989

    arXiv 2024

Show all 15 references
  1. [9]

    Analyzing Embedding Models for Embedding Vectors in Vector Databases,

    Paras Nath Singh, Sreya Talasila, and Shivaraj Veerappa Banakar, “Analyzing Embedding Models for Embedding Vectors in Vector Databases,” Dec. 2023, doi: https://doi.org/10.1109/ictbig59752.2023.10455990

    2023

  2. [10]

    Developing Retrieval Augmented Generation (RAG) based LLM Systems from PDF s: An Experience Report,

    A. A. Khan, M. T. Hasan, K. K. Kemell, J. Rasku, and P. Abrahamsson, “Developing Retrieval Augmented Generation (RAG) based LLM Systems from PDF s: An Experience Report,” arXiv.org,

  3. [11]

    A Comprehensive Survey on Vector Database: Storage and Retrieval Technique, Challenge,

    Y. Han, C. Liu, and P. Wang, “A Comprehensive Survey on Vector Database: Storage and Retrieval Technique, Challenge,” arXiv.org, Oct. 18, 2023. https://arxiv.org/abs/2310.11703 (ac cessed Nov. 26, 2023)

    2023

  4. [12]

    Generative Retrieval -Augmented Ontologic Graph and Multiagent Strategies for Interpretive Large Language Model-Based Materials Design,

    M. J. Buehler, “Generative Retrieval -Augmented Ontologic Graph and Multiagent Strategies for Interpretive Large Language Model-Based Materials Design,” ACS Engineering Au, Jan. 2024, doi: https://doi.org/10.1021/acsengineeringau.3c00058. 76 Computer Science & Information Tech...

    2024 doi

  5. [13]

    Retrieval augmented generation using engineering design knowledge,

    L. Siddharth and J. Luo, “Retrieval augmented generation using engineering design knowledge,” Knowledge-Based Systems, vol. 303, p. 112410, Nov. 2024, doi: https://doi.org/10.1016/j.knosys.2024.112410

    2024

  6. [14]

    Development of a Cognitive Assistant for Industrial Maintenance Based on Retrieval- Augmented Generation (RAG) at STMicroelectronics

    D. Machado, “Development of a Cognitive Assistant for Industrial Maintenance Based on Retrieval- Augmented Generation (RAG) at STMicroelectronics.” Available: https://repositorio.ufsc.br/bitstream/handle/123456789/256433/TCC_MACHADO_PDFA.pdf?sequen ce=6 Computer Science & Info...

    2025

  7. [2024]

    https://arxiv.org/abs/2410.15944

    arXiv

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.