{"id":"fe8b0c77-1360-4056-b49a-7a305361a2a6","arxiv_id":"2501.01237","paper_version":2,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"Error-based prompt rewriting and self-correction do not significantly improve GPT-4o product attribute extraction on OA-Mine or AE-110K, and both raise token costs.","lead":"This paper tested two automatic self-refinement tricks to help AI extract product attributes like brand and size from product titles. Both tricks raised computing cost without improving accuracy, and the authors recommend fine-tuning a model instead when you have enough data.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Central 'insignificant' claim rests on paired t-tests with only three runs; with df=2 at 99% confidence the test is too low-powered to support a negative conclusion.","rationale":"The reader's weakest assumption is exactly the low statistical power of the three-run paired t-tests, and my read agrees. The paper's tables and code are valuable, and the direction of the results (small changes plus large token overhead) is plausible, so this is not a reason to reject. However, the wording 'fail to significantly improve' should be softened or backed by a per-item significance test and reported effect sizes. I also note a secondary arithmetic concern: applying the formula in Section 5 to the token counts in Table 3 gives roughly 394,000/(1,315-172) = 345 offers for OA-Mine and 431,000/(1,351-177) = 367 for AE-110k, not 6,666, unless an unstated dollar-price model is used; this should be clarified or corrected in revision. Neither issue moves the verdict beyond CONDITIONAL, so the reader's verdict remains unchanged.","tokens_in":10360,"tokens_out":11813,"duration_ms":120728,"concrete_test":"Using the released code, compare Few-Shot vs Few-Shot+Self-Correction on OA-Mine with a paired bootstrap over the 491 test offers: resample offers with replacement, recompute the global F1 (or a per-offer F1) for both conditions in each resample, and report a 95% confidence interval for the mean difference. If the interval is wide enough to include a practically meaningful gain such as +2 F1, the paper should replace 'fails to significantly improve' with a statement about the precision of the estimate; if the interval excludes meaningful gains at high confidence, the negative conclusion is supported.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's main negative result is that self-refinement 'fails to significantly improve' extraction. This is supported by paired t-tests computed over three runs per configuration (Section 4). With df=2 and a 99% confidence level, the critical t is 9.925, so a difference must be roughly six standard deviations of the run-to-run variation to be called significant. The reported deltas (e.g., +0.4 F1 for zero-shot self-correction on OA-Mine in Table 6, and -0.3 and +0.2 for few-shot rewriting in Table 5) are far below what this design could detect. 'Insignificant' here therefore means 'not detectable with n=3', not 'no meaningful improvement.' No p-values, effect sizes, or confidence intervals are reported, and the F1 is aggregated over the whole test set, which hides per-offer variance. Because the abstract states that self-refinement fails to significantly improve performance, this low-power statistical basis is the load-bearing step; without a higher-powered analysis the central negative claim is not established.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper experimentally evaluates two automated self-refinement techniques for product attribute value extraction: error-based prompt rewriting (Section 6) and self-correction (Section 7). Both are applied on top of zero-shot, few-shot in-context learning, and fine-tuned GPT-4o prompts, and evaluated on OA-Mine and AE-110K subsets using F1 and token-usage metrics. The headline findings are that self-refinement yields no significant F1 improvement while increasing token costs, and that fine-tuning achieves the highest F1 and becomes cheaper than few-shot in-context learning beyond a reported break-even of 6,666 product offers.","tokens_in":10577,"tokens_out":11865,"duration_ms":112333,"significance":"If the results hold, this is a useful, clearly presented negative result for practitioners: it quantifies the token overhead of two popular self-refinement recipes and shows that fine-tuning is competitive on this task. The manuscript is internally consistent, uses standard external benchmarks, and releases code and data, which is a real strength. The main limitations are statistical: the 'no significant improvement' conclusion is supported by an extremely low-power test, and the cost break-even number does not reproduce from the paper's own Table 3. Because these two issues bear directly on the abstract's claims, the manuscript needs revision before the practical recommendations can be accepted.","major_comments":[{"comment":"The abstract's statement that self-refinement fails to significantly improve extraction is the paper's central claim, but the statistical test behind it is too weak to support a negative conclusion. Section 4 reports paired t-tests over three runs per configuration; with df=2 and alpha=0.01 (two-tailed), a difference must exceed roughly 9.9 standard errors, i.e., about 5.7 times the standard deviation of the paired differences, to be called significant. The observed deltas in Tables 5 and 6 (e.g., +0.4 F1 for zero-shot self-correction on OA-Mine; -0.3 and +0.2 F1 for few-shot rewriting) are far below that threshold, and no p-values, effect sizes, or confidence intervals are reported. 'Insignificant' is therefore better read as 'not detectable with n=3' than as 'no meaningful improvement.' The same absence of supporting statistics weakens the positive significance claims in Section 5 (e.g., attribute definitions +3.4 F1 on OA-Mine). Please report the actual test statistics and effect sizes for every pairwise comparison, or use an equivalence-testing/confidence-interval framework; if the pairing is instead over individual product offers rather than over the three runs, please say so explicitly, because the current wording suggests three paired F1 scores.","section":"Section 4 / Abstract"},{"comment":"The break-even point in Section 5 does not match the paper's own numbers. Using the formula FineTuningCost/(TokenCost_FewShot - TokenCost_FineTuning) with the values in Table 3 gives 394,000/(1,315-172) ≈ 345 offers for OA-Mine and 431,000/(1,351-177) ≈ 367 offers for AE-110K, not 6,666. The reported value is off by roughly a factor of 19. Since the abstract and conclusion use this break-even to justify fine-tuning for large numbers of product offers, either the calculation (including how the OpenAI input/output prices are applied) needs to be shown in detail and corrected, or Table 3 must be amended so the numbers are consistent.","section":"Section 5"},{"comment":"The conclusions are phrased in terms of 'LLMs' and 'self-refinement strategies', but all experiments use a single model, GPT-4o, and subsets of two datasets. This is a reasonable first probe, but it does not establish that the null result generalizes across model families or prompt formats; prior work cited by the authors shows that self-refinement effects vary by task and model. Please either temper the title and conclusion to 'GPT-4o' explicitly, add at least one additional model for the main comparisons, or clearly mark the single-model scope as an open limitation rather than a general negative result.","section":"Sections 5-7 / Scope"}],"minor_comments":[{"comment":"The text says Table 1 contains statistics for 'all four datasets', but Table 1 reports two datasets with development/test splits; change to 'both datasets'.","section":"Section 3"},{"comment":"The sentence reporting '90% of the 423 attribute value updates have no impact' does not name the dataset; from context this is AE-110K, but it should be stated explicitly.","section":"Section 7"},{"comment":"Error-based prompt rewriting uses 'five randomly selected product offers' and repeats the process three times, but no random seed or repeated sampling is reported; please state how randomness was controlled, since the rewriting output depends on this sample.","section":"Section 6"},{"comment":"The paper mentions the different OpenAI input/output token prices, but the break-even calculation does not show how they enter; a worked example of the cost computation would prevent the confusion noted in the major comments.","section":"Section 5"},{"comment":"There are small wording and formatting typos: 'related work[BSB25]' and 'Section3and Section4' lack spaces; Figure 4 says 'incorrectly and correctly extract attribute values' instead of 'extracted'; and the dataset name appears as both AE-110K and AE-110k.","section":"Throughout"}],"recommendation":"major_revision","confidential_remarks":"For the editor: the code and data release and the use of external benchmarks are strengths, and there is no circularity concern. The main risks are the underpowered significance testing for the headline negative result and an arithmetic inconsistency in the cost break-even. I would be comfortable with major revision; if the authors provide a corrected break-even and either stronger statistics or carefully qualified claims, the paper could become acceptable."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nThe paper evaluates two self-refinement methods—error-based prompt rewriting and self-correction—for LLM-based product attribute value extraction, using GPT-4o on OA-Mine and AE-110K. The main finding: neither method improves F1 meaningfully while token usage jumps. There's also a break-even analysis for fine-tuning at ~6,666 offers.\n\nWhat's genuinely useful: the detailed error analysis. For rewriting, they show 93% of rewrites add detail, and that this leads to overfitting to development data. For self-correction, they quantify corruption vs improvement: corruptions outnumber improvements, so the net change is near zero. This goes beyond simple 'it doesn't work' and gives a mechanism. Code and data are public.\n\nThe soft spot is the statistics. The central 'insignificant' claim is based on paired t-tests with three runs. The stress-test is right: with df=2 and 99% confidence, the critical t is ~9.9, so only a difference of about six standard deviations would register. The observed deltas (+0.4 F1, -0.3, +0.2) are nowhere near that. So 'insignificant' really means 'not detectable at n=3,' not 'no effect.' No p-values, effect sizes, or confidence intervals are reported, and F1 is aggregated over the whole test set.\n\nThat said, the practical conclusion doesn't collapse. The token cost increases are large and unambiguous, and the corruption/improvement analysis independently supports the idea that self-correction isn't reliably helpful. A larger run count might reveal a small effect, but it wouldn't make the approach cost-effective. So this is a moderate statistical flaw, not a fatal one.\n\nOther limitations: single model (GPT-4o 2024-08-06), subsets of two datasets, unseeded random sampling for rewriting. The authors acknowledge some of this. The paper would be stronger with more runs, confidence intervals, and perhaps a second model.\n\nWho is this for? E-commerce practitioners considering self-refinement. A researcher benchmarking LLM extraction might also use the break-even and error-analysis findings.\n\nRecommendation: send to peer review. It's a legitimate empirical contribution with public code and a clear negative result. A referee should push for better statistics, but the paper is informative as-is.","headline":"Useful negative result on self-refinement for product attribute extraction, but the 'insignificant' claim is underpowered.","tokens_in":11060,"tokens_out":2462,"would_cite":true,"duration_ms":22542,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper argues that two self-refinement strategies, error-based prompt rewriting and self-correction, fail to significantly improve LLM-based product attribute value extraction while raising token costs.","keywords":["product attribute value extraction","self-refinement","error-based prompt rewriting","self-correction","large language models","in-context learning","fine-tuning","token cost efficiency"],"falsifier":"Rerun the comparison with many more repetitions (say 20 per condition) and compute the full distribution of F1 differences; if self-correction or error-based rewriting shows a consistently positive and statistically significant F1 margin on the same test splits, the paper's central claim is wrong.","tokens_in":10156,"feed_emoji":"🏷️","tokens_out":5807,"duration_ms":52599,"temperature":0.7,"pith_summary":"The paper asks whether self-refinement techniques that help in code generation and text-to-SQL also help when a large language model extracts structured attribute values (brand, dosage, net content) from free-form product descriptions. Using GPT-4o across zero-shot, few-shot, and fine-tuned settings, it finds that both error-based prompt rewriting and self-correction fail to produce statistically significant F1 gains while roughly doubling to tripling token usage. The useful improvements come from simpler ingredients: attribute definitions and few-shot demonstrations, with fine-tuning giving the best accuracy. The authors calculate that fine-tuning's upfront cost pays off once a pipeline processes more than about 6,666 product offers.","feed_headline":"Self-refinement does not improve product attribute extraction","feed_subtitle":"Error-based rewriting and self-correction raise token costs without lifting accuracy; fine-tuning wins above 6,666 offers.","key_machinery":"The argument is carried by two automated self-refinement loops. Error-based prompt rewriting runs the prompt on a few development offers, collects wrong extractions, and asks the model to rewrite the attribute definition so it would have been correct. Self-correction runs the extraction prompt, then sends the first output back to the same model with a request to check and update it. The authors measure both with F1 (categorizing predictions into matches, misses, and wrong values) and with a token factor expressing how many times more tokens the refined pipeline consumes than the zero-shot baseline.","core_discovery":"The central claim is that for product attribute value extraction with GPT-4o, self-refinement is not worth its cost. Error-based prompt rewriting makes attribute definitions longer and more detailed, and in 93% of cases GPT-4o adds detail, but that extra detail overfits the development set and the test performance does not improve. Self-correction changes between 106 and 423 extracted values depending on scenario, yet corruptions occur more often than improvements, so F1 moves by at most a few tenths of a point. The only scenario where a second model pass makes sense is none of the tested ones: fine-tuning without any refinement reaches the highest F1 (83.2 on OA-Mine, 85.1 on AE-110k) at the lowest per-offer token cost.","pith_inferences":["Not tested here: the break-even point of 6,666 offers depends on token prices and model choice; with cheaper inference or more expensive fine-tuning, the threshold would shift, and the paper gives no sensitivity analysis.","A plausible mechanism behind the null result is that self-correction has no external signal to anchor to; in tasks with verifiable feedback (a code interpreter, a database error), the same technique might still pay off, but extraction from text offers no such check.","A testable variant of error-based rewriting would cap the length or number of rewrite iterations; since 97% of rewritten definitions get longer and overfit, constraining length might preserve the F1 gains of definitions while cutting tokens."],"forward_implications":["Fine-tuning without self-refinement is the strongest configuration: highest F1 and lowest runtime token use, so production pipelines that process many offers should prefer it.","Self-refinement costs scale steeply: error-based rewriting raises the token factor to 8.6-13.9 and self-correction to 2.3-19.8 depending on scenario, with no significant F1 improvement.","Attribute definitions are a cheap and effective lever: on AE-110k they lift F1 by 12.7 points over zero-shot at a token factor below 3.","The economic decision has a concrete threshold: fine-tuning pays off once a catalog exceeds roughly 6,666 product offers, because few-shot prompts consume far more tokens per offer.","Because self-correction's corruptions outnumber improvements, adding a second pass is likely to hurt precision-sensitive applications rather than help them."],"supporting_citations":[{"why":"Supplies the few-shot extraction prompting setup and the 60:40 development/test split used for all experiments.","marker":"[BSB25]"},{"why":"Provides the OA-Mine dataset with human-annotated product offers and attributes used as one benchmark.","marker":"[Zh22]"},{"why":"Provides the AE-110k dataset and the five-way prediction categorization from which F1 is computed.","marker":"[Xu19]"},{"why":"Establishes that attribute definitions improve LLM classification, the premise behind error-based prompt rewriting.","marker":"[Pe23]"},{"why":"Introduces the iterative self-refinement with self-feedback pattern that the self-correction implementation follows.","marker":"[Ma23]"},{"why":"Documents that LLMs often fail to self-correct without external feedback, the critical claim this paper's results support.","marker":"[Hu23]"},{"why":"Shows self-repair can fail for code generation, motivating the skeptical evaluation of self-refinement for extraction.","marker":"[Ol23]"}],"fun_headline_variants":["Self-refinement fails to boost product attribute extraction","Self-refinement raises costs, not accuracy, for attribute extraction","Fine-tuning outperforms self-refinement for product attributes","Self-refinement wastes tokens without improving extraction"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The conclusion rests on the assumption that three runs with one model (GPT-4o) on two dataset subsets are enough to detect any real performance difference; with so few repetitions, small gains could be hiding in the noise.","fun_headline_variants_meta":{"raw":{"variants":["Self-refinement fails to boost product attribute extraction","Self-refinement raises costs, not accuracy, for attribute extraction","Fine-tuning outperforms self-refinement for product attributes","Self-refinement wastes tokens without improving extraction"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000441,"raw_usage":{"total_tokens":2221,"prompt_tokens":915,"completion_tokens":1306,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":531,"completion_tokens_details":{"reasoning_tokens":1244}},"tokens_in":531,"tokens_out":1306,"duration_ms":10409,"temperature":1.0,"reasoning_tokens":1244,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T22:31:45.872887+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Rerun the comparison with many more repetitions (say 20 per condition) and compute the full distribution of F1 differences; if self-correction or error-based rewriting shows a consistently positive and statistically significant F1 margin on the same test splits, the paper's central claim is wrong.","supporting_citations":[],"review_version":1}