AQuilt: Weaving Logic and Self-Inspection into Low-Cost, High-Relevance Data Synthesis for Specialist LLMs

Hexuan Deng; Jun Rao; Jun Yu; Min Zhang; Xiaopeng Ke; Xuebo Liu; Zhenxi Song

REVIEW 4 major objections 7 minor 15 references

AQuilt: Weaving Logic and Self-Inspection into Low-Cost, High-Relevance Data Synthesis for Specialist LLMs

T0 review · 4 major / 7 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read This paper claims that a 7B data-synthesis model trained on 703k distilled examples can generate specialist instruction data that matches DeepSeek-V3 distillation at about 17% of the cost.

desk verdict AQuilt is a solid, incremental extension of the Bonito/Self-Instruct line with genuine ablations; the 17% cost claim is plausible but currently unauditable because the paper never itemizes the one-time training and distillation costs. read the letter →

arxiv 2507.18584 v1 pith:ZZ2YLUHI submitted 2025-07-24 cs.CL cs.AI

Xiaopeng Ke , Hexuan Deng , Xuebo Liu , Jun Rao , Zhenxi Song , Jun Yu , Min Zhang This is my paper · ORCID

classification cs.CLcs.AI

keywords datasynthesisinstructiontuningspecialistLLMsself-inspectionlogicreasoningunlabeleddomainadaptationlow-costtraining

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper proposes AQuilt, a 7B-parameter data-synthesis model that converts unlabeled domain text into instruction-tuning data for specialist LLMs. It is trained on 703k bilingual examples distilled from DeepSeek-V3, each carrying the question, answer, source text, a logic chain, and a self-inspection score, along with a task type. The central claim is that this smaller model produces data good enough that fine-tuned Qwen2.5-7B and Llama3-8B match DeepSeek-V3 distillation on five specialist tasks while spending about 17% of the production cost. The paper also argues that the logic and inspection components, plus a relevance filter, are what make the synthetic data more concentrated and less noisy than direct teacher output.

What carries the argument

The load-bearing object is the AQuilt quintuplet: unlabeled text, task type, question, logic, and answer, augmented by an inspection score. The framework first distills these quintuplets from DeepSeek-V3 over 33 bilingual source corpora, filters for relevance by removing questions that would be unanswerable without the source text and trimming stylistically overused words, and trains a 7B synthesizer. A LoRA-based self-inspection head then scores the synthesizer's own outputs, and low scores are filtered out during deployment. Logic forces the model to produce intermediate reasoning, inspection gives it a quality signal, and the task type, especially the open-book and closed-book QA catch-all, is what lets one synthesizer serve unseen tasks.

What would settle it

A reader could falsify the parity claim by running the same recipe on a held-out specialist domain not represented in the 33 training corpora, generating 20k samples with AQuilt and with DeepSeek-V3, fine-tuning both base models identically, and testing on an expert-annotated set; if AQuilt's average underperforms by a margin larger than the 0.02-point gap seen here on any task, the generality claim fails.

Watch

Extended reading notes

Core claim

AQuilt's central discovery is that data synthesis for specialist LLMs can be distilled into a small model that weaves six ingredients into each generated example: Answer, Question, Unlabeled data, Inspection, Logic, and Task type. Using unlabeled domain corpora as anchors, the model produces question-answer pairs with an explicit reasoning chain and a self-scored quality judgment, and it generalizes to unseen tasks by mapping them to open-book or closed-book QA with the task instruction prefixed to the question. Trained with LoRA on Qwen2.5-7B-Base for two epochs on 703k distilled samples, AQuilt generates 20k training examples per downstream task; fine-tuning Qwen2.5-7B-Instruct and Llama3-8B-Instruct on that data yields average scores of 51.45 and 46.68, compared with 51.47 and 46.01 for the DeepSeek-V3 Self-Instruct plus Unlabeled-Data baseline, at roughly one-sixth the cost.

Load-bearing premise

The claim that AQuilt is comparable to DeepSeek-V3 rests on the assumption that 703k distilled examples, the downsampling scheme, and the chosen hyperparameters are enough to transfer the synthesis skill to a 7B model, and that five evaluation tasks are representative enough that average scores of 51.45 versus 51.47 are not noise.

Editorial extensions

If this is right

Specialist LLM fine-tuning can be decoupled from large teacher APIs: a fixed 7B synthesizer produces the training data, so repeated domain adaptation costs roughly 17% of calling DeepSeek-V3 each time.
Removing either logic or self-inspection lowers downstream scores, from 46.18 to 44.64 without logic and to 45.06 without self-inspection on the Llama3-8B subset, attributing the quality gain to those components rather than to model scale.
Framing unseen tasks as open-book or closed-book QA with prefixed instructions lets a single synthesizer cover tasks it never saw during training, such as legal translation and legal essay writing.
Relevance-aware filtering cuts the fraction of synthetic questions that depend on the unlabeled source text from 5.23% for DeepSeek-V3 to 1.85% for AQuilt, which the paper links to fewer spurious correlations and better downstream adaptation.
AQuilt also beats a 72B instruction model under the same unlabeled-data setup while using about one-twentieth of the GPU hours, suggesting the gains come from the synthesis recipe rather than base-model size.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

A testable extension is to compute the silhouette score of AQuilt's synthetic question embeddings on a new domain and check whether it predicts downstream accuracy across domains; the paper reports the score but does not establish that predictive correlation.
The self-inspection score could be reused as a reward for reinforcement-learning fine-tuning of the synthesizer itself, turning a filtering step into a training signal that the paper only uses to select data.
If the distillation recipe transfers to successor teacher models, the same 7B synthesizer architecture could be refreshed with data from a stronger teacher without changing deployment cost, keeping the 17% cost ratio even as teacher quality rises.
The bilingual English-Chinese training mix and the open-book/closed-book abstraction suggest the synthesizer may generalize to mid-resource languages, but the paper explicitly leaves low-resource language performance untested.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Take this as a legitimate, incremental step on the Bonito/Self-Instruct line, not a breakthrough. The new pieces are real: training a 7B synthesis model on 703k DeepSeek-V3-distilled quintuplets with (a) an explicit logic field that forces a reasoning chain, (b) a learned self-inspection scorer that filters low-quality generations, (c) open/closed-book QA task types with question-prefix adaptation that lets the model handle unseen tasks, and (d) bilingual EN/ZH coverage. The ablations in Table 3 are informative: dropping logic or self-inspection costs 1-3.5 points on average, and the low-quality selection experiment (w/ Low-Quality) supports the filtering story. The relevance analysis in Table 4 is a nice, honest diagnostic: DeepSeek-V3 itself generates spurious correlations between questions and unlabeled context, and AQuilt's relevance-aware filtering reduces that. The GPT-4o scoring in Table 6 is a reasonable additional sanity check. The GitHub release plan (code, models, training data) is exactly the right move for this area.

The soft spots are real but fixable. The biggest is the 17% cost claim. Table 2's 'Cost' column is not itemized. The footnote says 'total expense for data synthesis and training,' but the AQuilt rows appear to cover only the 20k-sample synthesis and the specialist fine-tune; the one-time 703k distillation (DeepSeek-V3 API calls), the AQuilt LoRA training, and the inspection-model training are not line items. Maybe the authors mean 'production cost' for a new domain excluding one-time investments, and that can be a defensible amortization choice, but the paper never says so. The stress-test arithmetic on $1.48 for SquadQA is actually too pessimistic (vLLM on a 4090 is much faster than 100 tokens/s), but the non-auditability stands. Also, no seeds, no standard deviations, and the headline comparison is 51.45 vs 51.47 on Qwen; that could easily be noise. The inspection threshold (remove score <=2, unless >20% are 2, then remove only 1) is post hoc and not swept. And the cross-task generalization claim is partially in-distribution: SquadQA unlabeled data comes from the test set, and the training data includes SQuAD. That is a legitimate adaptation setting, but it weakens the 'unseen task' story.

On balance: worth a serious referee. The method is coherent, the ablations are meaningful, and the evaluation, while incomplete, is not misleading. The reviewer's job should be to demand itemized costs, error bars, an inspection-threshold sweep, and a clearer statement of what is amortized. I'd take it.

Referee Report

4 major / 7 minor

Summary. The paper proposes AQuilt, a 7B data synthesis model that generates instruction-tuning quintuplets (answer, question, unlabeled data, logic, task type) from domain unlabeled data, together with a self-inspection scoring LoRA trained on DeepSeek-V3 ratings of AQuilt's own outputs. The synthesis model is trained on 703k DeepSeek-V3-distilled examples covering 10 task types in Chinese and English. In downstream evaluation, the authors fine-tune Qwen2.5-7B-Instruct and Llama3-8B-Instruct on synthetic data from AQuilt and several baselines across five tasks (SquadQA, PubMedQA, C-Eval, legal Translation, and Legal EssayQA). The central claim is that AQuilt matches DeepSeek-V3 with Self-Instruct plus unlabeled data (SI+UD) while using about 17% of the production cost; ablations attribute part of the gain to logic and self-inspection, and a relevance analysis is used to argue that AQuilt's data is more concentrated and less noisy.

Significance. If the performance and cost claims hold, this is a practically useful contribution: it shows that a relatively small, openly releasable model can distill a much larger model's data-synthesis ability and apply it to new tasks and domains at lower inference cost. The paper's strengths are the breadth of the evaluation (two base models, five tasks, bilingual data), the internal consistency of the main result tables, the inclusion of ablations for logic and self-inspection, and the planned release of code, models, and data. The primary weaknesses are that the headline 17% cost figure is not auditable from the information provided, and the absence of repeated runs or variance estimates leaves the 'comparable to DeepSeek-V3' claim unsupported at the reported level of precision.

major comments (4)

[§4.3, Table 2] The headline claim that AQuilt achieves DeepSeek-V3-comparable performance at 17% of the production cost is not verifiable from the paper. The Cost column is labeled 'total expense for data synthesis and training,' but no itemization is given for (i) the DeepSeek-V3 API calls used to build the 703k training set in §3.2, (ii) the AQuilt base training described in §4.1, or (iii) the inspection-model LoRA training in §3.3. Without these one-time costs, the reported ratio may compare only per-task AQuilt synthesis plus specialist fine-tuning against per-task DeepSeek-V3 synthesis, which would not support the abstract's claim. Please provide a full cost model with API costs, GPU-hours, rental rates, and the amortization of the distillation and training stages, and reconcile how generating 20k SquadQA samples with a 7B model on rented 4090s plus fine-tuning can cost $1.48.
[§4.2, Tables 2 and 3] No standard deviations, confidence intervals, or number of seeds are reported for any experiment. The 'comparable to DeepSeek-V3' claim rests on average differences of 0.02 points on Qwen (51.45 vs. 51.47) and 0.67 points on Llama (46.68 vs. 46.01), which are likely within run-to-run noise at this scale. The ablation differences in Table 3 (about 1–3.5 points) are likewise reported as single numbers. Please rerun the main comparisons and ablations with at least three seeds and report mean and standard deviation, or explicitly state if only a single run was performed.
[§4.2, Domain Data Generation] The paper states that for SquadQA 'AQuilt requires domain-specific unlabeled data, which we source as follows: SquadQA uses test set data (Hu et al., 2023).' If the passages used to synthesize training data come from the same SQuAD split that is later used for evaluation, this is a potential train/test overlap that could inflate the reported gains. Please clarify precisely what 'test set data' means in this online-adaptation setting, confirm that no test questions or answer strings were used in synthesis, and explain how the setup follows the cited prior protocol.
[§3.3, §3.4] The self-inspection scorer is trained on DeepSeek-V3 scores assigned to AQuilt-generated data and is then applied to filter subsequently generated AQuilt data, yet no independent validation is provided that the 5-point scores are calibrated or that the default filtering threshold of 2 is robust. The free parameters, including the threshold, the 20% rule for simpler tasks, and the downsampling caps, are not swept. Since the self-inspection component is a core part of the framework and the ablation in Table 3 is a single run, a sensitivity analysis over the threshold would substantially strengthen the claim that the filter, rather than the fixed data volume, drives the improvement.

minor comments (7)

[§4.2] The text contains an unresolved reference: 'All other settings are consistent with those in Section ??.' Please replace this with the correct section number.
[Table 2, §3.2] There are typographical errors: 'DeepSeeek-V3' in Table 2 and 'Relevence-Aware Data Filtering' in the §3.2 heading; both should be corrected.
[§5.2] The relevance analysis reports that Silhouette Scores were computed and 'align with the scatter plot,' but the numeric Silhouette values are not given in the text or the figure; please report them.
[Appendix A] The text says 33 unlabeled datasets are collected, but the enumerated list in Appendix A appears to contain 32 entries; please verify the count.
[Table 5] The cost column in Table 5 is expressed as multiplicative ratios (19.92× and 1×) rather than dollar amounts, which is inconsistent with Table 2 and makes the comparison difficult to interpret; please use a consistent cost basis.
[§4.1, Table 2] §4.1 says the synthesis model is trained from Qwen2.5-7B-Base, while Table 2 states that all base models used for downstream training are instruction versions; please clarify which version underlies AQuilt and whether this distinction matters for the reported results.
[§4.2, Table 2] The notation 'CEV AL' is nonstandard; the underlying benchmark is C-Eval, and the paper should use a consistent name. Similarly, 'EassyQA' in the Appendix C prompt should be 'EssayQA.'

Circularity Check

1 steps flagged · score 4.0 of 10

Main cost/performance claim is empirical and non-circular; one supporting relevance analysis is self-definitional.

self definitional [§3.2 'Relevence-Aware Data Filtering' and §5.3 / Table 4]
"To achieve this, we explicitly guide the model's preferences through prompt engineering. We also filter out cases not meeting this criterion by identifying prohibited words, such as 'the context' and 'the text'... To analyze the underlying cause, we examine the proportion of generated questions that rely on unlabeled data to generate answers... We sample 2,000 instances from the generated dataset and use GPT-4o to evaluate."

The independence metric reported in Table 4 is the same criterion enforced during AQuilt construction. AQuilt's training prompt explicitly forbids phrases like 'according to the text' and its filter removes questions containing 'the context'/'the text'; the GPT-4o evaluator then classifies a question as dependent on unlabeled data precisely when it contains such keywords or asks about the content of a text. AQuilt's near-zero dependence (0.40% vs DeepSeek-V3's 6.90%) is therefore largely a direct consequence of its own prompt and filtering, not an independent discovery about data relevance.

full rationale

The paper's headline claim—AQuilt is comparable to DeepSeek-V3 at 17% of production cost—is an empirical comparison on external benchmarks (SquadQA, PubMedQA, CEVAL, LexEval), using the same fine-tuning protocol for all synthesis sources; it does not reduce by construction and is not supported by a self-citation chain. The self-inspection scorer is distilled from DeepSeek-V3 quality labels, which is a supervised distillation rather than a circular derivation. The one genuine circular step is the relevance/independence analysis in §5.3/Table 4: the metric operationalizes 'dependence on unlabeled data' with the same keyword and self-containment criteria that were used to filter and train AQuilt, so the observed advantage is partly self-definitional rather than independently measured. The 17% cost figure is not itemized and is hard to audit, but missing cost components are an accounting concern, not a circularity. Overall: central claim independent, one supporting analysis reduces to its own construction, hence score 4.

Assumptions & free parameters 3 free parameters · 5 assumptions · 0 invented entities

AQuilt does not introduce new physical or mathematical entities; its additional structure is a data-augmentation recipe (logic, inspection, task type). The free parameters are the filter thresholds and downsampling caps, plus task-specific training tweaks. The key domain assumptions are the reliability of DeepSeek-V3 as a teacher, the diversity of the 33-dataset corpus, and the significance of the five evaluation tasks. None of these are validated beyond the paper's own experiments.

free parameters (3)

Inspection score filtering threshold (default 2 on 5-point scale) = 2 (with exception: if >20% of data scores 2, only score 1 removed)
Chosen by hand in Section 3.4, no ablation varying the threshold; the main results depend on this threshold. The exception rule is a post-hoc condition.
Downsampling caps (50k per task/language; 2k per inspection score) = 50k and 2k
Chosen to balance the dataset; affects the 703k total size and the model's capability mix, but not ablated.
Per-task learning rate / single-epoch special cases (CEVAL Qwen LR 1e-7, TAPT single epoch) = 1e-7
For CEVAL on Qwen and TAPT, the authors use a lower LR and single epoch because they say Qwen overfits; this is a hand-fitted training recipe that contributes to the main results and is not a general setting.

assumptions (5)

domain assumption DeepSeek-V3 generated logic chains and answers are high quality and useful as training targets for the smaller student.
Central to Section 3.2; the entire student capability relies on the teacher's output quality. The paper does not measure teacher accuracy on the generation task.
domain assumption The 703k distilled dataset is diverse enough that the 7B model generalizes to unseen tasks through the closed/open-book classification plus question prefix.
Assumed throughout Section 3.4 and Section 4.3; the paper only tests two actually unseen task types (Translation, EssayQA) and one unseen language pair setup, so the generalization claim is evidence-limited.
standard math LoRA fine-tuning of the base model preserves sufficient general knowledge and instruction-following capability.
Assumed in Sections 3.3 and 4.2; this is a routine assumption for all LoRA experiments in the paper, not unique to AQuilt.
domain assumption Sentence embedding based t-SNE and silhouette scores reflect 'relevance' to downstream tasks.
Used in Section 5.2; concentration in embedding space is treated as a proxy for task relevance, but no downstream task correlation is computed.
domain assumption GPT-4o quality scores and independence judgments are reliable.
Used in Sections 5.3 and 5.5 for independence analysis and data quality evaluation; no agreement analysis with human judges is reported.

how reviews work

0 comments

Cite this review

Pith. "Pith review of AQuilt: Weaving Logic and Self-Inspection into Low-Cost, High-Relevance Data Synthesis for Specialist LLMs." pith.science (2026). https://pith.science/paper/ZZ2YLUHI

@misc{pith2026250718584,
  author       = {Pith},
  title        = {Pith review of: AQuilt: Weaving Logic and Self-Inspection into Low-Cost, High-Relevance Data Synthesis for Specialist LLMs},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/ZZ2YLUHI}},
  note         = {Machine review of arXiv:2507.18584}
}

read the original abstract

Despite the impressive performance of large language models (LLMs) in general domains, they often underperform in specialized domains. Existing approaches typically rely on data synthesis methods and yield promising results by using unlabeled data to capture domain-specific features. However, these methods either incur high computational costs or suffer from performance limitations, while also demonstrating insufficient generalization across different tasks. To address these challenges, we propose AQuilt, a framework for constructing instruction-tuning data for any specialized domains from corresponding unlabeled data, including Answer, Question, Unlabeled data, Inspection, Logic, and Task type. By incorporating logic and inspection, we encourage reasoning processes and self-inspection to enhance model performance. Moreover, customizable task instructions enable high-quality data generation for any task. As a result, we construct a dataset of 703k examples to train a powerful data synthesis model. Experiments show that AQuilt is comparable to DeepSeek-V3 while utilizing just 17% of the production cost. Further analysis demonstrates that our generated data exhibits higher relevance to downstream tasks. Source code, models, and scripts are available at https://github.com/Krueske/AQuilt.

Figures

Figures reproduced from arXiv: 2507.18584 by the authors.

**Figure 2.** Overview of the proposed AQuilt framework. The left side illustrates the training process of our data [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗

**Figure 3.** Overview of the sources of the unlabeled data [PITH_FULL_IMAGE:figures/full_fig_p003_3.png] view at source ↗

Figures from the paper (1 more)

**Figure 4.** Figure 4: Relevance analysis of synthesized domain [PITH_FULL_IMAGE:figures/full_fig_p008_4.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

15 extracted references · 10 canonical work pages

[1]

Requirement 2: The thought process should include the following steps: (1)

The thought process you create should meet the following requirements: Requirement 1: The generated thought process should be rich in content and logically clear, demonstrating how to infer the <Answer> from the <Question>. Requirement 2: The thought process should include the following steps: (1). Read the question: Understand the provided question. (2)....

work page
[2]

Brainstorm content: Develop key ideas or themes consistent with the input conditions. 3. Generate output: Create a well-structured and coherent response that follows the user’s instructions. Text Summarization : Task Type: text summarization Question Requirement: The text summarization problem should have a clear intent and be semantically clear, includin...

work page
[3]

Please carefully study the above requirements and then create a thought process based on the following <text>, <question>, and <answer> provided by the user: 20 Prompts for Synthetic Inspection Training Dataset (1/2) You are an AI instruction and response quality assessment assistant, please score the quality of the user’s instruction and response accordi...

work page
[6]

wanchuang cup

OpenAI o1 System Card. arXiv preprint arXiv:2412.16720. Su Jianlin. 2017. Baidu’s chinese question-answering dataset webqa. Qiao Jin, Bhuwan Dhingra, Zhengping Liu, William Cohen, and Xinghua Lu. 2019. PubMedQA: A dataset for biomedical research question answering. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing a...

arXiv 2017
[7]

Transactions of the Association for Computational Linguistics, 6:287– 302

Constructing datasets for multi-hop reading comprehension across documents. Transactions of the Association for Computational Linguistics, 6:287– 302. Yuxi Xie, Anirudh Goyal, Wenyue Zheng, Min-Yen Kan, Timothy P Lillicrap, Kenji Kawaguchi, and Michael Shieh. 2024. Monte carlo tree search boosts reasoning via iterative preference learning. In The First Wo...

arXiv 2024
[8]

arXiv preprint arXiv:2309.17421

The Dawn of LMMs: Preliminary Ex- plorations with GPT-4V(ision). arXiv preprint arXiv:2309.17421. Eric Zelikman, Yuhuai Wu, Jesse Mu, and Noah D. Goodman. 2022. Star: Bootstrapping reasoning with reasoning. In Advances in Neural Information Pro- cessing Systems 35: Annual Conference on Neural Information Processing Systems 2022, NeurIPS 2022, New Orleans,...

arXiv 2022
[11]

thought_process

Please generate the thought process in the following format: JSON { “thought_process”: “{{The thought process you created}}” }

work page
[13]

- The response contains noticeable grammatical errors, spelling mistakes, or awkward phrasing

The scoring criteria are as follows: 1 point - Low quality, minimal requirements met (Low Level): - The response is only partially relevant to the question and lacks depth or detail. - The response contains noticeable grammatical errors, spelling mistakes, or awkward phrasing. - The response fails to address the user’s questions or needs adequately. - The...

work page

Show all 15 references

[14]

Please analysis the quality in the following format: JSON { analysis_steps: {{your analysis for the quality}}, score: {{your rate to the qa_pair}} }
[15]

question

Please carefully study the above scoring criteria and strictly follow the scoring criteria above to score the following <qa_pair> based on the following <text> provided by the user: Different Task Prompts for AQuilt (1/5) single choice question answering : “““Please generate a...
[2018]

In Proceedings of the Workshop on Machine Reading for Question Answering, pages 37–46, Melbourne, Australia

DuReader: a Chinese machine reading com- prehension dataset from real-world applications. In Proceedings of the Workshop on Machine Reading for Question Answering, pages 37–46, Melbourne, Australia. Association for Computational Linguistics. Arian Hosseini, Xingdi Yuan, Nikola...
[2019]

A span-extraction dataset for Chinese ma- chine reading comprehension. In Proceedings of the 2019 Conference on Empirical Methods in Natu- ral Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 5883–5889, Hong Ko...

2019
[2021]

William L

2021 hai hua ai competition. William L. Hamilton, Kevin Clark, Jure Leskovec, and Dan Jurafsky. 2016. Inducing domain-specific senti- ment lexicons from unlabeled corpora. In Proceed- ings of the 2016 Conference on Empirical Methods in Natural Language Processing , pages 595–6...

2021 arXiv
[2023]

Proceedings of the National Academy of Sciences, 120(30):e2305016120

ChatGPT Outperforms Crowd-Workers for Text-Annotation Tasks. Proceedings of the National Academy of Sciences, 120(30):e2305016120. Giovanni Grano, Andrea Di Sorbo, Francesco Mercaldo, Corrado A Visaggio, Gerardo Canfora, and Sebas- tiano Panichella. 2017. Android apps and user...

2017 arXiv
[2024]

In First Conference on Language Modeling

V-STar: Training verifiers for self-taught rea- soners. In First Conference on Language Modeling. Baotian Hu, Qingcai Chen, and Fangze Zhu. 2015. LC- STS: A large scale Chinese short text summarization dataset. In Proceedings of the 2015 Conference on Empirical Methods in Natu...

2015

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.

[1] [1]

Requirement 2: The thought process should include the following steps: (1)

The thought process you create should meet the following requirements: Requirement 1: The generated thought process should be rich in content and logically clear, demonstrating how to infer the <Answer> from the <Question>. Requirement 2: The thought process should include the following steps: (1). Read the question: Understand the provided question. (2)....

work page

[2] [2]

Brainstorm content: Develop key ideas or themes consistent with the input conditions. 3. Generate output: Create a well-structured and coherent response that follows the user’s instructions. Text Summarization : Task Type: text summarization Question Requirement: The text summarization problem should have a clear intent and be semantically clear, includin...

work page

[3] [3]

Please carefully study the above requirements and then create a thought process based on the following <text>, <question>, and <answer> provided by the user: 20 Prompts for Synthetic Inspection Training Dataset (1/2) You are an AI instruction and response quality assessment assistant, please score the quality of the user’s instruction and response accordi...

work page

[4] [6]

wanchuang cup

OpenAI o1 System Card. arXiv preprint arXiv:2412.16720. Su Jianlin. 2017. Baidu’s chinese question-answering dataset webqa. Qiao Jin, Bhuwan Dhingra, Zhengping Liu, William Cohen, and Xinghua Lu. 2019. PubMedQA: A dataset for biomedical research question answering. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing a...

arXiv 2017

[5] [7]

Transactions of the Association for Computational Linguistics, 6:287– 302

Constructing datasets for multi-hop reading comprehension across documents. Transactions of the Association for Computational Linguistics, 6:287– 302. Yuxi Xie, Anirudh Goyal, Wenyue Zheng, Min-Yen Kan, Timothy P Lillicrap, Kenji Kawaguchi, and Michael Shieh. 2024. Monte carlo tree search boosts reasoning via iterative preference learning. In The First Wo...

arXiv 2024

[6] [8]

arXiv preprint arXiv:2309.17421

The Dawn of LMMs: Preliminary Ex- plorations with GPT-4V(ision). arXiv preprint arXiv:2309.17421. Eric Zelikman, Yuhuai Wu, Jesse Mu, and Noah D. Goodman. 2022. Star: Bootstrapping reasoning with reasoning. In Advances in Neural Information Pro- cessing Systems 35: Annual Conference on Neural Information Processing Systems 2022, NeurIPS 2022, New Orleans,...

arXiv 2022

[7] [11]

thought_process

Please generate the thought process in the following format: JSON { “thought_process”: “{{The thought process you created}}” }

work page

[8] [13]

- The response contains noticeable grammatical errors, spelling mistakes, or awkward phrasing

The scoring criteria are as follows: 1 point - Low quality, minimal requirements met (Low Level): - The response is only partially relevant to the question and lacks depth or detail. - The response contains noticeable grammatical errors, spelling mistakes, or awkward phrasing. - The response fails to address the user’s questions or needs adequately. - The...

work page