Pith. sign in

REVIEW 2 major objections 6 minor 85 references

Beyond In-Context Learning: Aligning Long-form Generation of Large Language Models via Task-Inherent Attribute Guidelines

T0 review · 2 major / 6 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read In-context learning demonstrations alone cannot teach LLMs the format and language of long-form outputs; LongGuide's task-derived guidelines improve both open- and closed-source models by over 5%.

desk verdict LongGuide is a practical, well-tested pipeline for improving long-form generation, but the automatic-selection and 'over 5%' claims need honest qualification; it deserves serious refereeing. read the letter →

arxiv 2506.01265 v1 pith:NYHMFZN2 submitted 2025-06-02 cs.CL

Do Xuan Long , Duong Ngoc Yen , Do Xuan Trong , Luu Anh Tuan , Kenji Kawaguchi , Shafiq Joty , Min-Yen Kan , Nancy F. Chen This is my paper
classification cs.CL
keywords long-formgenerationin-contextlearningpromptguidelinesLLMalignmenttextpropertytransferoutputconstraintsself-evaluationzero-shotandfew-shotprompting
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper argues that few-shot examples—in-context learning demonstrations—cannot by themselves teach a large language model the length, format, and phrasing its long-form answers should have, because a model that does not already know the task's distribution will not reliably copy those properties from examples. To fix this, it proposes LongGuide, which reads a small set of training outputs, extracts the task's characteristic metrics (like brevity or clarity) and its sentence and token statistics, and writes these into explicit textual guidelines. The model is then prompted with the best of four combinations of these guidelines, chosen automatically. The authors report that this lifts performance by over 5% on average across seven long-form generation tasks, in both zero- and few-shot settings, and that it works for open- and closed-source models alike. If correct, this shows that explicit, sample-derived instructions are a cheap and effective complement to demonstrations for aligning LLM outputs.

What carries the argument

LongGuide is a guideline-generation algorithm that turns at most 50 training samples into two parallel instruction streams: Metric Guidelines (MG), natural-language descriptions of the expected quality on self-evaluated metrics selected from a pool of 27, and Output Constraint Guidelines (OCG), a single sentence specifying the minimum, maximum, and average number of sentences and tokens observed in the training outputs. The load-bearing mechanism is the automatic selection step (Step 5), which evaluates the model with MG only, OCG only, both, or neither on the training set and picks whichever maximizes ROUGE-L, so the method can adapt to a model's existing strengths and weaknesses.

What would settle it

Scale the Section 2 property-maintenance experiment to large demonstration sets: choose one long-form task, fix demonstrations whose outputs share a single sentence count and token count, and measure what fraction of few-shot responses match those counts with 3, 10, 50, and 200 demonstrations; if the fraction approaches 100%, the claim that demonstrations cannot transfer format properties is false. Alternatively, construct a task with bimodal output lengths; LongGuide's selection step should choose no guideline there, and if the fixed min/max/average window still wins, the average-statistics assumption fails.

Watch

Extended reading notes

Core claim

The paper's central claim is that ICL demonstrations are insufficient for long-form generation because of what it calls the text property transfer (PT) problem: when the language model's probability distribution over text differs from the task's true distribution, demonstrations cannot make the model consistently preserve the task's language and format properties, such as sentence count, token count, and informativeness. It proves a related remark and supports it with experiments showing that even when every demonstration shares the same token count and sentence count, fewer than 5% of generated responses match those counts. The proposed remedy, LongGuide, derives two kinds of guidelines from at most 50 training examples: Metric Guidelines (MG), which tell the model to optimize self-evaluated quality metrics, and Output Constraint Guidelines (OCG), which impose minimum, maximum, and average sentence and token counts. The best of four guideline configurations is chosen by measuring ROUGE-L on the training set. The paper reports that this raises performance by over 5% on average across seven long-form generation tasks for both ChatGPT and Mistral, in both zero- and few-shot settings.

Load-bearing premise

The load-bearing premise is that a 50-sample training set faithfully represents the output length and style distribution of the whole task, so guidelines tuned on those samples will help at test time.

Editorial extensions

If this is right

  • LongGuide works even when its training set is just the 3–5 demonstrations used for ICL, so the method applies to tasks that lack a dedicated training split.
  • Guidelines learned by a weaker open-source model transfer to a stronger closed-source model, allowing cheap guideline discovery for expensive proprietary models.
  • LongGuide combines with automatic prompt optimizers: further optimizing the learned guidelines improves performance, especially when the detailed MG-OCG guideline is used.
  • The Output Constraint Guideline is especially effective on summarization, translation, and table-to-text tasks, and ablations show that removing either the token or the sentence constraint hurts performance.
  • The Jensen–Shannon divergence between generated and ground-truth distributions on the selected metrics correlates with ROUGE-L, linking the mechanism (better property transfer) to the reported quality gains.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A natural extension the paper leaves implicit: the same two-stream guideline recipe should transfer to other surface-constrained generation tasks where demonstrations alone underperform, such as code generation with length or style requirements; the OCG part in particular is model-independent and costs almost nothing to compute.
  • The paper's own failure case (high-variance outputs such as SWiPE) points to a concrete improvement: replacing the single average-based OCG with input-conditioned length predictions could extend the gains to tasks where a fixed min/max window cannot fit.
  • Because LongGuide's selection step uses ROUGE-L on the 50-sample train set as a proxy for test quality, the method inherits the usual risk of proxy-based selection; a testable extension would be to run the same selection with a learned reward model or a stronger evaluator and compare downstream performance.
  • The reported transfer of guidelines from a weak open model to a strong closed model suggests that guideline discovery could be separated from inference: teams with limited API budgets could amortize one-time guideline generation on cheap models and then apply the same text to expensive ones, but the paper only tests this in one direction, so the reverse pattern remains open.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

2 major / 6 minor

Summary. The paper argues that in-context learning (ICL) demonstrations are insufficient for long-form generation because they fail to reliably transfer task-specific language and format properties. To address this, the authors propose LongGuide, which learns two kinds of guidelines from at most 50 training samples: Metric Guidelines (MGs) that steer the model toward self-evaluated linguistic qualities, and Output Constraint Guidelines (OCGs) that impose token- and sentence-level length bounds. LongGuide then selects among {no guideline, MG, OCG, MG-OCG} by validation performance on the same 50 samples. The empirical study covers seven generation tasks plus AlpacaEval2 with Mistral-7B-it and ChatGPT, reporting average ROUGE-L / GPT-4o-Judge gains over 5% in zero- and few-shot settings, along with human evaluation, ablations, transfer to non-instruct models, and cost comparisons.

Significance. If the claimed effects hold, LongGuide is a useful practical contribution: it offers a cheap, task-level procedure that improves strong LLMs on long-form generation, and the transfer experiments suggest an attractive workflow where a weak open model learns guidelines that later improve a strong closed model. The evaluation is broad, includes human judgments, and the component ablations are informative. The central effect is plausible—enforcing output length and format statistics from a small training set directly addresses a well-known failure mode of long-form generation. However, the theoretical support is limited to the marginal language distribution P(X) rather than the generation distribution P(Y|X), the test sets are small (200 samples per task), and the automatic selection step has a documented suboptimal case.

major comments (2)
  1. [§3, Step 5; Apx. E.8 (Tables 14 and 4)] Step 5 of Algorithm 1 selects the guideline configuration on the same 50-sample Dtrain used to estimate the OCG statistics and MG score profiles, so the selection is not protected against overfitting to Dtrain. The paper itself documents a concrete failure in Apx. E.8: for IWSLT17 en-ja with ChatGPT few-shot, Dtrain selects MG-OCG (Table 14) while the best configuration on the test set is MG (Table 4), with ROUGE-L 38.43 for the selected MG-OCG versus 39.45 for MG. This weakens the abstract's claim that LongGuide 'automatically selects the best combination of guidelines.' Please add a stability check (e.g., bootstrap over Dtrain draws or multiple random Dtrain subsamples) and either qualify the selection claim or show that the selection is not systematically biased.
  2. [Apx. B, Remark B.1 and §2] The theoretical result is about the marginal input distribution: Remark B.1 proves PM(X|Df) ≠ PT(X) under PM ≠ PT. It does not directly address the output distribution PM(Y|X) or arg max_y PM(Y|X), which is what generation quality depends on. The abstract and §2 nevertheless claim that ICL demonstrations are 'theoretically' insufficient to teach 'the task's language and format distributions for generation.' The Limitations section explicitly concedes this gap, but the main text should be revised so that the theoretical contribution is not overstated. I suggest rephrasing the claim to say the theory supports the language-distribution mismatch, while the generation-distribution statement remains a hypothesis supported empirically.
minor comments (6)
  1. [Abstract and Table 3] The 'over 5%' improvement is an average across seven tasks and settings; several individual cells are far below 5% (e.g., SWiPE zero-shot ChatGPT shows 0.00 ROUGE-L gain in Table 3, and Mistral few-shot CNN shows 1.63). Please qualify the abstract statement as an average and, if space permits, report the per-task range.
  2. [§2, first paragraph] There is a typo: 'ansMistral' should be 'and Mistral'. Also, the name 'Suzzanne' in Figure 1 is inconsistent with 'Suzanne' in the text and other examples.
  3. [Algorithm 1 header] The string 'MGâ˘A¸SOCG' appears to be an encoding artifact; it should likely read 'MG-OCG'. Please check all occurrences of this artifact in the appendix.
  4. [§G.3, Table 20] The cost comparison states LongGuide is 'approximately at least 3.75 times more cost-efficient' than prompt optimizers; Table 20 shows it is 3.75× cheaper than adv-ICL but 18.75× cheaper than APO. Please make the comparison explicit per baseline.
  5. [§5.2, Figure 5] The 'Gen. Gui. on Demos' baseline is only briefly defined; please clarify how the general guideline prompt is elicited from demonstrations, since this is the main comparison for the 'learned from demonstrations' claim.
  6. [§5.1] The human evaluation is based on 50 samples and only 5 MG metrics; while useful, the small scale should be acknowledged in the main text (it is currently only implicit in the setup).

Circularity Check

0 steps flagged · score 2.0 of 10

No significant circularity: the central claims are validated against held-out test sets and external priors; self-citations are not load-bearing.

full rationale

LongGuide's main empirical claim—over 5% gains on seven generation tasks—is evaluated on held-out test samples (Table 3), not on the 50-sample Dtrain used to build the OCG/MG or to select among them in Step 5 (Algorithm 1, lines 19-20). The Step 5 choice is a validation procedure: the paper reports test ROUGE-L/GPT-4o-Judge, and E.8 explicitly discloses the one case where the Dtrain-selected guideline is not the test-best (IWSLT17 en-ja, ChatGPT few-shot), demonstrating that the reported gains are not forced by construction. The theoretical Remark B.1 is a conditional proof under explicitly stated assumptions (PM != PT and zero-equivalence); it does not derive the conclusion from itself, and the assumption is supported by external examples rather than by the theorem. The paper's self-citations (adv-ICL baseline and multi-expert prompting related work) are comparisons or background, not premises of LongGuide; no uniqueness claim or ansatz is imported from the authors' prior work. The Limitations section's caveats about task-level average statistics and high-variance tasks are robustness/overfitting concerns, which the paper itself acknowledges, rather than evidence that any prediction is definitionally equal to its inputs. Overall, no circular step can be exhibited.

Assumptions & free parameters 7 free parameters · 5 assumptions · 0 invented entities

The method relies on a small number of hand-chosen hyperparameters (K=5, batch size=5, top-5, self-consistency=3) and on several domain assumptions about LLM capabilities and distributional representativeness. No new physical entities are introduced.

free parameters (7)
  • K (metric selection iterations) = 5
    Number of iterations to sample batches for metric selection; hand-chosen, not tuned. Affects which metrics are selected.
  • batch_size for Step 1 = 5
    Batch size for metric selection; hand-chosen.
  • top_5 constraints per iteration = 5
    Number of metrics the model is asked to select per batch; hand-chosen.
  • self_consistency_samples = 3
    Number of samples for self-consistency in Step 2 and property scoring; hand-chosen.
  • train_set_size_cap = 50
    Maximum number of training samples used by LongGuide; hand-chosen for efficiency.
  • metric_score_scale = 1-5
    Scale for self-evaluation scores; hand-chosen.
  • metric_pool_size = 27
    Number of evaluation metrics in the pool S; hand-chosen from four sources.
assumptions (5)
  • domain assumption Existence of x with P_M(x) != P_T(x) (Assumption B.1)
    Assumes the LLM does not fully capture the task language distribution; needed for Remark B.1.
  • domain assumption Functional zero equivalence of P_M and P_T (Assumption B.2)
    Assumes P_M and P_T have the same support; used in the proof of Remark B.1.
  • ad hoc to paper The task can be approximated by a sum of text property tasks (Hypothesis B.1)
    Proposed to justify LongGuide; not proven, only empirically supported.
  • domain assumption LLMs can follow the generated guidelines and self-evaluate reliably
    LongGuide relies on model instruction-following and self-evaluation capabilities; acknowledged in Limitations.
  • domain assumption Length statistics from Dtrain generalize to the test distribution
    OCG uses min/max/avg sentence and token counts from 50 training samples; if unrepresentative, OCG can hurt (as on SWiPE).

how reviews work

0 comments
Cite this review

Pith. "Pith review of Beyond In-Context Learning: Aligning Long-form Generation of Large Language Models via Task-Inherent Attribute Guidelines." pith.science (2026). https://pith.science/paper/NYHMFZN2

@misc{pith2026250601265,
  author       = {Pith},
  title        = {Pith review of: Beyond In-Context Learning: Aligning Long-form Generation of Large Language Models via Task-Inherent Attribute Guidelines},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/NYHMFZN2}},
  note         = {Machine review of arXiv:2506.01265}
}
read the original abstract

In-context learning (ICL) is an important yet not fully understood ability of pre-trained large language models (LLMs). It can greatly enhance task performance using a few examples, termed demonstrations, without fine-tuning. Although effective in question answering, ICL often underperforms in long-form generation tasks such as summarization. Under appropriately realistic assumptions, we empirically and theoretically show that ICL demonstrations alone are insufficient to teach LLMs the task language and format distributions for generation. We argue for explicit exposure to the task distributions and hypothesize that defining them by prompting enhances model performance. To this end, we present LongGuide, which efficiently generates two parallel streams of guidelines capturing task language and format properties: (i) Metric Guidelines (MGs) that instruct models to optimize self-evaluated metrics; and (ii) Output Constraint Guidelines (OCGs) that constrain generation at both token and sentence levels. LongGuide automatically selects the best combination of guidelines, improving both strong open- and closed-source LLMs by over 5% in both zero- and few-shot settings. We show that LongGuide is generalizable, learnable by weak models to enhance strong ones, and integrates synergistically with automatic prompt optimizers.

Figures

Figures reproduced from arXiv: 2506.01265 by the authors.

Figure 1
Figure 1. ChatGPT results on a SAMSum example (Gliwa et al., 2019). With LongGuide, the generated output aligns better with ground truth, and the quality is also improved by removing verbose details: “The dialogue...” (ZS), “...discuss their day” (FS). See Appx.- [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Overview of LongGuide. Orange and blue boxes denote the learned metric guideline and output constraint guideline. lect appropriate language evaluation metrics for self-evaluation. For this purpose, we first con￾struct a pool of evaluation metrics, S, applicable to any text generation task. S consists of 27 distinct metrics from 4 main sources (Appx.- [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. GPT-4-Judge scores over criteria. markedly more effective than APO optimizing only ROUGE-L on limited data. Thirdly, we observe that LongGuide achieves the highest improvements on CommonGen-Challenge with 15.62% and low￾est on SWiPE with 1.85% on ROUGE-L. These improvements are mainly because the answers gen￾erated by the baselines are often far longer than the ground truth. LongGuide rectifies this issue by control… view at source ↗
Figures from the paper (16 more)
Figure 4
Figure 4. Figure 4: Win/Draw/Loss rates of w/ versus w/o LongGuide. We perform a human evaluation to quantify whether LongGuide helps LLMs optimize the se￾lected metrics and enhance generation quality, as no automatic methods can address this need to date. For this purpose, we randomly se…
Figure 5
Figure 5. Figure 5: LongGuide learned from demonstrations substan￾tially enhances Mistral performance (ROUGE-L). evance” obtains the lowest winning rate of 12%, possibly because ZS models can already generate highly relevant outcomes. Meanwhile, on the OCG metrics, LongGuide achieves a su…
Figure 6
Figure 6. Figure 6: Entropy of attention over the input context across 32 Mistral layers. To analyze LongGuide’s impact on LLMs, we perform a simple attention analysis to investigate (1) how LLMs attend to Long￾Guide and (2) utilize the input context when conditioning on LongGuide. Specif…
Figure 7
Figure 7. Figure 7: Property maintenance experiments with ICL. See Appx.- [PITH_FULL_IMAGE:figures/full_fig_p021_7.png]
Figure 8
Figure 8. Figure 8: Property maintenance experiments with ICL full results. IT is the adding a simple guideline baseline. the final layer, the model distributes attention more evenly with guidelines than without. Generally, MG stabilizes context use across layers, while OCG shows greater …
Figure 9
Figure 9. Figure 9: Pairwise Pearson correlation coefficient of metrics. Source Metrics # The ABC’s of Communication (Wagner, 1963) Accuracy, Brevity, Clarity 3 BARTScore (Yuan et al., 2021) Relevance, Coherence 2 GPTScore (Fu et al., 2023) Semantic Coverage, Factuality, Fluency, Informat…
Figure 10
Figure 10. Figure 10: Density plots of MG and OCG metrics selected by Mistral under the few-shot (FS) setting, measured on ground-truth, FS, and FS w/ LongGuide answers. For JensenâA¸SShannon divergence, ˘ lower is better [PITH_FULL_IMAGE:figures/full_fig_p023_10.png]
Figure 11
Figure 11. Figure 11: Fine-grained human evaluation results on evaluated MG metrics [PITH_FULL_IMAGE:figures/full_fig_p024_11.png]
Figure 12
Figure 12. Figure 12: Frequency of metrics selected as the metric guideline. Mistral ChatGPT [PITH_FULL_IMAGE:figures/full_fig_p026_12.png]
Figure 13
Figure 13. Figure 13: Average scores of metrics as the metric guideline. Methods CNN (3.0.0) IWSLT17 en-ja CommGen-Chall. Zero-shot (ZS) 19.23±0.34 13.12±1.39 10.12±0.02 + LongGuide 22.46±0.64 16.53±0.59 25.20±1.89 + LongGuide w/o Token Constraint 21.54±0.52↓ 14.09±1.07↓ 21.49±2.15↓ + Long…
Figure 14
Figure 14. Figure 14: Full example of ChatGPT results on a SAMSum example (Gliwa et al., 2019) w/ LongGuide guidelines (Zero-shot + LongGuide and Few-shot + LongGuide) from [PITH_FULL_IMAGE:figures/full_fig_p031_14.png]
Figure 15
Figure 15. Figure 15: A sample from SAMSum dataset where MG and OCG supplement each other and are not interchangeable to increase the performance in final answer [PITH_FULL_IMAGE:figures/full_fig_p032_15.png]
Figure 16
Figure 16. Figure 16: An example of SWiPE (Laban et al., 2023) where the record contains fewer tokens than the expected average. This reduces the effectiveness of OCG and MG individually, but their combination could enhance performance [PITH_FULL_IMAGE:figures/full_fig_p033_16.png]
Figure 17
Figure 17. Figure 17: A CommonGen-Challenge example (Lin et al., 2020), where output with high Conciseness score could have low Informativeness score and vice versa Zero-shot answer (ROUGE-L = 16.22) George attempts to make arrangements to meet Donna to discuss a personal matter, but she i…
Figure 18
Figure 18. Figure 18: A SAMSum example, where skipping step 2 worsens the performance due to lack of clarity in metrics [PITH_FULL_IMAGE:figures/full_fig_p034_18.png]
Figure 19
Figure 19. Figure 19: Full text for an example in §2 [PITH_FULL_IMAGE:figures/full_fig_p035_19.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

85 extracted references · 21 canonical work pages

  1. [1]

    Yushi Bai, Xin Lv, Jiajie Zhang, Hongchang Lyu, Jiankai Tang, Zhidian Huang, Zhengxiao Du, Xiao Liu, Aohan Zeng, Lei Hou, Yuxiao Dong, Jie Tang, and Juanzi Li. 2024. https://doi.org/10.18653/v1/2024.acl-long.172 L ong B ench: A bilingual, multitask benchmark for long context understanding . In Proceedings of the 62nd Annual Meeting of the Association for ...

    doi:10.18653/v1/2024.acl-long.172 2024

  2. [2]

    Do, Yan Xu, and Pascale Fung

    Yejin Bang, Samuel Cahyawijaya, Nayeon Lee, Wenliang Dai, Dan Su, Bryan Wilie, Holy Lovenia, Ziwei Ji, Tiezheng Yu, Willy Chung, Quyet V. Do, Yan Xu, and Pascale Fung. 2023. https://doi.org/10.18653/v1/2023.ijcnlp-main.45 A multitask, multilingual, multimodal evaluation of C hat GPT on reasoning, hallucination, and interactivity . In Proceedings of the 13...

    doi:10.18653/v1/2023.ijcnlp-main.45 2023

  3. [3]

    Steven Bird and Edward Loper. 2004. https://aclanthology.org/P04-3031 NLTK : The natural language toolkit . In Proceedings of the ACL Interactive Poster and Demonstration Sessions , pages 214--217, Barcelona, Spain. Association for Computational Linguistics

    2004

  4. [4]

    Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020. Language models are few-shot learners. Advances in neural information processing systems, 33:1877--1901

    2020

  5. [5]

    Mauro Cettolo, Marcello Federico, Luisa Bentivogli, Jan Niehues, Sebastian St \"u ker, Katsuhito Sudoh, Koichiro Yoshino, and Christian Federmann. 2017. https://aclanthology.org/2017.iwslt-1.1 Overview of the IWSLT 2017 evaluation campaign . In Proceedings of the 14th International Conference on Spoken Language Translation, pages 2--14, Tokyo, Japan. Inte...

    2017

  6. [6]

    Daixuan Cheng, Shaohan Huang, and Furu Wei. 2024. https://openreview.net/forum?id=y886UXPEZ0 Adapting large language models via reading comprehension . In The Twelfth International Conference on Learning Representations

    2024

  7. [7]

    Shazeer, Vinodkumar Prabhakaran, Emily Reif, Nan Du, Benton C

    Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, Parker Schuh, Kensen Shi, Sasha Tsvyashchenko, Joshua Maynez, Abhishek Rao, Parker Barnes, Yi Tay, Noam M. Shazeer, Vinodkumar Prabhakaran, Emily Reif, Nan Du, Benton C. Hutchinson, Reiner Pope, Ja...

    2022

  8. [8]

    Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, et al. 2021. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168

    arXiv 2021

Show all 85 references
  1. [9]

    Chen, Kenji Kawaguchi, Michael Shieh, and Junxian He

    Xuan Long Do, Yiran Zhao, Hannah Brown, Yuxi Xie, James Xu Zhao, Nancy F. Chen, Kenji Kawaguchi, Michael Shieh, and Junxian He. 2024. https://doi.org/10.18653/v1/2024.acl-long.395 Prompt optimization via adversarial in-context learning . In Proceedings of the 62nd Annual Meeti...

    2024 doi

  2. [10]

    Qingxiu Dong, Lei Li, Damai Dai, Ce Zheng, Zhiyong Wu, Baobao Chang, Xu Sun, Jingjing Xu, and Zhifang Sui. 2022. A survey on in-context learning. arXiv preprint arXiv:2301.00234

    2022 arXiv

  3. [11]

    Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, Anirudh Goyal, Anthony S. Hartshorn, Aobo Yang, Archi Mitra, Archie Sravankumar, Artem Korenev, Arthur Hinsvark, Arun Rao, Aston ...

    2024 arXiv

  4. [12]

    Yann Dubois, Percy Liang, and Tatsunori Hashimoto. 2024. https://openreview.net/forum?id=CybBmzWBX0 Length-controlled alpacaeval: A simple debiasing of automatic evaluators . In First Conference on Language Modeling

    2024

  5. [13]

    Angela Fan, David Grangier, and Michael Auli. 2018 a . https://doi.org/10.18653/v1/W18-2706 Controllable abstractive summarization . In Proceedings of the 2nd Workshop on Neural Machine Translation and Generation, pages 45--54, Melbourne, Australia. Association for Computation...

    2018 doi

  6. [14]

    Angela Fan, Yacine Jernite, Ethan Perez, David Grangier, Jason Weston, and Michael Auli. 2019. https://doi.org/10.18653/v1/P19-1346 ELI 5: Long form question answering . In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 3558--356...

    2019 doi

  7. [15]

    Angela Fan, Mike Lewis, and Yann Dauphin. 2018 b . https://doi.org/10.18653/v1/P18-1082 Hierarchical neural story generation . In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 889--898, Melbourne, Austral...

    2018 doi

  8. [16]

    Marcio Fonseca and Shay B Cohen. 2024. Can large language model summarizers adapt to diverse scientific communication goals? arXiv preprint arXiv:2401.10415

    2024 arXiv

  9. [17]

    Jinlan Fu, See-Kiong Ng, Zhengbao Jiang, and Pengfei Liu. 2023. Gptscore: Evaluate as you desire. arXiv preprint arXiv:2302.04166

    2023 arXiv

  10. [18]

    Claire Gardent, Anastasia Shimorina, Shashi Narayan, and Laura Perez-Beltrachini. 2017. https://doi.org/10.18653/v1/W17-3518 The W eb NLG challenge: Generating text from RDF data . In Proceedings of the 10th International Conference on Natural Language Generation, pages 124--1...

    2017 doi

  11. [19]

    Bogdan Gliwa, Iwona Mochol, Maciej Biesek, and Aleksander Wawer. 2019. https://doi.org/10.18653/v1/D19-5409 SAMS um corpus: A human-annotated dialogue dataset for abstractive summarization . In Proceedings of the 2nd Workshop on New Frontiers in Summarization, pages 70--79, Ho...

    2019 doi

  12. [20]

    Qingyan Guo, Rui Wang, Junliang Guo, Bei Li, Kaitao Song, Xu Tan, Guoqing Liu, Jiang Bian, and Yujiu Yang. 2024. https://openreview.net/forum?id=ZG3RaNIsO8 Connecting large language models with evolutionary algorithms yields powerful prompt optimizers . In The Twelfth Internat...

    2024

  13. [21]

    Muhammad Usman Hadi, Rizwan Qureshi, Abbas Shah, Muhammad Irfan, Anas Zafar, Muhammad Bilal Shaikh, Naveed Akhtar, Jia Wu, Seyedali Mirjalili, et al. 2023. A survey on large language models: Applications, challenges, limitations, and practical usage. Authorea Preprints

    2023

  14. [22]

    Saiful Islam, Kazi Mubasshir, Yuan-Fang Li, Yong-Bin Kang, M

    Tahmid Hasan, Abhik Bhattacharjee, Md. Saiful Islam, Kazi Mubasshir, Yuan-Fang Li, Yong-Bin Kang, M. Sohel Rahman, and Rifat Shahriyar. 2021. https://doi.org/10.18653/v1/2021.findings-acl.413 XL -sum: Large-scale multilingual abstractive summarization for 44 languages . In Fin...

    2021 doi

  15. [23]

    Junxian He, Wojciech Kryscinski, Bryan McCann, Nazneen Rajani, and Caiming Xiong. 2022. https://doi.org/10.18653/v1/2022.emnlp-main.396 CTRL sum: Towards generic controllable text summarization . In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Pr...

    2022 doi

  16. [24]

    Ari Holtzman, Jan Buys, Li Du, Maxwell Forbes, and Yejin Choi. 2019. The curious case of neural text degeneration. In International Conference on Learning Representations

    2019

  17. [25]

    Yukun Huang, Yixin Liu, Raghuveer Thirukovalluru, Arman Cohan, and Bhuwan Dhingra. 2024. Calibrating long-form generations from large language models. arXiv preprint arXiv:2402.06544

    2024 arXiv

  18. [26]

    Pegah Jandaghi, XiangHai Sheng, Xinyi Bai, Jay Pujara, and Hakim Sidahmed. 2023. Faithful persona-based conversational dataset generation with large language models. arXiv preprint arXiv:2312.10007

    2023 arXiv

  19. [27]

    Albert Q Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, et al. 2023. Mistral 7b. arXiv preprint arXiv:2310.06825

    2023 arXiv

  20. [28]

    Klaus Krippendorff. 2022. The reliability of generating data. Chapman and Hall/CRC

    2022

  21. [29]

    Philippe Laban, Jesse Vig, Wojciech Kryscinski, Shafiq Joty, Caiming Xiong, and Chien-Sheng Wu. 2023. https://doi.org/10.18653/v1/2023.acl-long.596 SW i PE : A dataset for document-level simplification of W ikipedia pages . In Proceedings of the 61st Annual Meeting of the Asso...

    2023 doi

  22. [30]

    Surafel Melaku Lakew, Mattia Di Gangi, and Marcello Federico. 2019. https://aclanthology.org/2019.iwslt-1.31 Controlling the output length of neural machine translation . In Proceedings of the 16th International Conference on Spoken Language Translation, Hong Kong. Association...

    2019

  23. [31]

    Junyi Li, Tianyi Tang, Wayne Xin Zhao, Jian-Yun Nie, and Ji-Rong Wen. 2024. Pre-trained language models for text generation: A survey. ACM Computing Surveys, 56(9):1--39

    2024

  24. [32]

    Bill Yuchen Lin, Wangchunshu Zhou, Ming Shen, Pei Zhou, Chandra Bhagavatula, Yejin Choi, and Xiang Ren. 2020. https://doi.org/10.18653/v1/2020.findings-emnlp.165 C ommon G en: A constrained text generation challenge for generative commonsense reasoning . In Findings of the Ass...

    2020 doi

  25. [33]

    Chin-Yew Lin. 2004. https://aclanthology.org/W04-1013 ROUGE : A package for automatic evaluation of summaries . In Text Summarization Branches Out, pages 74--81, Barcelona, Spain. Association for Computational Linguistics

    2004

  26. [34]

    Jianhua Lin. 1991. Divergence measures based on the shannon entropy. IEEE Transactions on Information theory, 37(1):145--151

    1991

  27. [35]

    Do Xuan Long, Duong Ngoc Yen, Anh Tuan Luu, Kenji Kawaguchi, Min-Yen Kan, and Nancy F. Chen. 2024. https://doi.org/10.18653/v1/2024.emnlp-main.1135 Multi-expert prompting improves reliability, safety and usefulness of large language models . In Proceedings of the 2024 Conferen...

    2024 doi

  28. [36]

    Albert Lu, Hongxin Zhang, Yanzhe Zhang, Xuezhi Wang, and Diyi Yang. 2023. https://doi.org/10.18653/v1/2023.findings-eacl.148 Bounding the capabilities of large language models in open text generation with prompt constraints . In Findings of the Association for Computational Li...

    2023 doi

  29. [37]

    Louis Martin, \'E ric de la Clergerie, Beno \^ t Sagot, and Antoine Bordes. 2020. https://aclanthology.org/2020.lrec-1.577 Controllable sentence simplification . In Proceedings of the Twelfth Language Resources and Evaluation Conference, pages 4689--4698, Marseille, France. Eu...

    2020

  30. [38]

    Sewon Min, Kalpesh Krishna, Xinxi Lyu, Mike Lewis, Wen-tau Yih, Pang Koh, Mohit Iyyer, Luke Zettlemoyer, and Hannaneh Hajishirzi. 2023. https://doi.org/10.18653/v1/2023.emnlp-main.741 FA ct S core: Fine-grained atomic evaluation of factual precision in long form text generatio...

    2023 doi

  31. [39]

    OpenAI. 2022. https://openai.com/blog/chatgpt Introducing chatgpt

    2022

  32. [40]

    OpenAI. 2023. https://openai.com/index/gpt-4/ Gpt-4 is openai’s most advanced system, producing safer and more useful responses

    2023

  33. [41]

    OpenAI. 2024. https://openai.com/index/hello-gpt-4o/ Hello gpt-4o

    2024

  34. [42]

    Rangeet Pan, Ali Reza Ibrahimzada, Rahul Krishna, Divya Sankar, Lambert Pouguem Wassi, Michele Merler, Boris Sobolev, Raju Pavuluri, Saurabh Sinha, and Reyhaneh Jabbarvand. 2024. Lost in translation: A study of bugs introduced by large language models while translating code. I...

    2024

  35. [43]

    Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. https://doi.org/10.3115/1073083.1073135 B leu: a method for automatic evaluation of machine translation . In Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics, pages 311--3...

    2002

  36. [44]

    Arkil Patel, Satwik Bhattamishra, and Navin Goyal. 2021. https://doi.org/10.18653/v1/2021.naacl-main.168 Are NLP models really able to solve simple math word problems? In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Ling...

    2021 doi

  37. [45]

    Karl Pearson. 1895. Vii. note on regression and inheritance in the case of two parents. proceedings of the royal society of London, 58(347-352):240--242

  38. [46]

    Reid Pryzant, Dan Iter, Jerry Li, Yin Lee, Chenguang Zhu, and Michael Zeng. 2023. https://doi.org/10.18653/v1/2023.emnlp-main.494 Automatic prompt optimization with `` gradient descent '' and beam search . In Proceedings of the 2023 Conference on Empirical Methods in Natural L...

    2023 doi

  39. [47]

    Yevgeniy Puzikov and Iryna Gurevych. 2018. https://doi.org/10.18653/v1/W18-6557 E 2 E NLG challenge: Neural models vs. templates . In Proceedings of the 11th International Conference on Natural Language Generation, pages 463--471, Tilburg University, The Netherlands. Associati...

    2018 doi

  40. [48]

    Chengwei Qin, Aston Zhang, Anirudh Dagar, and Wenming Ye. 2023. In-context learning with iterative demonstration selection. arXiv preprint arXiv:2310.09881

    2023 arXiv

  41. [49]

    Qwen Team . 2024. https://qwenlm.github.io/blog/qwen2.5/ Qwen2.5: A party of foundation models

    2024

  42. [50]

    Marc'Aurelio Ranzato, Sumit Chopra, Michael Auli, and Wojciech Zaremba. 2015. Sequence level training with recurrent neural networks. arXiv preprint arXiv:1511.06732

    2015 arXiv

  43. [51]

    Jie Ren, Yao Zhao, Tu Vu, Peter J Liu, and Balaji Lakshminarayanan. 2023. Self-evaluation improves selective generation in large language models. In Proceedings on, pages 49--64. PMLR

    2023

  44. [52]

    Kyle Richardson, Sina Zarrie , and Jonas Kuhn. 2017. https://doi.org/10.18653/v1/W17-3516 The C ode2 T ext challenge: Text generation in source libraries . In Proceedings of the 10th International Conference on Natural Language Generation, pages 115--119, Santiago de Compostel...

    2017 doi

  45. [53]

    Nikunj Saunshi, Sadhika Malladi, and Sanjeev Arora. 2020. A mathematical exploration of why language models help solve downstream tasks. arXiv preprint arXiv:2010.03648

    2020 arXiv

  46. [54]

    Liu, and Christopher D

    Abigail See, Peter J. Liu, and Christopher D. Manning. 2017. https://doi.org/10.18653/v1/P17-1099 Get to the point: Summarization with pointer-generator networks . In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers...

    2017 doi

  47. [55]

    Aaditya K Singh and DJ Strouse. 2024. Tokenization counts: the impact of tokenization on arithmetic in frontier llms. arXiv preprint arXiv:2402.14903

    2024 arXiv

  48. [56]

    Jiao Sun, Yufei Tian, Wangchunshu Zhou, Nan Xu, Qian Hu, Rahul Gupta, John Wieting, Nanyun Peng, and Xuezhe Ma. 2023 a . https://doi.org/10.18653/v1/2023.emnlp-main.190 Evaluating large language models on controlled generation tasks . In Proceedings of the 2023 Conference on E...

    2023 doi

  49. [57]

    Jiao Sun, Yufei Tian, Wangchunshu Zhou, Nan Xu, Qian Hu, Rahul Gupta, John Frederick Wieting, Nanyun Peng, and Xuezhe Ma. 2023 b . Evaluating large language models on controlled generation tasks. arXiv preprint arXiv:2310.14542

    2023 arXiv

  50. [58]

    Gemini Team, Rohan Anil, Sebastian Borgeaud, Yonghui Wu, Jean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, et al. 2023. Gemini: a family of highly capable multimodal models. arXiv preprint arXiv:2312.11805

    2023 arXiv

  51. [59]

    Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timoth \'e e Lacroix, Baptiste Rozi \`e re, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. 2023. https://api.semanticscholar.org...

    2023 arXiv

  52. [60]

    Sara P Wagner. 1963. The abc's of communication. American Association of Industrial Nurses Journal, 11(8):8--11

    1963

  53. [61]

    Jiaan Wang, Yunlong Liang, Fandong Meng, Zengkui Sun, Haoxiang Shi, Zhixu Li, Jinan Xu, Jianfeng Qu, and Jie Zhou. 2023 a . https://doi.org/10.18653/v1/2023.newsum-1.1 Is C hat GPT a good NLG evaluator? a preliminary study . In Proceedings of the 4th New Frontiers in Summariza...

    2023 doi

  54. [62]

    Rui Wang, Hongru Wang, Fei Mi, Yi Chen, Ruifeng Xu, and Kam-Fai Wong. 2023 b . Self-critique prompting with large language models for inductive instructions. arXiv preprint arXiv:2305.13733

    2023 arXiv

  55. [63]

    Xinyi Wang, Wanrong Zhu, Michael Saxon, Mark Steyvers, and William Yang Wang. 2024. Large language models are latent variable models: Explaining and finding good demonstrations for in-context learning. Advances in Neural Information Processing Systems, 36

    2024

  56. [64]

    Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc V Le, Ed H Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou. 2022 a . Self-consistency improves chain of thought reasoning in language models. In The Eleventh International Conference on Learning Representations

    2022

  57. [65]

    Yizhong Wang, Swaroop Mishra, Pegah Alipoormolabashi, Yeganeh Kordi, Amirreza Mirzaei, Atharva Naik, Arjun Ashok, Arut Selvan Dhanasekaran, Anjana Arunkumar, David Stap, Eshaan Pathak, Giannis Karamanolakis, Haizhi Lai, Ishan Purohit, Ishani Mondal, Jacob Anderson, Kirby Kuzni...

    2022

  58. [66]

    Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. 2022. Chain-of-thought prompting elicits reasoning in large language models. Advances in neural information processing systems, 35:24824--24837

    2022

  59. [67]

    Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Remi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mari...

    2020 doi

  60. [68]

    Sang Michael Xie, Aditi Raghunathan, Percy Liang, and Tengyu Ma. 2021. An explanation of in-context learning as implicit bayesian inference. In International Conference on Learning Representations

    2021

  61. [69]

    Chengrun Yang, Xuezhi Wang, Yifeng Lu, Hanxiao Liu, Quoc V Le, Denny Zhou, and Xinyun Chen. 2024. https://openreview.net/forum?id=Bb4VGOWELI Large language models as optimizers . In The Twelfth International Conference on Learning Representations

    2024

  62. [70]

    Xianjun Yang, Yan Li, Xinlu Zhang, Haifeng Chen, and Wei Cheng. 2023 a . Exploring the limits of chatgpt for query or aspect-based text summarization. arXiv preprint arXiv:2302.08081

    2023 arXiv

  63. [71]

    Zhao Yang, Yuanzhe Zhang, Dianbo Sui, Cao Liu, Jun Zhao, and Kang Liu. 2023 b . https://doi.org/10.18653/v1/2023.emnlp-main.331 Representative demonstration selection for in-context learning with two-stage determinantal point process . In Proceedings of the 2023 Conference on ...

    2023 doi

  64. [72]

    Fan Yin, Jesse Vig, Philippe Laban, Shafiq Joty, Caiming Xiong, and Chien-Sheng Wu. 2023. https://doi.org/10.18653/v1/2023.acl-long.172 Did you read the instructions? rethinking the effectiveness of task definitions in instruction learning . In Proceedings of the 61st Annual M...

    2023 doi

  65. [73]

    Weizhe Yuan, Graham Neubig, and Pengfei Liu. 2021. https://openreview.net/forum?id=5Ya8PbvpZ9 BARTS core: Evaluating generated text as text generation . In Advances in Neural Information Processing Systems

    2021

  66. [74]

    Weinberger, and Yoav Artzi

    Tianyi Zhang, Varsha Kishore, Felix Wu*, Kilian Q. Weinberger, and Yoav Artzi. 2020. https://openreview.net/forum?id=SkeHuCVFDr Bertscore: Evaluating text generation with bert . In International Conference on Learning Representations

    2020

  67. [75]

    Gonzalez, and Ion Stoica

    Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, Hao Zhang, Joseph E. Gonzalez, and Ion Stoica. 2023. https://openreview.net/forum?id=uccHPGDlao Judging LLM -as-a-judge with MT -bench and chatbot ...

    2023

  68. [76]

    Yongchao Zhou, Andrei Ioan Muresanu, Ziwen Han, Keiran Paster, Silviu Pitis, Harris Chan, and Jimmy Ba. 2023 a . https://openreview.net/forum?id=92gvk82DE- Large language models are human-level prompt engineers . In The Eleventh International Conference on Learning Representations

    2023

  69. [77]

    Yongxin Zhou, Fabien Ringeval, and Fran c ois Portet. 2023 b . Can gpt models follow human summarization guidelines? evaluating chatgpt and gpt-4 for dialogue summarization. arXiv preprint arXiv:2310.16810

    2023

  70. [78]

    online" 'onlinestring :=

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list...

  71. [79]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

  72. [80]

    URL: " 'urlintro :=

    ENTRY address author booktitle chapter edition editor howpublished institution journal key month note number organization pages publisher school series title type volume year eprint doi pubmed url lastchecked label extra.label sort.label short.list INTEGERS output.state before...

  73. [81]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

  74. [82]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

  75. [83]

    @esa (Ref

    \@ifxundefined[1] #1\@undefined \@firstoftwo \@secondoftwo \@ifnum[1] #1 \@firstoftwo \@secondoftwo \@ifx[1] #1 \@firstoftwo \@secondoftwo [2] @ #1 \@temptokena #2 #1 @ \@temptokena \@ifclassloaded agu2001 natbib The agu2001 class already includes natbib coding, so you should ...

  76. [84]

    \@lbibitem[] @bibitem@first@sw\@secondoftwo \@lbibitem[#1]#2 \@extra@b@citeb \@ifundefined br@#2\@extra@b@citeb \@namedef br@#2 \@nameuse br@#2\@extra@b@citeb \@ifundefined b@#2\@extra@b@citeb @num @parse #2 @tmp #1 NAT@b@open@#2 NAT@b@shut@#2 \@ifnum @merge>\@ne @bibitem@firs...

  77. [85]

    @open @close @open @close and [1] URL: #1 \@ifundefined chapter * \@mkboth \@ifxundefined @sectionbib * \@mkboth * \@mkboth\@gobbletwo \@ifclassloaded amsart * \@ifclassloaded amsbook * \@ifxundefined @heading @heading NAT@ctr thebibliography [1] @ \@biblabel @NAT@ctr \@bibset...

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.