Pith. sign in

REVIEW 5 major objections 5 minor 43 references

Shuttle Between the Instructions and the Parameters of Large Language Models

T0 review · 5 major / 5 minor · reviewed 2026-08-09 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read This paper claims that a large language model's task instructions and its trainable parameters are two compressions of the same task data, and that a single 10-token latent code can shuttle between them.

desk verdict A promising, well-engineered paper whose central mutual-mapping claim is only partially supported; the induction results are strong, but the parameters-to-instructions direction needs a causal control. read the letter →

arxiv 2502.02315 v3 pith:GP4KRIHV submitted 2025-02-04 cs.LG cs.CL

Wangtao Sun , Haotian Xu , Huanxuan Liao , Xuanqing Yu , Zhongtao Jiang , Shizhu He , Jun Zhao , Kang Liu This is my paper · ORCID
classification cs.LGcs.CL
keywords largelanguagemodelsinstructiondeductioninductionprompttuningvariationalautoencoderinformationbottleneckinductivereasoningparameter-efficientfine-tuning
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper sets out to show that the natural-language instructions we give a large language model and the numeric parameter changes that fine-tuning produces are not two separate things: both are compressions of the same task data, so one should be learnable from the other. To that end it builds SHIP, a framework that encodes an instruction into a latent vector of 10 prompt-tuning soft tokens, uses that vector as the model's task-specific parameters, and decodes the same vector back into the instruction. On instruction deduction (given an instruction, predict the output) SHIP beats supervised fine-tuning and a meta-learning baseline; on instruction induction (given input-output examples, recover the instruction) it surpasses the baselines by large margins, reaching 85.56 percent accuracy on seen SNI tasks versus 49.20 percent for vanilla SFT. The paper also shows that combining the two directions — inducing an instruction from examples, then re-encoding it — yields strong few-shot inductive reasoning without any instruction at test time. The claim, read sympathetically, is that task knowledge in an LLM has a common code that is legible both as text and as parameters.

What carries the argument

The load-bearing object is the latent instruction code $z$, a 10-token soft-prompt vector of dimension $10 \times 4096 = 40960$ that plays two roles at once: it is the trainable parameter block that the frozen Task LLM conditions on to solve the task (the variational information bottleneck objective), and it is the bottleneck variable from which the decoder reconstructs the instruction text (the variational autoencoder objective). This hybrid VAE-VIB setup, trained end-to-end with a KL regularizer, is what lets a single vector be translated into both a parameter update and a natural-language instruction. A second mechanism, the indirect training trick, encodes a trainable tensor $\tilde{k}$ into $\tilde{z}$ during induction fine-tuning so that the converged $z^*$ stays consistent with the encoder used at inference time.

What would settle it

Re-run the induction task but score the predicted instruction $\hat{k}$ by exact match or by human semantic-equivalence rating instead of the paper's external LLM judge, and compare SHIP's margin over the prompting baseline; if the margin largely vanishes under stricter scoring, the claim that the parameters carry recoverable instruction content is not supported.

Watch

Extended reading notes

Core claim

SHIP's central discovery is that the mutual mapping between instructions and parameters is learnable: a 40,960-dimensional latent vector $z$, obtained by prompt tuning 10 soft tokens on a frozen Llama-2-7b-chat model, can serve simultaneously as the task-specific parameters of the LLM and as the bottleneck code of a variational autoencoder that reconstructs the instruction. Training minimizes three losses — a task loss (variational information bottleneck), an instruction reconstruction loss (VAE), and a KL regularizer — with textual conditions (the instruction for the task LLM, one input-output pair for the decoder) to keep both models on their pretrained language distributions. With that single objective, SHIP learns both directions: $\mathrm{Enc}(k)$ maps an instruction to parameters for deduction, and a converged $z^*$ from fine-tuning on a handful of examples maps back to a human-readable instruction $\hat{k}$. The paper demonstrates that this mapping outperforms SFT, TAGI, and ItD on their respective tasks, and that the induction-then-deduction loop (SHIP-Refined) clearly beats in-context learning and prior instruction-induction methods on inductive reasoning.

Load-bearing premise

The load-bearing premise is that a 10-token soft-prompt vector is a faithful enough stand-in for the model's parameters that a mapping between instructions and this vector really is a mapping between instructions and parameters; if prompt tuning cannot capture task-specific knowledge, SHIP becomes just a variational autoencoder over instructions with an auxiliary task loss.

Editorial extensions

If this is right

  • Instructions alone can become task-specific parameters, so a new task could be handled without collecting supervised fine-tuning data.
  • Trained parameters can be decoded into human-readable instructions, giving a route to explaining what a fine-tuned model has learned.
  • The induce-then-re-encode loop provides a parameter-based method for few-shot inductive reasoning that outperforms in-context learning and prior instruction-induction methods.
  • A version of SHIP pretrained only on general instruction-following data still transfers to unseen tasks, suggesting the shuttle improves with scale.
  • Induction accuracy keeps improving with more pretraining data after deduction accuracy has already converged, consistent with induction being the harder direction.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the shuttle generalizes beyond the two studied datasets, one could edit a model's behavior by editing the induced instruction text rather than the weights, effectively a soft form of interpretable model surgery.
  • The shared latent code suggests a 'task language' may emerge in the 40,960-dimensional space; clustering $z$ vectors across tasks could yield a taxonomy of task similarity that is neither purely textual nor purely behavioral.
  • A testable extension would be to replace the natural-language instruction with formal rules (the authors list this as future scope) and check whether the encoder maps rules to parameters more or less reliably than instructions.
  • One could probe whether the mapping is genuinely bidirectional by finding two tasks that share the same induced instruction but require different input-output behavior; if SHIP then fails on one, the code may capture surface text rather than task semantics.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 5 minor

Summary. The paper proposes SHIP, a variational autoencoder/information-bottleneck framework that learns bidirectional mappings between task instructions k and a trainable soft-prompt vector z. An encoder maps k to z, a decoder reconstructs k from z while conditioned on an example pair (x*, y*), and a frozen Task LLM uses z to predict task outputs; the two directions are evaluated as instruction deduction (k, x -> y) and instruction induction (examples -> k) on SNI and P3, plus a combined inductive-reasoning task. The authors report large induction gains over baselines (e.g., 85.56 vs. 49.20 for SFT on SNI seen tasks, Table 1) and substantial improvements in inductive reasoning (Table 3), with code and data released.

Significance. If the mutual-mapping claim were fully established, SHIP would be a useful contribution to parameter-efficient instruction-conditioned adaptation and to interpreting task-specific latent states, and the released code and data are a clear strength for reproducibility. The strongest empirical results are in induction and in the combined inductive-reasoning task, where the reported gaps over baselines are large and non-obvious. However, the significance is currently limited by three unresolved issues: the 'parameters' are only 10 soft-prompt tokens rather than model weights; the deduction claim is not uniformly supported by Table 1; and the parameter-to-instruction direction is entangled with example conditioning, as shown by the ablation in Table 2.

major comments (5)
  1. [Section 4, Table 1 (SNI, unseen tasks, deduction)] SHIP-in-domain scores 21.11 on deduction for SNI unseen tasks, below vanilla SFT (28.56) and TAGI (23.33). This directly contradicts the abstract and Section 4.1, which state that SHIP 'performs better than existing baseline methods in terms of deductive capabilities' and 'demonstrates better deduction ability compared to SFT and TAGI.' The claim should be restricted to the settings where it holds, or the SNI unseen discrepancy must be explained.
  2. [Section 2.3, Eq. (14), and Table 2] Induction decoding conditions the decoder on a randomly sampled example pair (x*, y*). Removing this textual condition collapses induction accuracy from 85.56 to 0.53 on SNI seen tasks and from 78.33 to 11.67 on P3 seen tasks. Since the decoder is itself a LoRA-adapted LLM, the reported 'parameters-to-instructions' mapping may be mostly example-based instruction induction, with z* contributing little. The paper needs a control that varies z* while holding (x*, y*) fixed (e.g., decode from a random or untrained z* under the same example pair) and shows that the optimized z* is necessary; without such a control, the claim in Section 4 that SHIP maps parameters to instructions is not established.
  3. [Section 3] z is defined as 10 soft-prompt tokens (dimension 10 x 4096 = 40960), and all other Task LLM parameters are frozen. This makes SHIP a shuttle between instructions and a prompt-embedding vector, not between instructions and 'the parameters of LLMs' as claimed in the title and abstract. The motivational applications in Section 1 (rapid parametric adaptation, verifying learned representations in model parameters) require evidence that prompt-level z behaves like, or transfers to, actual weight/parameter states. Please either narrow the terminology throughout or add an experiment comparing z to full-parameter or LoRA task vectors.
  4. [Section 2.2, Eq. (9), and Table 2] Deductive inference supplies the instruction k as a textual condition to the Task LLM in addition to the encoded z. The ablation 'w/o textual condition k' degrades deduction substantially (e.g., SNI unseen deduction falls from 21.11 to 4.44 for SHIP-in-domain). Thus the paper has not isolated the contribution of the learned mapping from instructions to z; the improvement over prompting may partly reflect the exposed instruction. A deduction variant using p_task(y|z; x) without k, or an explicit discussion of this confound, is needed to support the 'from-instructions-to-parameters' direction.
  5. [Section 3 and NeurIPS checklist item 7] The experiments are reported as a single run with no error bars, and the checklist states that repeated experiments were not performed. Several comparisons that support the claims are small (e.g., SNI seen deduction 33.26 vs. TAGI 32.02; P3 unseen induction 28.57 vs. ItD 28.57). Without variance estimates or multiple seeds over task splits and initializations, the relative statements in Section 4.1 are not robust. Please report at least 3-5 runs and confidence intervals for the main tables, or clearly mark the claims as preliminary.
minor comments (5)
  1. [Abstract and Section 5] There is a typo in the transition: 'SHIP can combine the mappings to perform inductive reasoning.inductive reasoning requires' should be split into two sentences with proper capitalization.
  2. [Section 4.1, Table 1] The text says SHIP outperforms ItD and vanilla SFT on unseen tasks 'by a large margin,' but on P3 unseen induction SHIP-in-domain ties ItD at 28.57; the statement should be qualified.
  3. [Section 3, dataset description] The paper does not report the number of tasks or test instances in the seen/unseen splits. Accuracy values such as 21.11 suggest small denominators, so the sample sizes should be stated explicitly.
  4. [Section 5.2, Figure 5] The t-SNE plots are qualitative; please add a quantitative measure of alignment between SHIP-Refined and ground-truth latents (e.g., average cosine similarity or distance in z-space) to support the claim that refinement brings z closer to the true semantic representation.
  5. [References] References [26] and [27] are the same paper (Jeffrey Zhou et al., arXiv:2311.07911); one should be removed or the two citations should be merged.

Circularity Check

1 steps flagged · score 6.0 of 10

Seen-task instruction 'prediction' reduces by construction to the decoder's trained (x,y)→k reconstruction (Table 2: 85.56→0.53 without the textual pair); otherwise the paper is empirically self-contained, with no load-bearing self-citation.

  1. fitted input called prediction [Sec. 2.1 (Eq. 4), Sec. 2.3 (Eq. 14), Sec. 3 (Dataset), Table 2]
    "Lrecon = − log pdec(k|z; x, y) (4) ... we add textual condition: instruction k for the Task LLM, and one pair of instance x, y for the Decoder (ablation results are in Table 2) ... ˆk ∼ pdec(·|z∗; x∗, y∗) (14) ... For each subtask of instruction k, we only leave 5 instances x, y as test samples ... (Table 2) w/o textual condition x, y: 0.53."

    Decoder training (Eq. 4) reconstructs k from (Enc(k), x, y) with (x,y) from the same task. Induction inference (Eq. 14) feeds the decoder (Enc(k*), x*, y*), where k* is fitted on the task's own 5 test samples and (x*, y*) is sampled from that same set; for seen tasks the decoded target k is a decoder training label (the split holds out samples, not instructions). The seen-task 'prediction' therefore reduces by construction to the trained (x,y)→k association: the paper's own ablation 'w/o textual condition x, y' drops SNI seen induction from 85.56 to 0.53 (Table 2), while SHIP-pretrain, which never saw these labels, reaches only 36.36 on the same split.

full rationale

This is a framework paper without a formal derivation chain; the 'shuttle' is an empirical VAE+VIB hybrid trained end-to-end on (k,x,y) triples and applied by Eqs. 8-14. The deduction evaluation is standard conditional generation with an external judge (gpt-4o-mini) and external/hypernetwork baselines, so it is not circular. The one partial circularity found is confined to the flagship seen-task induction numbers. Because the dataset split holds out only 5 samples per task while keeping the instruction k inside the decoder's LoRA training distribution, the induction pipeline — fit k* on the 5 samples, encode it, decode it under the textual condition (x*, y*) drawn from the same 5 samples — regenerates a near-memorized training label. Table 2 makes the reduction explicit: dropping the textual (x*, y*) condition collapses induction from 85.56 to 0.53 on SNI seen, and SHIP-pretrain, which never trained on those labels, reaches only 36.36. The genuinely out-of-distribution induction numbers (SNI unseen 44.44, P3 unseen 28.57 for SHIP-in-domain) are non-trivial and honestly reported, as are the weak OOD deduction numbers (21.11 SNI unseen), so the central claim retains independent content. The ablation 'w/o indirect training' (1.59) also shows that directly optimized prompts cannot be decoded, i.e., the parameters→instructions mapping only holds for Encoder-produced latents — a scoping limitation the paper discloses rather than hides. Self-citations are present (ItD [18], TAGI [12], and related papers share author overlaps) but are used as baselines or related work, and the 'induction is harder than deduction' remark is co-supported by external references [2,20]; there is no load-bearing self-citation chain, no uniqueness theorem, and no ansatz imported from the authors' own prior work. The 'parameters are a 10-token soft prompt' identification is a framing choice, not a self-definitional circularity, since z is consistently defined as the trainable Task-LLM parameters. Overall: one headline prediction partially reduces to its training label by construction, which places this paper above the fully-self-contained band; the candid ablations and external benchmarks prevent a higher score.

Assumptions & free parameters 3 free parameters · 4 assumptions · 0 invented entities

SHIP does not posit new physical entities; it is a neural architecture with a learned latent. The main commitments are the compression-correlation premise, the prompt-tuning-as-parameters equivalence, and the reliability of LLM-based evaluation.

free parameters (3)
  • loss weights w0, w1, w2 = 1e-3, 1.0, 1.0
    Chosen by hand; they balance KL regularization, task prediction, and instruction reconstruction, and no sensitivity analysis is reported.
  • number of soft prompt tokens = 10
    Sets the dimension of the latent 'parameters' z to 10 x 4096; no sweep over this value is reported.
  • LoRA ranks for Encoder/Decoder = 16 and 1
    Rank choices for parameter-efficient adaptation; no ablation is reported.
assumptions (4)
  • domain assumption Instructions and optimized parameters are both compression of task data and therefore strongly correlated.
    Section 1 states this premise without proof; the entire framework is built on it.
  • domain assumption A 10-token soft prompt can serve as the task parameters of a frozen LLM.
    Section 3 uses prompt tuning as z; the paper provides no evidence that this space is representative of actual LLM parameters.
  • domain assumption The external LLM judge (gpt-4o-mini) provides reliable correctness and equivalence judgments.
    All accuracy numbers in Tables 1 to 3 are derived from this judge; no human validation is reported.
  • domain assumption The optimized z* obtained by the indirect training trick lies in the training latent distribution.
    Section 2.3 introduces the indirect trick to avoid train-inference inconsistency, but the paper offers only ablations, not a distributional guarantee.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Shuttle Between the Instructions and the Parameters of Large Language Models." pith.science (2026). https://pith.science/paper/GP4KRIHV

@misc{pith2026250202315,
  author       = {Pith},
  title        = {Pith review of: Shuttle Between the Instructions and the Parameters of Large Language Models},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/GP4KRIHV}},
  note         = {Machine review of arXiv:2502.02315}
}
read the original abstract

The interaction with Large Language Models (LLMs) through instructions has been extensively investigated in the research community. While instructions have been widely used as the guidelines for task solving, this paper further notices that both instructions and parameters are the compression of task data. Therefore, they could be strongly correlated and can be learned to predict one from the other. This paper proposes a novel neural network framework, SHIP (\textbf{Sh}uttle between the \textbf{I}nstructions and the \textbf{P}arameters), to model and learn the mutual mappings between the instructions and the parameters of LLMs. We verify that SHIP can effectively map one of the instructions/parameters to the other by evaluating it on the tasks of instruction deduction and induction. The results show that SHIP performs better than existing baseline methods in terms of deductive capabilities while significantly surpassing them in inductive capabilities. Moreover, SHIP can effectively combine the two mapping processes to perform excellent inductive reasoning. The code and data for this paper are released at https://anonymous.4open.science/r/Shuttle-Between-Instructions-Parameters/.

Figures

Figures reproduced from arXiv: 2502.02315 by the authors.

Figure 1
Figure 1. The concept of shuttling between the instructions and parameters, and the verification tasks. [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. The framework of SHIP. The Training process is represented with filled colors and the [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. The loss curve of training SHIP. Dataset. We adopt two popular multi-task instruction datasets: Super-Natural Instructions (SNI, [24]) and T0 split of P3 (P3, [17]) for evaluation. We first split each dataset into seen tasks (90%) and unseen tasks (10%). For each subtask of instruction k, we only leave 5 instances x, y as test samples, and use the rest as training samples. Therefore, for methods that are trained on … view at source ↗
Figures from the paper (8 more)
Figure 4
Figure 4. Figure 4: Analysis of SHIP’s generalization ability and few-shot induction ability. [PITH_FULL_IMAGE:figures/full_fig_p007_4.png]
Figure 5
Figure 5. Figure 5: The t-SNE result of latent z. 6 Related Work Instruction-based LLM Deduction. Given the task instruction and an input, how to enable LLM to faithfully perform deduction based on it, i.e., instruction following, has been widely considered by researchers. Previous studie…
Figure 6
Figure 6. Figure 6: The prompt for the external LLM to judge if the deduction result [PITH_FULL_IMAGE:figures/full_fig_p012_6.png]
Figure 7
Figure 7. Figure 7: The prompt for the external LLM to judge if the induction result [PITH_FULL_IMAGE:figures/full_fig_p012_7.png]
Figure 8
Figure 8. Figure 8: The prompt for the Baselines in Deduction. [PITH_FULL_IMAGE:figures/full_fig_p012_8.png]
Figure 9
Figure 9. Figure 9: The prompt for the Baselines in Induction. [PITH_FULL_IMAGE:figures/full_fig_p013_9.png]
Figure 10
Figure 10. Figure 10: An example case of the deduction task. 13 [PITH_FULL_IMAGE:figures/full_fig_p013_10.png]
Figure 11
Figure 11. Figure 11: An example case of the induction task. 14 [PITH_FULL_IMAGE:figures/full_fig_p014_11.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

43 extracted references · 17 canonical work pages

  1. [1]

    Deep variational information bottleneck

    Alexander A Alemi, Ian Fischer, Joshua V Dillon, and Kevin Murphy. Deep variational information bottleneck. arXiv preprint arXiv:1612.00410, 2016

    arXiv 2016

  2. [2]

    A multitask, multilingual, multimodal evaluation of chatgpt on reasoning, hallucination, and interactivity

    Yejin Bang, Samuel Cahyawijaya, Nayeon Lee, Wenliang Dai, Dan Su, Bryan Wilie, Holy Love- nia, Ziwei Ji, Tiezheng Yu, Willy Chung, et al. A multitask, multilingual, multimodal evaluation of chatgpt on reasoning, hallucination, and interactivity. arXiv preprint arXiv:2302.04023, 2023

    arXiv 2023

  3. [3]

    Generating sentences from a continuous space

    Samuel R Bowman, Luke Vilnis, Oriol Vinyals, Andrew M Dai, Rafal Jozefowicz, and Samy Bengio. Generating sentences from a continuous space. arXiv preprint arXiv:1511.06349, 2015

    arXiv 2015

  4. [4]

    Language modeling is compression

    Grégoire Delétang, Anian Ruoss, Paul-Ambroise Duquenne, Elliot Catt, Tim Genewein, Christo- pher Mattern, Jordi Grau-Moya, Li Kevin Wenliang, Matthew Aitchison, Laurent Orseau, et al. Language modeling is compression. arXiv preprint arXiv:2309.10668, 2023

    arXiv 2023

  5. [5]

    Large language models are not abstract reasoners

    Gaël Gendron, Qiming Bao, Michael Witbrock, and Gillian Dobbie. Large language models are not abstract reasoners. arXiv preprint arXiv:2305.19555, 2023

    arXiv 2023

  6. [6]

    Bowman, and Omer Levy

    Or Honovich, Uri Shaham, Samuel R. Bowman, and Omer Levy. Instruction induction: From few examples to natural language task descriptions. In Anna Rogers, Jordan Boyd-Graber, and Naoaki Okazaki, editors, Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1935–1952, Toronto, Canada, July

    work page 1935

  7. [7]

    Lora: Low-rank adaptation of large language models

    Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. arXiv preprint arXiv:2106.09685, 2021

    arXiv 2021

  8. [8]

    HINT: Hypernetwork instruction tuning for efficient zero- and few-shot generalisation

    Hamish Ivison, Akshita Bhagia, Yizhong Wang, Hannaneh Hajishirzi, and Matthew Peters. HINT: Hypernetwork instruction tuning for efficient zero- and few-shot generalisation. In Anna Rogers, Jordan Boyd-Graber, and Naoaki Okazaki, editors, Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , pages...

    work page 2023

Show all 43 references
  1. [9]

    Auto-encoding variational bayes

    Diederik P Kingma. Auto-encoding variational bayes. arXiv preprint arXiv:1312.6114, 2013

    2013 arXiv

  2. [10]

    Variational dropout and the local reparame- terization trick

    Durk P Kingma, Tim Salimans, and Max Welling. Variational dropout and the local reparame- terization trick. Advances in neural information processing systems, 28, 2015

    2015

  3. [11]

    The power of scale for parameter-efficient prompt tuning

    Brian Lester, Rami Al-Rfou, and Noah Constant. The power of scale for parameter-efficient prompt tuning. arXiv preprint arXiv:2104.08691, 2021

    2021 arXiv

  4. [12]

    From instance training to instruction learning: Task adapters generation from instructions

    Huanxuan Liao, Yao Xu, Shizhu He, Yuanzhe Zhang, Yanchao Hao, Shengping Liu, Kang Liu, and Jun Zhao. From instance training to instruction learning: Task adapters generation from instructions. arXiv preprint arXiv:2406.12382, 2024

    2024 arXiv

  5. [13]

    Large language models as general pattern machines

    Suvir Mirchandani, Fei Xia, Pete Florence, Brian Ichter, Danny Driess, Montserrat Gonzalez Arenas, Kanishka Rao, Dorsa Sadigh, and Andy Zeng. Large language models as general pattern machines. arXiv preprint arXiv:2307.04721, 2023

    2023 arXiv

  6. [14]

    Comparing humans, gpt-4, and gpt-4v on abstraction and reasoning tasks

    Melanie Mitchell, Alessandro B Palmarini, and Arseny Moskvichev. Comparing humans, gpt-4, and gpt-4v on abstraction and reasoning tasks. arXiv preprint arXiv:2311.09247, 2023. 10

    2023 arXiv

  7. [15]

    Infobench: Evaluating instruction following ability in large language models

    Yiwei Qin, Kaiqiang Song, Yebowen Hu, Wenlin Yao, Sangwoo Cho, Xiaoyang Wang, Xuan- sheng Wu, Fei Liu, Pengfei Liu, and Dong Yu. Infobench: Evaluating instruction following ability in large language models. arXiv preprint arXiv:2401.03601, 2024

    2024 arXiv

  8. [16]

    Phenomenal yet puzzling: Testing inductive reasoning capabilities of language models with hypothesis refinement

    Linlu Qiu, Liwei Jiang, Ximing Lu, Melanie Sclar, Valentina Pyatkin, Chandra Bhagavatula, Bailin Wang, Yoon Kim, Yejin Choi, Nouha Dziri, et al. Phenomenal yet puzzling: Testing inductive reasoning capabilities of language models with hypothesis refinement. arXiv preprint arXi...

    arXiv

  9. [17]

    Multitask prompted training enables zero-shot task generalization

    Victor Sanh, Albert Webson, Colin Raffel, Stephen H Bach, Lintang Sutawika, Zaid Alyafeai, Antoine Chaffin, Arnaud Stiegler, Teven Le Scao, Arun Raja, et al. Multitask prompted training enables zero-shot task generalization. arXiv preprint arXiv:2110.08207, 2021

    arXiv

  10. [18]

    Itd: Large language models can teach themselves induction through deduction

    Wangtao Sun, Haotian Xu, Xuanqing Yu, Pei Chen, Shizhu He, Jun Zhao, and Kang Liu. Itd: Large language models can teach themselves induction through deduction. arXiv preprint arXiv:2403.05789, 2024

    2024 arXiv

  11. [19]

    Beyond instruction following: Evaluating inferential rule following of large language models

    Wangtao Sun, Chenxiang Zhang, XueYou Zhang, Xuanqing Yu, Ziyang Huang, Pei Chen, Haotian Xu, Shizhu He, Jun Zhao, and Kang Liu. Beyond instruction following: Evaluating inferential rule following of large language models. arXiv preprint arXiv:2407.08440, 2024

    2024 arXiv

  12. [20]

    Large language models are in-context semantic reasoners rather than symbolic reasoners

    Xiaojuan Tang, Zilong Zheng, Jiaqi Li, Fanxu Meng, Song-Chun Zhu, Yitao Liang, and Muhan Zhang. Large language models are in-context semantic reasoners rather than symbolic reasoners. arXiv preprint arXiv:2305.14825, 2023

    2023 arXiv

  13. [21]

    Llama 2: Open foundation and fine-tuned chat models

    Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288, 2023

    2023 arXiv

  14. [22]

    Visualizing data using t-sne

    Laurens Van der Maaten and Geoffrey Hinton. Visualizing data using t-sne. Journal of machine learning research, 9(11), 2008

    2008

  15. [23]

    Hypothesis search: Inductive reasoning with language models.arXiv preprint arXiv:2309.05660, 2023

    Ruocheng Wang, Eric Zelikman, Gabriel Poesia, Yewen Pu, Nick Haber, and Noah D Goodman. Hypothesis search: Inductive reasoning with language models.arXiv preprint arXiv:2309.05660, 2023

    2023 arXiv

  16. [24]

    Super-naturalinstructions: Generalization via declarative instructions on 1600+ nlp tasks

    Yizhong Wang, Swaroop Mishra, Pegah Alipoormolabashi, Yeganeh Kordi, Amirreza Mirzaei, Anjana Arunkumar, Arjun Ashok, Arut Selvan Dhanasekaran, Atharva Naik, David Stap, et al. Super-naturalinstructions: Generalization via declarative instructions on 1600+ nlp tasks. arXiv pre...

    2022 arXiv

  17. [25]

    Llm-driven instruction following: Progresses and concerns

    Wenpeng Yin, Qinyuan Ye, Pengfei Liu, Xiang Ren, and Hinrich Schütze. Llm-driven instruction following: Progresses and concerns. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing: Tutorial Abstracts, pages 19–25, 2023

    2023

  18. [27]

    True" or “False

    Jeffrey Zhou, Tianjian Lu, Swaroop Mishra, Siddhartha Brahma, Sujoy Basu, Yi Luan, Denny Zhou, and Le Hou. Instruction-following evaluation for large language models. arXiv preprint arXiv:2311.07911, 2023. 11 A Instruction-following Data for Pretraining SHIP We collect and pro...

    2023 arXiv

  19. [29]

    Guidelines: • The answer NA means that the abstract and introduction do not include the claims made in the paper

    Claims Question: Do the main claims made in the abstract and introduction accurately reflect the paper’s contributions and scope? Answer: [Yes] Justification: We clearly claim the contributions and scope of this paper in the abstract and introduction. Guidelines: • The answer ...

  20. [30]

    Limitations

    Limitations Question: Does the paper discuss the limitations of the work performed by the authors? Answer: [Yes] Justification: We discuss the limitations of the work in the Section 7. Guidelines: • The answer NA means that the paper has no limitation while the answer No means...

  21. [31]

    Guidelines: • The answer NA means that the paper does not include theoretical results

    Theory assumptions and proofs Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof? Answer: [NA] 15 Justification: This paper does not include theoretical results. Guidelines: • The answer NA means that th...

  22. [32]

    Guidelines: • The answer NA means that the paper does not include experiments

    Experimental result reproducibility Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and da...

  23. [33]

    Guidelines: • The answer NA means that paper does not include experiments requiring code

    Open access to data and code Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material? 16 Answer: [Yes] Justification: The data and code can be...

  24. [34]

    Guidelines: • The answer NA means that the paper does not include experiments

    Experimental setting/details Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results? Answer: [Yes] Justification: The settings of experiments a...

  25. [35]

    Guidelines: • The answer NA means that the paper does not include experiments

    Experiment statistical significance Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments? Answer: [No] Justification: The training and evaluation rely on locally deploye...

  26. [36]

    Guidelines: • The answer NA means that the paper does not include experiments

    Experiments compute resources Question: For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments? Answer: [Yes] Justification: We include these detail...

  27. [37]

    Guidelines: • The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics

    Code of ethics Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines? Answer: [Yes] Justification: The research conducted in the paper conforms, in every respect, with the NeurIP...

  28. [38]

    We believe its impact should be confined to the academic domain

    Broader impacts Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed? Answer: [NA] Justification: We do not foresee any social impact of our work in its current form. We believe its impact should be confi...

  29. [39]

    Guidelines: • The answer NA means that the paper poses no such risks

    Safeguards Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)? Answer: [NA] Justification: Our models have...

  30. [40]

    Guidelines: • The answer NA means that the paper does not use existing assets

    Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected? Answer: [Yes] Justification: All code, data, ...

  31. [41]

    Guidelines: • The answer NA means that the paper does not release new assets

    New assets Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets? Answer: [Yes] Justification: We provide a README in our code release, which we plan to gradually improve in our open-source repository. Guideline...

  32. [42]

    Guidelines: • The answer NA means that the paper does not involve crowdsourcing nor research with human subjects

    Crowdsourcing and research with human subjects Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)? A...

  33. [43]

    Guidelines: • The answer NA means that the paper does not involve crowdsourcing nor research with human subjects

    Institutional review board (IRB) approvals or equivalent for research with human subjects Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals...

  34. [44]

    Answer: [NA] Justification: The core method development in this research does not involve LLMs as any important, original, or non-standard components

    Declaration of LLM usage Question: Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the ...

    2025

  35. [2023]

    Association for Computational Linguistics

Pith tools

Reviewed August 9, 2026 · model on record in the stance chip above.