REVIEW 7 cited by
Tuning Language Models by Proxy
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
read the original abstract
Despite the general capabilities of large pretrained language models, they consistently benefit from further adaptation to better achieve desired behaviors. However, tuning these models has become increasingly resource-intensive, or impossible when model weights are private. We introduce proxy-tuning, a lightweight decoding-time algorithm that operates on top of black-box LMs to achieve the same end as direct tuning, but by accessing only its predictions over the output vocabulary, not its parameters. Our method tunes a smaller LM, then applies the difference between the predictions of the small tuned and untuned LMs to shift the original predictions of the larger untuned model in the direction of tuning, while retaining the benefits of larger-scale pretraining. In experiments, when we apply proxy-tuning to Llama2-70B using proxies of only 7B size, we can close 88% of the gap between Llama2-70B and its truly-tuned chat version, when evaluated across knowledge, reasoning, and safety benchmarks. We then demonstrate the generality of proxy-tuning by applying it to domain adaptation on code, and task-specific finetuning on question-answering and math problems. Finally, we show how to proxy-tune a truly black-box LM, GPT-3.5, for temporal adaptation, increasing its knowledge about recent events. Our work demonstrates the promise of using small tuned LMs to efficiently customize large, potentially proprietary LMs through decoding-time guidance.
Forward citations
Cited by 7 Pith papers
-
Weak-to-Strong Generalization via Direct On-Policy Distillation
Transferring the log-ratio of a small model's pre-RL and post-RL checkpoints provides a dense implicit reward that improves stronger student models at a fraction of the cost of direct RL.
-
Weak-to-Strong On-Policy Distillation
A strong LLM is improved by distilling from the logit difference of two weaker models instead of from a stronger teacher.
-
Proxy Exploration and Reusable Guidance: A Modular LLM Post-Training Paradigm via Proxy-Guided Update Signals
Relative policy-improvement signals from a weak proxy model, after simple calibration, can be transferred to improve stronger primary LLMs without re-exploring on the primary.
-
A Survey on Training-free Alignment of Large Language Models
A survey that catalogs and categorizes training-free LLM alignment methods into pre-decoding, in-decoding, and post-decoding, with a limited experimental comparison on one model.
-
Logit Arithmetic Elicits Long Reasoning Capabilities Without Training
ThinkLogit blends logits from a small reasoning guider into a frozen 32B model, improving math pass@1 by up to 29% without training the large model.
-
Scalable, Symbiotic, AI and Non-AI Agent Based Parallel Discrete Event Simulations
PDES orchestration of small language models with non-AI verifier agents raises accuracy on four toy tasks from about 23 percent to 68 percent, with the verifiers supplying most of the correctness.
-
Neural Incompatibility: The Unbridgeable Gap of Cross-Scale Parametric Knowledge Transfer in Large Language Models
Directly transferring parameters between differently-sized language models is unreliable; the paper proposes a pre-alignment method (LaTen) and explains the failure via 'Neural Incompatibility'.
Discussion (0). Sign in to comment.