REVIEW 17 cited by
Efficient and Effective Text Encoding for Chinese LLaMA and Alpaca
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
read the original abstract
Large Language Models (LLMs), such as ChatGPT and GPT-4, have dramatically transformed natural language processing research and shown promising strides towards Artificial General Intelligence (AGI). Nonetheless, the high costs associated with training and deploying LLMs present substantial obstacles to transparent, accessible academic research. While several large language models, such as LLaMA, have been open-sourced by the community, these predominantly focus on English corpora, limiting their usefulness for other languages. In this paper, we propose a method to augment LLaMA with capabilities for understanding and generating Chinese text and its ability to follow instructions. We achieve this by extending LLaMA's existing vocabulary with an additional 20,000 Chinese tokens, thereby improving its encoding efficiency and semantic understanding of Chinese. We further incorporate secondary pre-training using Chinese data and fine-tune the model with Chinese instruction datasets, significantly enhancing the model's ability to comprehend and execute instructions. Our experimental results indicate that the newly proposed model markedly enhances the original LLaMA's proficiency in understanding and generating Chinese content. Additionally, the results on the C-Eval dataset yield competitive performance among the models with several times the size of ours. We have made our pre-trained models, training scripts, and other resources available through GitHub, fostering open research for our community. Chinese LLaMA series: \url{https://github.com/ymcui/Chinese-LLaMA-Alpaca} and Chinese Llama-2 series: \url{https://github.com/ymcui/Chinese-LLaMA-Alpaca-2}
Forward citations
Cited by 17 Pith papers
-
Beyond Initialization Loss: A Systematic Study of Token Embedding Initialization Strategies for LLM Vocabulary Extension
For Hindi vocabulary extension of a 30B LLM, the best embedding initialization is uniform subword averaging with Hindi norm calibration on the input and character-length-weighted averaging on the output, cutting conti...
-
In-Place Tokenizer Expansion for Pre-trained LLMs
Continuing a model's own BPE merges and training only new embedding rows preserves quality while cutting token counts 2.4–4× for previously under-tokenized languages.
-
It's All About In-Context Learning! Teaching Extremely Low-Resource Languages to LLMs
Zero-shot in-context learning with word- or sentence-level English alignment outperforms fine-tuning for LLMs on extremely low-resource languages with rare scripts.
-
Preperiodic points, finiteness, and structures of semigroups of algebraic morphisms
The paper proves finiteness and structural results for preperiodic points of algebraic morphisms, including Burnside-type and Northcott-type theorems.
-
CoVE: Compressed Vocabulary Expansion Makes Better LLM-based Recommender Systems
CoVE assigns each item a unique token ID, tunes item embeddings and the LM head, and predicts the next item from logits, beating finetune-and-retrieval baselines by up to 62 percent with a 16x compressed embedding table.
-
Thunder-LLM: Efficiently Adapting LLMs to Korean with Minimal Resources
A cost-effective recipe consisting of tokenizer extension, continual pretraining, FP8 training, and SFT/DPO post-training yields Korean-English bilingual 8B models with top Korean benchmark scores.
-
Training-free LLM Merging for Multi-task Learning
Hi-Merging merges two task-specialized LLMs by pruning and scaling delta vectors at model and layer level, reporting gains over prior merging and multi-task fine-tuning on English and Chinese MCQA and QA tasks.
-
The Emergence of Abstract Thought in Large Language Models Beyond Any Language
Across 20 open LLMs, shared multilingual neurons grow in number and per-neuron importance over release generations, which the authors interpret as evidence of language-agnostic abstract thought and use to guide neuron...
-
Multi-objective Aligned Bidword Generation Model for E-commerce Search Advertising
A generator trained with a three-objective discriminator and multi-objective preference alignment improves bidword generation for e-commerce search, improving offline retrieval metrics and online ad revenue.
-
CC-Tuning: A Cross-Lingual Connection Mechanism for Improving Joint Multilingual Supervised Fine-Tuning
CC-Tuning fuses English feed-forward activations into non-English inputs during multilingual supervised fine-tuning, using a trainable Decision Maker and a least-squares Transform Matrix to simulate the connection at ...
-
Cross-Lingual Optimization for Language Transfer in Large Language Models
CLO, a modified DPO loss that contrasts English and translated target-language responses in the same batch, improves target-language instruction following and preserves English better than standard SFT.
-
Evaluating Reliability Asymmetries in Chinese Factual Search and AI Answers
Chinese search engines, LLMs, and AI Overviews are comparably accurate when they commit to an answer but differ sharply in answer rate, and all are systematically better on yes-labeled than no-labeled health-dominated...
-
DoPI: Doctor-like Proactive Interrogation LLM for Traditional Chinese Medicine
DoPI pairs a knowledge-graph-guided questioning model with a TCM expert model and claims 84.68% diagnostic accuracy, but the benchmark is built from the same symptom-disease rules that drive the system.
-
Pruning General Large Language Models into Customized Expert Models
Cus-Prun identifies and removes neurons that are irrelevant to a user's target language, domain, and task, producing specialized expert models without post-training.
-
Beyond the Sentence: A Survey on Context-Aware Machine Translation with Large Language Models
A survey of context-aware machine translation with large language models, categorizing prompting, fine-tuning, and agent-based approaches.
-
Distilling Closed-Source LLM's Knowledge for Locally Stable and Economic Biomedical Entity Linking
A closed-source LLM's re-ranking labels are distilled into a locally deployable open-source LLM, producing small but consistent Acc@1 gains in low-resource biomedical entity linking on two datasets.
- Prompt, Translate, Fine-Tune, Re-Initialize, or Instruction-Tune? Adapting LLMs for In-Context Learning in Low-Resource Languages
Discussion (0). Sign in to comment.