Pith. sign in

REVIEW 11 cited by

OpenELM: An Efficient Language Model Family with Open Training and Inference Framework

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2404.14619 v2 pith:OSBNGAQT submitted 2024-04-22 cs.CL cs.AIcs.LG

Sachin Mehta , Mohammad Hossein Sekhavat , Qingqing Cao , Maxwell Horton , Yanzi Jin , Chenfan Sun , Iman Mirzadeh , Mahyar Najibi
show 3 more authors
Dmitry Belenko Peter Zatloukal Mohammad Rastegari
This is my paper · ORCID
classification cs.CLcs.AIcs.LG
keywords modelopenopenelmlanguagereleasetrainingapplecode
verification ladder T0 review T1 audit T2 compute T3 formal
0 comments
abstract

The reproducibility and transparency of large language models are crucial for advancing open research, ensuring the trustworthiness of results, and enabling investigations into data and model biases, as well as potential risks. To this end, we release OpenELM, a state-of-the-art open language model. OpenELM uses a layer-wise scaling strategy to efficiently allocate parameters within each layer of the transformer model, leading to enhanced accuracy. For example, with a parameter budget of approximately one billion parameters, OpenELM exhibits a 2.36% improvement in accuracy compared to OLMo while requiring $2\times$ fewer pre-training tokens. Diverging from prior practices that only provide model weights and inference code, and pre-train on private datasets, our release includes the complete framework for training and evaluation of the language model on publicly available datasets, including training logs, multiple checkpoints, and pre-training configurations. We also release code to convert models to MLX library for inference and fine-tuning on Apple devices. This comprehensive release aims to empower and strengthen the open research community, paving the way for future open research endeavors. Our source code along with pre-trained model weights and training recipes is available at \url{https://github.com/apple/corenet}. Additionally, \model models can be found on HuggingFace at: \url{https://huggingface.co/apple/OpenELM}.

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 11 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. FantasyTalking2: Timestep-Layer Adaptive Preference Optimization for Audio-Driven Portrait Animation

    cs.CV 2025-08 unverdicted novelty 6.0 of 10

    A three-part system, Talking-Critic, Talking-NSQ, and TLPO, aligns diffusion portrait animation models to human preferences and improves lip-sync, motion naturalness, and visual quality.

  2. Minifinetuning: Low-Data Generation Domain Adaptation through Corrective Self-Distillation

    cs.LG 2025-05 conditional novelty 6.0 of 10

    MFT, a corrective self-distillation objective, reports 2 to 10 times better specialization-to-degeneralization ratios than standard finetuning across many models and three specialized domains.

  3. Next Token Perception Score: Analytical Assessment of your LLM Perception Skills

    cs.CL 2025-05 conditional novelty 6.0 of 10

    A subspace-overlap score computed from frozen LLM representations and labels is proposed, with a bounding theorem in a linear setting and empirical correlations with linear probe accuracy and LoRA gains.

  4. Crown, Frame, Reverse: Layer-Wise Scaling Variants for LLM Pre-Training

    cs.CL 2025-09 conditional novelty 5.0 of 10

    At 180M parameters and 5B tokens, all layer-wise scaling variants beat the paper's 18-layer uniform baseline, yet the 12-layer uniform baseline remains best.

  5. RefineX: Learning to Refine Pre-training Data at Scale from Expert-Guided Programs

    cs.CL 2025-07 conditional novelty 5.0 of 10

    A deletion-only program refiner, trained on expert end-to-end edits converted via minimum edit distance, improves LLM pretraining data and downstream accuracy.

  6. EdgeLoRA: An Efficient Multi-Tenant LLM Serving System on Edge Devices

    cs.DC 2025-07 conditional novelty 5.0 of 10

    EdgeLoRA combines automatic adapter routing, LRU caching with a memory pool, and grouped LoRA batching to serve thousands of LoRA adapters on edge devices with up to 4x higher throughput than llama.cpp.

  7. AlignHuman: Improving Motion and Fidelity via Timestep-Segment Preference Optimization for Audio-Driven Human Animation

    cs.CV 2025-06 conditional novelty 5.0 of 10

    Timestep-segment preference optimization with separate motion and fidelity LoRAs improves audio-driven human animation quality and allows a 3.3x inference speedup.

  8. Beyond Text Compression: Evaluating Tokenizers Across Scales

    cs.CL 2025-06 conditional novelty 5.0 of 10

    Tokenizer choice matters mostly for multilingual tasks, and 350M-parameter models can predict 2.7B model ranking on translation but not on English benchmarks.

  9. Rethinking Data Mixture for Large Language Models: A Comprehensive Survey and New Perspectives

    cs.CL 2025-05 accept novelty 5.0 of 10

    A survey organizing LLM data mixture methods into offline and online families, with a fine-grained taxonomy based on optimization frameworks.

  10. Collaborative Inference and Learning between Edge SLMs and Cloud LLMs: A Survey of Algorithms, Execution, and Open Challenges

    cs.DC 2025-07 conditional novelty 4.0 of 10

    A survey that builds a taxonomy of edge-cloud LLM-SLM collaboration for inference and training, claiming to be the first to unify both phases.

  11. Effective and Efficient Schema-aware Information Extraction Using On-Device Large Language Models

    cs.CL 2025-05 conditional novelty 4.0 of 10

    DLISC, a dual-LoRA two-stage schema-aware extraction method with incremental schema caching, reports better F1 and lower latency than three RAG baselines on two IE datasets, though the comparison lacks error bars and code.

Pith tools