REVIEW 11 cited by
LLM360: Towards Fully Transparent Open-Source LLMs
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
read the original abstract
The recent surge in open-source Large Language Models (LLMs), such as LLaMA, Falcon, and Mistral, provides diverse options for AI practitioners and researchers. However, most LLMs have only released partial artifacts, such as the final model weights or inference code, and technical reports increasingly limit their scope to high-level design choices and surface statistics. These choices hinder progress in the field by degrading transparency into the training of LLMs and forcing teams to rediscover many details in the training process. We present LLM360, an initiative to fully open-source LLMs, which advocates for all training code and data, model checkpoints, and intermediate results to be made available to the community. The goal of LLM360 is to support open and collaborative AI research by making the end-to-end LLM training process transparent and reproducible by everyone. As a first step of LLM360, we release two 7B parameter LLMs pre-trained from scratch, Amber and CrystalCoder, including their training code, data, intermediate checkpoints, and analyses (at https://www.llm360.ai). We are committed to continually pushing the boundaries of LLMs through this open-source effort. More large-scale and stronger models are underway and will be released in the future.
Forward citations
Cited by 11 Pith papers
-
Fluid Language Model Benchmarking
Fluid Benchmarking, combining IRT-based ability estimation with Fisher-information-based adaptive item selection, improves LM evaluation across efficiency, validity, variance, and saturation in pretraining settings.
-
Telescope: Improving Zero Shot Detection of LLM Generated Content By Measuring Token Repetition Probability
Telescope Perplexity, the average negative log probability a reference LM assigns to each token immediately after seeing it, yields strong zero-shot LLM-text detection by probing an early-training aversion to repetition.
-
OLMoASR: Open Models and Data for Training Robust Speech Recognition Models
An open 1M-hour English speech dataset plus Whisper-architecture models trained on it match Whisper's word error rates on short and long-form benchmarks.
-
CollabEval: Statistically Efficient Collaborative Model Evaluation via Matrix Completion
CollabEval turns model evaluation into low-rank matrix completion and uses the imputations as control variates to cut CI width and MSE at fixed annotation budget while preserving unbiasedness.
-
Falcon-H1: A Family of Hybrid-Head Language Models Redefining Efficiency and Performance
Falcon-H1 reports competitive benchmark scores for a 0.5B to 34B family of parallel hybrid attention/Mamba-2 models, claiming 2x to 4x parameter efficiency versus dense transformers.
-
The Common Pile v0.1: An 8TB Dataset of Public Domain and Openly Licensed Text
A new 8TB openly-licensed text corpus trains 7B LLMs that are competitive with Llama 1/2, showing that performant models need not depend on unlicensed web data.
-
Is Your Model Fairly Certain? Uncertainty-Aware Fairness Evaluation for LLMs
UCerF scores LLM fairness by both correctness and confidence, and SynthBias provides 31,756 gender-occupation coreference samples for benchmark testing.
-
ZUNA1.1: A more flexible EEG foundation model for Denoising and Super-resolution
ZUNA1.1, an open-source 380M EEG diffusion autoencoder, reconstructs variable-length, flexibly masked EEG at least as well as its predecessor and far better than spherical spline interpolation.
-
Index SLM Technical Report
Index-1.9B-Base reaches 64.92 average benchmark score via WSD training with late curated data plus Norm-Head, with open Pure/Boost controls isolating instruction-data inflation.
-
MEraser: An Effective Fingerprint Erasure Approach for Large Language Models
By fine-tuning on mismatched pairs and then clean pairs, MEraser drops fingerprint success rate to zero on three backdoor-based fingerprinting schemes across multiple LLMs, with a reusable LoRA adapter for transfer.
-
Steel-LLM:From Scratch to Open Source -- A Personal Journey in Building a Chinese-Centric LLM
Steel-LLM is a fully open 1B-parameter Chinese-centric LLM trained from scratch on ~1T tokens with 8 GPUs, reaching 41.9 C-Eval and 36.1 CMMLU after SFT and DPO.
Discussion (0). Continue with ORCID to comment.